Bug: Class missing at runtime
Status: Resolved | Priority: Critical | Resolution: Not A Bug
Created: 25/Aug/22 02:04
Affects Version/s: 2.3.4

We found that some class cannot be found after the regionserver has been running for a period of time(more than a month). And more than half of the machines in our cluster have this problem. Some are still running normally, but _/rs-status_ cannot be opened，and others have various problems(such as rit, replication failed, abort failed)

Some exceptions are as follows:
{code:java}
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl
	at org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:49)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
	at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.hbase.http.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:52)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1495)
{code}
{code:java}
2022-08-24 19:22:52,536 ERROR [RS_CLOSE_REGION-regionserver/xxxx:26020-1] regionserver.HRegionServer: ***** ABORTING region server 10.x.x.x,26020,1659357427101: Replay of WAL required. Forcing server shutdown *****
org.apache.hadoop.hbase.DroppedSnapshotException: region: xxxx:xxxx,xxxxxx,1661298619920.943104cbcf4a74db9896fd4abd051411.
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2906)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2575)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2547)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2538)
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1652)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1591)
        at org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:114)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker$DeleteResult
        at org.apache.hadoop.hbase.regionserver.querymatcher.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:108)
        at org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.checkDeleted(ScanQueryMatcher.java:209)
        at org.apache.hadoop.hbase.regionserver.querymatcher.MinorCompactionScanQueryMatcher.match(MinorCompactionScanQueryMatcher.java:54)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:627)
        at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:127)
        at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:69)
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1067)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2442)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2842)
        ... 10 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.querymatcher.DeleteTracker$DeleteResult
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 19 more

{code}