Bug: Out of Memory when mistakenly set decay-scheduler.metrics.top.user.count to a large number
Status: Open | Priority: Critical | Resolution: 
Created: 20/Jul/23 16:43

h2. What happened:

When setting {{decay-scheduler.metrics.top.user.count}} to a large number, {{DecayRpcScheduler}} in Hcommon throws an out-of-memory exception due to inappropriate checking and handling.
Hcommon only checks the value should be larger than 0.
h2. Buggy code:

In DecayRpcScheduler.java
{noformat}
public DecayRpcScheduler(int numLevels, String ns, Configuration conf) {
  ...
  topUsersCount =                                                                                      
    conf.getInt(DECAYSCHEDULER_METRICS_TOP_USER_COUNT,                                               
      DECAYSCHEDULER_METRICS_TOP_USER_COUNT_DEFAULT);    <<---- topUsersCount gets the config value                                          
  Preconditions.checkArgument(topUsersCount > 0,     <<--- Only checks for positivity                      
    "the number of top users for scheduler metrics must be at least 1");
  ...
}
private void addTopNCallerSummary(MetricsRecordBuilder rb) {                                           
  TopN topNCallers = getTopCallers(topUsersCount);    <<--- calls getTopCallers with n equals topUsersCount
  ...
}
private TopN getTopCallers(int n) {                                                                    
  TopN topNCallers = new TopN(n); <<--- starts an priorityQ with initial capacity n, causing out of memory
  ...
}{noformat}
h2. StackTrace:
{noformat}
java.lang.OutOfMemoryError: Java heap space                                                                    
        at java.base/java.util.PriorityQueue.<init>(PriorityQueue.java:172)                                    
        at java.base/java.util.PriorityQueue.<init>(PriorityQueue.java:139)                                    
        at org.apache.hadoop.metrics2.util.Metrics2Util$TopN.<init>(Metrics2Util.java:80)                      
        at org.apache.hadoop.ipc.DecayRpcScheduler.getTopCallers(DecayRpcScheduler.java:1002)                  
        at org.apache.hadoop.ipc.DecayRpcScheduler.addTopNCallerSummary(DecayRpcScheduler.java:982)            
        at org.apache.hadoop.ipc.DecayRpcScheduler.getMetrics(DecayRpcScheduler.java:935)                      
        at org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.getMetrics(DecayRpcScheduler.java:893)         
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)      
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMB
eanServerInterceptor.java:329)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServ
erInterceptor.java:315)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:100)
        at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:73)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
        at org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.registerMetrics2Source(DecayRpcScheduler.java:8
19)
        at org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.<init>(DecayRpcScheduler.java:792)
        at org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.getInstance(DecayRpcScheduler.java:800)
        at org.apache.hadoop.ipc.DecayRpcScheduler.<init>(DecayRpcScheduler.java:260){noformat}
h2. Reproduce:

(1) Set {{decay-scheduler.metrics.top.user.count}} to a large value, e.g., 1419140791
(2) Run a simple test that exercises this parameter, e.g. {{org.apache.hadoop.ipc.TestDecayRpcScheduler#testNPEatInitialization}}

 

For an easy reproduction, run the reproduce.sh in the attachment.

We are happy to provide a patch if this issue is confirmed.