Bug: Race condition while registering source in MetricsSystem 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Apr/21 09:12
Affects Version/s: 2.3.2

{{MetricsSystem}} manages {{mutable.ArrayBuffer}} of metric sources. {{registerSource}} and {{removeSource}} methods are provided to add/remove new source from Metric system. Both these methods are not synchronised. Also, underlying {{mutable.ArrayBuffer}} not being thread safe, unexpected behaviours are possible if called concurrently.

Some background:
We have created one custom RpcEndPoint which receives messages from executors and create new metrics by registering custom sources. These messages are processed concurrently on driver side causing this issue. 

Also, this will go unnoticed as {{Inbox}} will ignore these exceptions.

We found this issue in Spark 2.3.2, but it should be present in all later versions.

For ex, we got below exception due to race condition
{noformat}
java.lang.ArrayIndexOutOfBoundsException
	at scala.collection.mutable.ResizableArray$class.ensureSize(ResizableArray.scala:104)
	at scala.collection.mutable.ArrayBuffer.ensureSize(ArrayBuffer.scala:48)
	at scala.collection.mutable.ArrayBuffer.$plus$eq(ArrayBuffer.scala:84)
	at org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:157)

	* some closure *

	at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745){noformat}
While shutting down
{noformat}
Exception in thread "stream execution thread for [id = 9a36a08a-f1be-4ad8-b1dd-093f0b53d37d, runId = d668a19c-aced-45c4-963c-c0b93411d1a4]" java.lang.ArrayIndexOutOfBoundsException: 32
	at scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:44)
	at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:48)
	at scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:195)
	at scala.collection.mutable.ArrayBuffer.segmentLength(ArrayBuffer.scala:48)
	at scala.collection.IndexedSeqOptimized$class.indexWhere(IndexedSeqOptimized.scala:204)
	at scala.collection.mutable.ArrayBuffer.indexWhere(ArrayBuffer.scala:48)
	at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:145)
	at scala.collection.AbstractSeq.indexOf(Seq.scala:41)
	at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:129)
	at scala.collection.AbstractSeq.indexOf(Seq.scala:41)
	at scala.collection.mutable.BufferLike$class.$minus$eq(BufferLike.scala:127)
	at scala.collection.mutable.AbstractBuffer.$minus$eq(Buffer.scala:49)
	at org.apache.spark.metrics.MetricsSystem.removeSource(MetricsSystem.scala:167)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runStream$2.apply(StreamExecution.scala:324)
	at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77)
	at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:308)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189){noformat}