Bug: Task throw Exception call cleanUpAllAllocatedMemory cause throw NPE 
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/23 03:32
Affects Version/s: 3.2.1

{code:java}
23/01/03 21:41:18 INFO SortBasedPusher: Pushdata is not empty , do push.
Traceback (most recent call last):
  File "/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/daemon.py", line 186, in manager
  File "/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/daemon.py", line 74, in worker
  File "/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/worker.py", line 643, in main
    if read_int(infile) == SpecialLengths.END_OF_STREAM:
  File "/mnt/ssd/0/yarn/nm-local-dir/usercache/rcmd_feature/appcache/application_1671694574014_2488441/container_e260_1671694574014_2488441_01_000107/pyspark.zip/pyspark/serializers.py", line 564, in read_int
    raise EOFError
EOFError
23/01/03 21:41:29 ERROR Executor: Exception in task 605.1 in stage 94.0 (TID 58026)
java.lang.NullPointerException
	at org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:399)
	at org.apache.spark.shuffle.rss.SortBasedPusher.pushData(SortBasedPusher.java:155)
	at org.apache.spark.shuffle.rss.SortBasedPusher.spill(SortBasedPusher.java:317)
	at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:177)
	at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:289)
	at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:116)
	at org.apache.spark.sql.execution.python.HybridRowQueue.createNewQueue(RowQueue.scala:227)
	at org.apache.spark.sql.execution.python.HybridRowQueue.add(RowQueue.scala:250)
	at org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$10(EvalPythonExec.scala:125)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
	at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:1159)
	at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:1174)
	at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1212)
	at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:1215)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:307)
	at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.writeIteratorToStream(PythonUDFRunner.scala:53)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:397)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2066)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:232)
 {code}