Bug: spark stop failed to delete temp dir when running spark-submit on Windows with local[*] mode
Status: Open | Priority: Major | Resolution: 
Created: 19/Dec/24 22:22
Affects Version/s: 3.2.4

I am reopenning a very old issue that is still not resolved (but seems easy to resolve)
https://issues.apache.org/jira/browse/SPARK-12216.

{noformat}
Stopping spark context.
15/12/08 16:24:22 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\Stefan\AppData\Local\Temp\spark-18f2a418-e02f-458b-8325-60642868fdff
java.io.IOException: Failed to delete: C:\Users\Stefan\AppData\Local\Temp\spark-18f2a418-e02f-458b-8325-60642868fdff
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:884)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:63)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:60)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:60)
{noformat}


in 2015, this issue was already created, and closed soon after with reason "workaround, move to linux". It is occurring only on Windows, and only when doing "spark-submit" in "--master local[*]" (default) mode , which is admitedly a very rare case for real usages.

It appears that it give a very bad image of spark while teaching it to students who are mostly working on windows, and learning how to use "spark-submit" mode.

The root cause of the bug is that on windows, you can not delete a directory if there are still locked (=opened) files in it, and the temporary copy of the jar is indeed still locked by the Executor classLoader  

When you do
{noformat}
C:> spark-submit  --class %MY_CLASS%  "submitted-app.jar"
{noformat}

Internally, because it is "local[*]" mode, spark JVM has internally (>=) 2 classLoaders, 
- 1 classLoader for the "driver" part that points to the "submitted-app.jar" file given in argument (this file is OK, the directory is not touched), 
- 1 classLoader for the "executor" part, that points to a copy of the jar in a created temporary directory "C:\Users\<username>\AppData\Local\Temp\spark-<uuid>\submitted-app.jar".... This file causes the error at stop time.

The fix to it is simply to call the Executor URLClassLoader.close() method when everything else is already closed for the Executor, to really close correctly all files ressources.
This is in class Executor, accessible by "defaultSessionState.urlClassLoader"

{noformat}
diff --git a/core/src/main/scala/org/apache/spark/executor/Executor.scala b/core/src/main/scala/org/apache/spark/executor/Exeutor.scala
index f1087b695..23bf4ad06 100644
--- a/core/src/main/scala/org/apache/spark/executor/Executor.scala
+++ b/core/src/main/scala/org/apache/spark/executor/Executor.scala
@@ -467,6 +467,8 @@ private[spark] class Executor(
       if (!isLocal) {
         env.stop()
       }
+      logInfo("close Executor classLoader")
+      defaultSessionState.urlClassLoader.close(); // cf [SPARK-12216]
     }
   }
{noformat}

here are the new logs when closing correctly, with "<<<<<<<<" marker to show the added cleanup.

{noformat}
{"ts":"2024-12-19T21:38:16.860Z","level":"INFO","msg":"SparkContext is stopping with exitCode 0 from stop at SparkSqlApp.java:67.","context":{"exit_code":"0","stop_site_short_form":"stop at SparkSqlApp.java:67"},"logger":"SparkContext"}
{"ts":"2024-12-19T21:38:16.865Z","level":"INFO","msg":"Stopped Server@7affc159{STOPPING}[11.0.24,sto=30000]","logger":"Server"}
{"ts":"2024-12-19T21:38:16.865Z","level":"INFO","msg":"Shutdown Server@7affc159{STOPPING}[11.0.24,sto=30000]","logger":"Server"}
{"ts":"2024-12-19T21:38:16.872Z","level":"INFO","msg":"Stopped Spark@17295e14{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}","logger":"AbstractConnector"}
{"ts":"2024-12-19T21:38:16.883Z","level":"INFO","msg":"Stopped Spark web UI at http://DesktopArnaud:4040","context":{"web_url":"http://DesktopArnaud:4040"},"logger":"SparkUI"}

<<<<<<<<
{"ts":"2024-12-19T21:38:16.889Z","level":"INFO","msg":"close Executor classLoader ","logger":"Executor"}
<<<<<<<<

{"ts":"2024-12-19T21:38:16.902Z","level":"INFO","msg":"MapOutputTrackerMasterEndpoint stopped!","logger":"MapOutputTrackerMasterEndpoint"}
{"ts":"2024-12-19T21:38:16.961Z","level":"INFO","msg":"MemoryStore cleared","logger":"MemoryStore"}
{"ts":"2024-12-19T21:38:16.962Z","level":"INFO","msg":"BlockManager stopped","logger":"BlockManager"}
{"ts":"2024-12-19T21:38:16.976Z","level":"INFO","msg":"BlockManagerMaster stopped","logger":"BlockManagerMaster"}
{"ts":"2024-12-19T21:38:16.982Z","level":"INFO","msg":"OutputCommitCoordinator stopped!","logger":"OutputCommitCoordinator$OutputCommitCoordinatorEndpoint"}
{"ts":"2024-12-19T21:38:16.988Z","level":"INFO","msg":"Successfully stopped SparkContext","logger":"SparkContext"}
{"ts":"2024-12-19T21:38:16.991Z","level":"INFO","msg":"Shutdown hook called","logger":"ShutdownHookManager"}
{"ts":"2024-12-19T21:38:16.994Z","level":"INFO","msg":"Deleting directory C:\\Users\\arnaud\\AppData\\Local\\Temp\\spark-e5ecac01-38d5-44ea-a16a-7467d396aa6d","context":{"path":"C:\\Users\\arnaud\\AppData\\Local\\Temp\\spark-e5ecac01-38d5-44ea-a16a-7467d396aa6d"},"logger":"ShutdownHookManager"}
{"ts":"2024-12-19T21:38:16.996Z","level":"INFO","msg":"Deleting directory C:\\Users\\arnaud\\AppData\\Local\\Temp\\spark-97671847-5c37-4905-9bf7-fc33ff488a2b","context":{"path":"C:\\Users\\arnaud\\AppData\\Local\\Temp\\spark-97671847-5c37-4905-9bf7-fc33ff488a2b"},"logger":"ShutdownHookManager"}
{"ts":"2024-12-19T21:38:16.998Z","level":"INFO","msg":"Deleting directory C:\\apps\\td5-spark-java\\artifacts\\spark-7e6446f5-95d7-4c4f-85f9-ce1de8110955","context":{"path":"C:\\apps\\td5-spark-java\\artifacts\\spark-7e6446f5-95d7-4c4f-85f9-ce1de8110955"},"logger":"ShutdownHookManager"}
{noformat}


I don't think it is necessary to keep the logInfo() if merging the Pull-Request.

I don't think there are regression risk to close correctly the URLClassLoader at the end, because everything should be already stopped, and no new class would be defined after.