Bug: Using spark-sql -f sqlfile, some jobs failed with  "Stage cancelled because SparkContext was shut down"
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/24 12:43
Affects Version/s: 3.4.4

The task was run successfully using spark-sql -f sqlfile, but the spark-web ui showed some jobs failed with "Stage cancelled because SparkContext was shut down"
Is this as expected?  it should exit only after all stages are completed?
{code:java}
spark-sql --master local[2] --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=debug.properties" -f /home/hadoop/spark/saprk3/query95.sql 
{code}
query95.sql 
{code:java}
use autotest_hive;
SELECT count(distinct ws1.ws_order_number) as order_count,
               sum(ws1.ws_ext_ship_cost) as total_shipping_cost,
               sum(ws1.ws_net_profit) as total_net_profit
FROM web_sales ws1
JOIN customer_address ca ON (ws1.ws_ship_addr_sk = ca.ca_address_sk)
JOIN web_site s ON (ws1.ws_web_site_sk = s.web_site_sk)
JOIN date_dim d ON (ws1.ws_ship_date_sk = d.d_date_sk)
LEFT SEMI JOIN (SELECT ws2.ws_order_number as ws_order_number
FROM web_sales ws2 JOIN web_sales ws3
ON (ws2.ws_order_number = ws3.ws_order_number)
 WHERE ws2.ws_warehouse_sk <> ws3.ws_warehouse_sk) ws_wh1
ON (ws1.ws_order_number = ws_wh1.ws_order_number)
LEFT SEMI JOIN (SELECT wr_order_number
FROM web_returns wr
JOIN (SELECT ws4.ws_order_number as ws_order_number
FROM web_sales ws4 JOIN web_sales ws5
ON (ws4.ws_order_number = ws5.ws_order_number)
WHERE ws4.ws_warehouse_sk <> ws5.ws_warehouse_sk) ws_wh2
ON (wr.wr_order_number = ws_wh2.ws_order_number)) tmp1
ON (ws1.ws_order_number = tmp1.wr_order_number)
WHERE d.d_date between '2002-05-01' and '2002-06-30' and ca.ca_state = 'GA' and s.web_company_name = 'pri'; {code}
some log info
{code:java}
24/12/16 20:14:52 DEBUG SparkSQLCLIDriver: processCmd:
SELECT count(distinct ws1.ws_order_number) as order_count,
               sum(ws1.ws_ext_ship_cost) as total_shipping_cost,
               sum(ws1.ws_net_profit) as total_net_profit
FROM web_sales ws1
JOIN customer_address ca ON (ws1.ws_ship_addr_sk = ca.ca_address_sk)
JOIN web_site s ON (ws1.ws_web_site_sk = s.web_site_sk)
JOIN date_dim d ON (ws1.ws_ship_date_sk = d.d_date_sk)
LEFT SEMI JOIN (SELECT ws2.ws_order_number as ws_order_number
FROM web_sales ws2 JOIN web_sales ws3
ON (ws2.ws_order_number = ws3.ws_order_number)
 WHERE ws2.ws_warehouse_sk <> ws3.ws_warehouse_sk) ws_wh1
ON (ws1.ws_order_number = ws_wh1.ws_order_number)
LEFT SEMI JOIN (SELECT wr_order_number
FROM web_returns wr
JOIN (SELECT ws4.ws_order_number as ws_order_number
FROM web_sales ws4 JOIN web_sales ws5
ON (ws4.ws_order_number = ws5.ws_order_number)
WHERE ws4.ws_warehouse_sk <> ws5.ws_warehouse_sk) ws_wh2
ON (wr.wr_order_number = ws_wh2.ws_order_number)) tmp1
ON (ws1.ws_order_number = tmp1.wr_order_number)
WHERE d.d_date between '2002-05-01' and '2002-06-30' and ca.ca_state = 'GA' and s.web_company_name = 'pri' with ret:0
24/12/16 20:14:52 DEBUG SparkSQLCLIDriver: ignoreErrors:false
24/12/16 20:14:52 DEBUG SparkSQLCLIDriver: cli.processFile res:0
24/12/16 20:14:52 DEBUG SparkSQLEnv: Shutting down Spark SQL Environment
24/12/16 20:14:52 INFO SparkContext: SparkContext is stopping with exitCode 0.
........
24/12/16 20:14:52 INFO DAGScheduler: ShuffleMapStage 4 (processLine at CliDriver.java:336) failed in 80.209 s due to Stage cancelled because SparkContext was shut down
24/12/16 20:14:52 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 

{code}


{code:java}
SparkSQLCLIDriver 
      if (sessionState.fileName != null) {
        exit(cli.processFile(sessionState.fileName))
      } 
cli.processFile execution returns, but some stages are not fully completed, and the exit action is executed, causing some stages to be canceled
{code}

spark version 3.4.4 
hadoop version 2.7.2 
hive version 2.3.3