Repository: apache/airflow

Issue:
Spark driver relaunches if "driverState" is not found in curl response due to transient network issue 
### Apache Airflow Provider(s)

apache-spark

### Versions of Apache Airflow Providers

apache-airflow-providers-apache-spark==2.0.1

### Apache Airflow version

2.1.4

### Operating System

Amazon Linux 2

### Deployment

Virtualenv installation

### Deployment details

_No response_

### What happened

In the file `airflow/providers/apache/spark/hooks/spark_submit.py`, the function [_process_spark_status_log](https://github.com/apache/airflow/blob/main/airflow/providers/apache/spark/hooks/spark_submit.py#L516-L535) iterates through a `curl` response to get the driver state of a `SparkSubmitOperator` task.

If there's a transient network issue and there is no valid response from the cluster (e.g. timeout, etc.), there is no "driverState" in the `curl` response, which makes the driver state "UNKNOWN".

That state [exits the loop](https://github.com/apache/airflow/blob/main/airflow/providers/apache/spark/hooks/spark_submit.py#L573) and then makes the task to go on a [retry](https://github.com/apache/airflow/blob/main/airflow/providers/apache/spark/hooks/spark_submit.py#L464-L467), while the original task is actually still in a "RUNNING" state.

### What you expected to happen

I would expect the task not to go on a retry while the original task is running. The function `_process_spark_status_log` should probably ensure the `curl` response is valid before changing the driver state, e.g. check that there is a "submissionId"  in the response as well, otherwise leave the state to `None` and continue with the polling loop. A valid response would be something like this:
```
curl http://spark-host:6066/v1/submissions/status/driver-FOO-BAR

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "RUNNING",
  "serverSparkVersion" : "2.4.6",
  "submissionId" : "driver-FOO-BAR,
  "success" : true,
  "workerHostPort" : "FOO:BAR",
  "workerId" : "worker-FOO-BAR-BAZ"
}
```
### How to reproduce

Use any DAG with a `SparkSubmitOperator` task on a Spark Standalone cluster where you can reset the network connection, or modify the `curl` command to return something other than the response above.

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:

Resolving patch:
diff --git a/airflow/providers/apache/spark/hooks/spark_submit.py b/airflow/providers/apache/spark/hooks/spark_submit.py
--- a/airflow/providers/apache/spark/hooks/spark_submit.py
+++ b/airflow/providers/apache/spark/hooks/spark_submit.py
@@ -520,10 +520,15 @@ def _process_spark_status_log(self, itr: Iterator[Any]) -> None:
         :param itr: An iterator which iterates over the input of the subprocess
         """
         driver_found = False
+        valid_response = False
         # Consume the iterator
         for line in itr:
             line = line.strip()
 
+            # A valid Spark status response should contain a submissionId
+            if "submissionId" in line:
+                valid_response = True
+
             # Check if the log line is about the driver status and extract the status.
             if "driverState" in line:
                 self._driver_status = line.split(' : ')[1].replace(',', '').replace('\"', '').strip()
@@ -531,7 +536,7 @@ def _process_spark_status_log(self, itr: Iterator[Any]) -> None:
 
             self.log.debug("spark driver status log: %s", line)
 
-        if not driver_found:
+        if valid_response and not driver_found:
             self._driver_status = "UNKNOWN"
 
     def _start_driver_status_tracking(self) -> None:


Test patch:
diff --git a/tests/providers/apache/spark/hooks/test_spark_submit.py b/tests/providers/apache/spark/hooks/test_spark_submit.py
--- a/tests/providers/apache/spark/hooks/test_spark_submit.py
+++ b/tests/providers/apache/spark/hooks/test_spark_submit.py
@@ -728,6 +728,21 @@ def test_process_spark_driver_status_log(self):
 
         assert hook._driver_status == 'RUNNING'
 
+    def test_process_spark_driver_status_log_bad_response(self):
+        # Given
+        hook = SparkSubmitHook(conn_id='spark_standalone_cluster')
+        log_lines = [
+            'curl: Failed to connect to http://spark-standalone-master:6066'
+            'This is an invalid Spark response',
+            'Timed out',
+        ]
+        # When
+        hook._process_spark_status_log(log_lines)
+
+        # Then
+
+        assert hook._driver_status is None
+
     @patch('airflow.providers.apache.spark.hooks.spark_submit.renew_from_kt')
     @patch('airflow.providers.apache.spark.hooks.spark_submit.subprocess.Popen')
     def test_yarn_process_on_kill(self, mock_popen, mock_renew_from_kt):