Repository: skypilot-org/skypilot

Issue:
SLURM: squeue --only-job-state not supported in SLURM 20.x
## Description

SkyPilot's SLURM provisioner fails on clusters running SLURM 20.11.x because it uses the `--only-job-state` flag with `squeue`, which was not available in that version.

The job is submitted successfully via `sbatch`, but the subsequent `squeue` call to check job state fails immediately, causing SkyPilot to terminate the cluster.

## Environment

- SkyPilot version: `1.0.0.dev20260220` (nightly)
- SLURM version: `20.11.7`
- Cluster: Oxford ARC HTC

## Error log

```
squeue -h --only-job-state --jobs 7207577 -o "%T"
squeue: unrecognized option '--only-job-state'
Try "squeue --help" for more information
```

Full provisioner log:

```
Successfully submitted Slurm job 7207577 to partition short for cluster sky-7404-reece-4d69be2f with 1 nodes
Failed to provision 'sky-7404-reece' on Slurm (short).
sky.exceptions.CommandError: Command squeue -h --only-job-state --jobs 7207577 -o "%T" failed with return code 1.
Failed to get job state for job 7207577.
```

## Expected behavior

SkyPilot should fall back to a compatible `squeue` invocation (e.g. `squeue -h --jobs <id> -o "%T"`) when `--only-job-state` is not available, or detect the SLURM version and adjust accordingly.

## Steps to reproduce

1. Configure `~/.slurm/config` pointing to a SLURM 20.x cluster
2. Run `sky launch` with any task
3. Job submits but immediately fails on the `squeue` status check

Discussion hints:

Solution patch:
diff --git a/sky/adaptors/slurm.py b/sky/adaptors/slurm.py
index 5dec3b81121..e193fbe63bd 100644
--- a/sky/adaptors/slurm.py
+++ b/sky/adaptors/slurm.py
@@ -348,8 +348,13 @@ def get_job_state(self, job_id: str) -> Optional[str]:
         """
         # Use --only-job-state since we only need the job state.
         # This reduces the work required by slurmctld.
+        # Fall back to the command without --only-job-state for older
+        # Slurm versions (< 21.08) that don't support this flag.
         cmd = f'squeue -h --only-job-state --jobs {job_id} -o "%T"'
         rc, stdout, stderr = self._run_slurm_cmd(cmd)
+        if rc != 0 and 'unrecognized option' in stderr:
+            cmd = f'squeue -h --jobs {job_id} -o "%T"'
+            rc, stdout, stderr = self._run_slurm_cmd(cmd)
         subprocess_utils.handle_returncode(
             rc,
             cmd,


Test patch:
diff --git a/tests/unit_tests/test_sky/adaptors/test_slurm_adaptor.py b/tests/unit_tests/test_sky/adaptors/test_slurm_adaptor.py
index dcf1ab2be9d..43f0add2efa 100644
--- a/tests/unit_tests/test_sky/adaptors/test_slurm_adaptor.py
+++ b/tests/unit_tests/test_sky/adaptors/test_slurm_adaptor.py
@@ -137,6 +137,68 @@ def test_returns_false_on_command_failure(self):
             assert client.check_job_has_nodes('12345') is False
 
 
+class TestGetJobState:
+    """Test SlurmClient.get_job_state()."""
+
+    def test_get_job_state_with_only_job_state_flag(self):
+        """Test that get_job_state uses --only-job-state when supported."""
+        client = slurm.SlurmClient(
+            ssh_host='localhost',
+            ssh_port=22,
+            ssh_user='root',
+            ssh_key=None,
+        )
+
+        with mock.patch.object(client._runner, 'run') as mock_run:
+            mock_run.return_value = (0, 'RUNNING\n', '')
+            result = client.get_job_state('12345')
+            mock_run.assert_called_once_with(
+                'squeue -h --only-job-state --jobs 12345 -o "%T"',
+                require_outputs=True,
+                separate_stderr=True,
+                stream_logs=False,
+            )
+            assert result == 'RUNNING'
+
+    def test_get_job_state_falls_back_on_old_slurm(self):
+        """Test fallback when --only-job-state is not supported (Slurm < 21.08)."""
+        client = slurm.SlurmClient(
+            ssh_host='localhost',
+            ssh_port=22,
+            ssh_user='root',
+            ssh_key=None,
+        )
+
+        with mock.patch.object(client._runner, 'run') as mock_run:
+            mock_run.side_effect = [
+                (1, '', "squeue: unrecognized option '--only-job-state'"),
+                (0, 'PENDING\n', ''),
+            ]
+            result = client.get_job_state('12345')
+            assert mock_run.call_count == 2
+            mock_run.assert_called_with(
+                'squeue -h --jobs 12345 -o "%T"',
+                require_outputs=True,
+                separate_stderr=True,
+                stream_logs=False,
+            )
+            assert result == 'PENDING'
+
+    def test_get_job_state_returns_none_for_empty_output(self):
+        """Test returns None when job is not found."""
+        client = slurm.SlurmClient(
+            ssh_host='localhost',
+            ssh_port=22,
+            ssh_user='root',
+            ssh_key=None,
+        )
+
+        with mock.patch.object(client._runner, 'run') as mock_run:
+            mock_run.return_value = (0, '', '')
+            result = client.get_job_state('99999')
+            assert result is None
+
+
 class TestGetJobsStateByName:
     """Test SlurmClient.get_jobs_state_by_name()."""