Repository: Lightning-AI/pytorch-lightning
Pull request: Allow cross-device local checkpoints with `fsspec>=2025.5.0`
## What does this PR do?

Cross-device transactions via fsspec (used for example in ModelCheckpoint) resulted in permission errors (#20270). The permission errors were caused by attempts to change file modes on different filesystem. This was fixed in fsspec 2025.5.0. This PR proposes to increase the minimum version to `fsspec>=2025.5.0`.

Fixes #20270

<details>
  <summary><b>Before submitting</b></summary>

- Was this **discussed/agreed** via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/Lightning-AI/lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your **PR does only one thing**, instead of bundling different changes together?
- [x] Did you make sure to **update the documentation** with your changes? (if necessary)
- [x] Did you write any **new necessary tests**? (not for typos and docs)
- [x] Did you verify new and **existing tests pass** locally with your changes?
- [x] Did you list all the **breaking changes** introduced by this pull request?
- [x] Did you **update the CHANGELOG**? (not for typos, docs, test updates, or minor internal changes/refactors)

<!-- In the CHANGELOG, separate each item in the unreleased section by a blank line to reduce collisions -->

</details>

## PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the [review guidelines](https://github.com/Lightning-AI/lightning/wiki/Review-guidelines). In short, see the following bullet-list:

<details>
  <summary>Reviewer checklist</summary>

- [ ] Is this pull request ready for review? (if not, please submit in draft mode)
- [ ] Check that all items from **Before submitting** are resolved
- [ ] Make sure the title is self-explanatory and the description concisely explains the PR
- [ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

</details>

<!--

Did you have fun?

Make sure you had fun coding 🙃

-->


<!-- readthedocs-preview pytorch-lightning start -->
----
📚 Documentation preview 📚: https://pytorch-lightning--20780.org.readthedocs.build/en/20780/

<!-- readthedocs-preview pytorch-lightning end -->

Problem statement:
_atomic_save with transaction cause "Invalid cross-device link" error
### Bug description

I Upgraded my pl verison from 2.0.4 to 2.4. The same training code will have the error below
```
2024-09-06 23:01:53 [on_validation_epoch_end] collecting matched instance from 64 ranks
2024-09-06 23:02:17 [on_validation_epoch_end] end
Traceback (most recent call last):
  File "/usr/lib/python3.10/shutil.py", line 816, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmp7ue72_bg' -> '/xxx/model/ckpt-epoch=000.ckpt'
```
When I check the corresponding code difference, I find the difference below
``` diff
@@ -65,8 +78,69 @@ def _atomic_save(checkpoint: Dict[str, Any], filepath: Union[str, Path]) -> None
             accepts.
         filepath: The path to which the checkpoint will be saved.
             This points to the file that the checkpoint will be stored in.
+
     """
     bytesbuffer = io.BytesIO()
+    log.debug(f"Saving checkpoint: {filepath}")
     torch.save(checkpoint, bytesbuffer)
-    with fsspec.open(filepath, "wb") as f:
+
+    # We use a transaction here to avoid file corruption if the save gets interrupted
+    fs, urlpath = fsspec.core.url_to_fs(str(filepath))
+    with fs.transaction, fs.open(urlpath, "wb") as f:
         f.write(bytesbuffer.getvalue())
```
**It seems that the using of transaction will cause the error when moving a file between different filesystem. Is there any solution?** 

### What version are you seeing the problem on?

v2.4

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```


### Environment

<details>
  <summary>Current environment</summary>

```
#- PyTorch Lightning Version (e.g., 2.4.0):
#- PyTorch Version (e.g., 2.4):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```

</details>


### More info

_No response_

Hints from issue discussion:
Same thing happening to me on 2.4.0 version, but it works fine on 2.3.1
> Same thing happening to me on 2.4.0 version, but it works fine on 2.3.1

Yes, same situation..
Same thing happening for me on version 2.4, but only occasionally, curious why we see this only sometimes?
Same thing happening for me on version 2.5.0.post0.
I wantted to save model checkpoints into disk mounted by [Cloud Storage FUSE](https://cloud.google.com/storage/docs/gcs-fuse?hl=en), but "Invalid cross-device link" error is occured.
Same, when saving to the "slow" storage of our slurm cluster. 

Same, when saving a checkpoint to a network drive.
The last working version is 2.3.3.

https://github.com/Lightning-AI/pytorch-lightning/blob/cf348673eda662cc2e9aa71a72a19b8774f85718/src/lightning/fabric/utilities/cloud_io.py#L65-L80
Same with 2.5.1. During the handling of the exception above, another exception occurs.

<details>

<summary>PermissionError: [Errno 1] Operation not permitted</summary>

```python
Traceback (most recent call last):
  File "/run.py", line 116, in <module>
    trainer.fit(model, data)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 561, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 48, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 599, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 1012, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 1056, in _run_stage
    self.fit_loop.run()
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/loops/fit_loop.py", line 217, in run
    self.on_advance_end()
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/loops/fit_loop.py", line 470, in on_advance_end
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 227, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 325, in on_train_epoch_end
    self._save_topk_checkpoint(trainer, monitor_candidates)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 385, in _save_topk_checkpoint
    self._save_monitor_checkpoint(trainer, monitor_candidates)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 705, in _save_monitor_checkpoint
    self._update_best_and_save(current, trainer, monitor_candidates)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 757, in _update_best_and_save
    self._save_checkpoint(trainer, filepath)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 390, in _save_checkpoint
    trainer.save_checkpoint(filepath, self.save_weights_only)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 1397, in save_checkpoint
    self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/strategies/strategy.py", line 491, in save_checkpoint
    self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
  File "/opt/conda/lib/python3.11/site-packages/lightning/fabric/plugins/io/torch_io.py", line 58, in save_checkpoint
    _atomic_save(checkpoint, path)
  File "/opt/conda/lib/python3.11/site-packages/lightning/fabric/utilities/cloud_io.py", line 89, in _atomic_save
    with fs.transaction, fs.open(urlpath, "wb") as f:
  File "/opt/conda/lib/python3.11/site-packages/fsspec/transaction.py", line 28, in __exit__
    self.complete(commit=exc_type is None)
  File "/opt/conda/lib/python3.11/site-packages/fsspec/transaction.py", line 42, in complete
    f.commit()
  File "/opt/conda/lib/python3.11/site-packages/fsspec/implementations/local.py", line 356, in commit
    shutil.move(self.temp, self.path)
  File "/opt/conda/lib/python3.11/shutil.py", line 873, in move
    copy_function(src, real_dst)
  File "/opt/conda/lib/python3.11/shutil.py", line 449, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/conda/lib/python3.11/shutil.py", line 380, in copystat
    lookup("utime")(dst, ns=(st.st_atime_ns, st.st_mtime_ns),
PermissionError: [Errno 1] Operation not permitted
```

</details>
Are you open to a PR? I'd like to work on it.

Would it make sense to do the following?
- `try ... except ...` and catch the permission error and fallback in case of this permission error.
- Always doing the fallback might possibly be a breaking change, but changing `TorchCheckpointIO.save_checkpoint`'s `storage_options` to include a `non_atomic_fallback` key that can be used by a new `class NoAtomicCheckpointIO(TorchCheckpointIO)` allows users to choose the fallback for themselves.
- Users can be pointed to resorting to another checkpointing class via the permission error, something like "Target filesystem doesn't support atomic writes. Use `Trainer(plugins=[NoAtomicCheckpointIO()])` to save checkpoints, accepting potential corruption risk."
I see there are discussions that might be related to this issue here https://github.com/Lightning-AI/pytorch-lightning/pull/20011 here https://github.com/fsspec/filesystem_spec/issues/1797. Looks like an issue with `fsspec` itself. @brandon-smith-xperi, do you have an update?
@siemdejong I haven't had a chance to dive into the problem deeply, but I think you have probably addressed the underlying issue here (https://github.com/fsspec/filesystem_spec/issues/1797), thanks!

Commit message: Merge branch 'bugfix/20270_invalid-cross-device-link' of https://github.com/siemdejong/pytorch-lightning into bugfix/20270_invalid-cross-device-link

Patch to review:
diff --git a/requirements/fabric/base.txt b/requirements/fabric/base.txt
index 2561b2324b772..c046c12970724 100644
--- a/requirements/fabric/base.txt
+++ b/requirements/fabric/base.txt
@@ -2,7 +2,7 @@
 #  in case you want to preserve/enforce restrictions on the latest compatible version, add "strict" as an in-line comment
 
 torch >=2.1.0, <2.8.0
-fsspec[http] >=2022.5.0, <2025.4.0
+fsspec[http] >=2025.5.0, <=2025.5.0
 packaging >=20.0, <=25.0
 typing-extensions >=4.4.0, <4.14.0
 lightning-utilities >=0.10.0, <0.15.0
diff --git a/requirements/pytorch/base.txt b/requirements/pytorch/base.txt
index 00889581f6407..6c9e0838d77c7 100644
--- a/requirements/pytorch/base.txt
+++ b/requirements/pytorch/base.txt
@@ -4,7 +4,7 @@
 torch >=2.1.0, <2.8.0
 tqdm >=4.57.0, <4.68.0
 PyYAML >=5.4, <6.1.0
-fsspec[http] >=2022.5.0, <2025.4.0
+fsspec[http] >=2025.5.0, <=2025.5.0
 torchmetrics >=0.7.0, <1.8.0
 packaging >=20.0, <=25.0
 typing-extensions >=4.4.0, <4.14.0
diff --git a/src/lightning/fabric/CHANGELOG.md b/src/lightning/fabric/CHANGELOG.md
index f67eec28deeeb..2368f885d345b 100644
--- a/src/lightning/fabric/CHANGELOG.md
+++ b/src/lightning/fabric/CHANGELOG.md
@@ -12,11 +12,21 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 -
 
 
+### Changed
+
+-
+
+
 ### Removed
 
 -
 
 
+### Fixed
+
+- Allow cross-device local checkpoints with `fsspec>=2025.5.0` ([#20780](https://github.com/Lightning-AI/pytorch-lightning/pull/20780))
+
+
 ---
 
 ## [2.5.1] - 2025-03-18
diff --git a/src/lightning/pytorch/CHANGELOG.md b/src/lightning/pytorch/CHANGELOG.md
index 0095367e9187a..3fe97dd7513c4 100644
--- a/src/lightning/pytorch/CHANGELOG.md
+++ b/src/lightning/pytorch/CHANGELOG.md
@@ -16,7 +16,6 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 -
 
-
 ### Removed
 
 -
@@ -26,6 +25,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 - Fixed logger_connector has edge case where step can be a float ([#20692](https://github.com/Lightning-AI/pytorch-lightning/issues/20692))
 
+- Allow cross-device local checkpoints with `fsspec>=2025.5.0` ([#20780](https://github.com/Lightning-AI/pytorch-lightning/pull/20780))
+
 
 ---
 
diff --git a/src/lightning/pytorch/callbacks/model_checkpoint.py b/src/lightning/pytorch/callbacks/model_checkpoint.py
index 85bfb65c0ea6e..d9bef1635715f 100644
--- a/src/lightning/pytorch/callbacks/model_checkpoint.py
+++ b/src/lightning/pytorch/callbacks/model_checkpoint.py
@@ -155,6 +155,10 @@ class ModelCheckpoint(Checkpoint):
         If the checkpoint's ``dirpath`` changed from what it was before while resuming the training,
         only ``best_model_path`` will be reloaded and a warning will be issued.
 
+        If you provide a ``filename`` on a mounted device where changing permissions is not allowed (causing ``chmod``
+        to raise a ``PermissionError``), the error is cached, the file's permissions remain unchanged, and the
+        checkpoint is still saved.
+
     Raises:
         MisconfigurationException:
             If ``save_top_k`` is smaller than ``-1``,
diff --git a/tests/tests_pytorch/trainer/connectors/test_checkpoint_connector.py b/tests/tests_pytorch/trainer/connectors/test_checkpoint_connector.py
index 722742a3ccae0..5baa55cbc3d21 100644
--- a/tests/tests_pytorch/trainer/connectors/test_checkpoint_connector.py
+++ b/tests/tests_pytorch/trainer/connectors/test_checkpoint_connector.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import errno
 import os
 from unittest import mock
 from unittest.mock import ANY, Mock
@@ -105,7 +106,7 @@ def test_hpc_max_ckpt_version(tmp_path):
     )
 
 
-def test_ckpt_for_fsspec():
+def test_ckpt_for_fsspec(tmpdir):
     """Test that the _CheckpointConnector is able to write to fsspec file systems."""
     model = BoringModel()
     # hardcoding dir since `tmp_path` can be windows path
@@ -118,6 +119,13 @@ def test_ckpt_for_fsspec():
     trainer.save_checkpoint("memory://test_ckpt_for_fsspec/hpc_ckpt_3.ckpt")
     trainer.save_checkpoint("memory://test_ckpt_for_fsspec/hpc_ckpt_33.ckpt")
 
+    # Simulate the behavior of fsspec when writing to a local file system but other device.
+    with (
+        mock.patch("os.rename", side_effect=OSError(errno.EXDEV, "Invalid cross-device link")),
+        mock.patch("os.chmod", side_effect=PermissionError("Operation not permitted")),
+    ):
+        trainer.save_checkpoint(tmpdir + "/test_ckpt_for_fsspec/hpc_ckpt_18.ckpt")
+
     assert trainer._checkpoint_connector._hpc_resume_path == "memory://test_ckpt_for_fsspec/hpc_ckpt_33.ckpt"
     assert (
         trainer._checkpoint_connector._CheckpointConnector__max_ckpt_version_in_folder("memory://test_ckpt_for_fsspec")


Code review comments:
diff_hunk: @@ -118,6 +119,13 @@ def test_ckpt_for_fsspec():
     trainer.save_checkpoint("memory://test_ckpt_for_fsspec/hpc_ckpt_3.ckpt")
     trainer.save_checkpoint("memory://test_ckpt_for_fsspec/hpc_ckpt_33.ckpt")
 
+    # Simulate the behavior of fsspec when writing to a local file system but other device.
+    with (
+        mock.patch("os.rename", side_effect=OSError(errno.EXDEV, "Invalid cross-device link")),
+        mock.patch("os.chmod", side_effect=PermissionError("Operation not permitted")),
+    ):
+        trainer.save_checkpoint(tmpdir + "/test_ckpt_for_fsspec/hpc_ckpt_18.ckpt")
path: tests/tests_pytorch/trainer/connectors/test_checkpoint_connector.py
text: @user1:
I think we shall add a new test covering this case, which will be conditioned by `fsspec[http] >=2025.5.0`, but there is no reason to be such aggressive and deny other past versions

@author:
Okay! Are you suggesting the following:
1. move the test to a new function
2. condition the test function on `fsspec>=2025.5.0`
3. verify that the test fails with `fsspec<2025.5.0`
4. do not force the install `fsspec>=2025.5.5`, keep it the current minimum version

Would it then be good to have a warning like
```
Operation not permitted: upgrade fsspec to enable cross-device local checkpoints: pip install "fsspec[http]>=2025.5.0"
```
shown when an attempt is made to write a cross-device local checkpoints if `fsspec<2025.5.0` before raising expected error if `fsspec>=2025.5.5` is not available?