Repository: iterative/dvc

Issue:
checkpoint: write permission denied
Surely `checkpoint: true` should un-protect hardlink/symlinks?

```python
$ git clone https://github.com/iterative/dvc-checkpoints-mnist && cd dvc-checkpoints-mnist \
  && pip install -r requirements.txt && dvc exp run
...
Traceback (most recent call last):
...
  File "train.py", line 99, in main
    torch.save(model.state_dict(), "model.pt")
...
PermissionError: [Errno 13] Permission denied: 'model.pt'
$ ls -l model.pt | cut -d' ' -f1
-r--r--r--
```
```bash
$ dvc doctor
DVC version: 2.1.0 (conda)
---------------------------------
Platform: Python 3.8.10 on Linux-4.15.0-58-generic-x86_64-with-glibc2.10
Supports: http, https
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git
```


Discussion:
@casperdcl unable to reproduce on my machine. After you run into the issue, could you show

```
dvc config --list
cat dvc.lock
ls -la model.pt
ls -la .dvc/cache/1b/2634aaed71b79a7bed3a3c05f98278
```

please?
I don't have `1b/263...` but do have `b4/bc4...`

```bash
iterative/dvc-checkpoints-mnist (live)$ dvc config --list

core.analytics=false
cache.protected=true
cache.type=reflink,hardlink,symlink,copy


iterative/dvc-checkpoints-mnist (live)$ cat dvc.lock
schema: '2.0'
stages:
  train:
    cmd: python train.py
    deps:
    - path: train.py
      md5: 34803fd28d3381c7bf111f5ccf3ff46c
      size: 3075
    outs:
    - path: dvclive
      md5: edf232f92f2ddefc40b55a836b668380.dir
      size: 44
      nfiles: 1
    - path: model.pt
      md5: b4bc424827a3d542fb298f1a5f09a491
      size: 410743
iterative/dvc-checkpoints-mnist (live)$ ls -la model.pt
-r--r--r-- 2 casperdcl casperdcl 410743 Jun  4 01:28 model.pt
~/gits/iterative/dvc-checkpoints-mnist (live)$ ls -la .dvc/cache/*/*
-r--r--r-- 2 casperdcl casperdcl 410743 Jun  4 01:28 .dvc/cache/b4/bc424827a3d542fb298f1a5f09a491
-r--r--r-- 2 casperdcl casperdcl     44 Jun  4 01:28 .dvc/cache/cc/85455a03b55204018144d5fdacea96
-r--r--r-- 1 casperdcl casperdcl     67 Jun  4 01:28 .dvc/cache/ed/f232f92f2ddefc40b55a836b668380.dir

.dvc/cache/runs/b4:
total 12
drwxrwxr-x 3 casperdcl casperdcl 4096 Jun  4 01:28 .
drwxrwxr-x 3 casperdcl casperdcl 4096 Jun  4 01:28 ..
drwxrwxr-x 2 casperdcl casperdcl 4096 Jun  4 01:28 b4725c37aad9550aa87eeb077fda792550ece80df1bd06363d87b0075d66c41e

@casperdcl Sorry for the delay. Thank you! Can confirm the bug, looks like we are forgetting to unprotect the checkpoint somewhere. Here is a universal reproducer:

```
$ git clone https://github.com/iterative/dvc-checkpoints-mnist && cd dvc-checkpoints-mnist \
  && pip install -r requirements.txt && dvc config cache.type hardlink && dvc exp run
```


Looking into it...
Somewhat related: maybe `dvc doctor` should also output `dvc config --list`?
Ok, so when launching a run we correctly unprotect `model.pt`, but then when processing the first checkpoint we are committing in monitor https://github.com/iterative/dvc/blob/1335c7f5e4ae5a3553e5d231235cbfca55594489/dvc/stage/monitor.py#L77 , which by default tries to replace the artifact with a link to cache, so that's why we get a read-only hardlink there. So we just need to either not link or at least unprotect there.


> Somewhat related: maybe dvc doctor should also output dvc config --list?

That will dump out private options as well, so probably not worth it. `dvc doctor` tries to not expose any confidential information.




Resolving patch:
diff --git a/dvc/stage/monitor.py b/dvc/stage/monitor.py
--- a/dvc/stage/monitor.py
+++ b/dvc/stage/monitor.py
@@ -75,6 +75,7 @@ def __init__(
     def _run_callback(stage, callback_func):
         stage.save(allow_missing=True)
         stage.commit(allow_missing=True)
+        stage.unprotect_outs()
         logger.debug("Running checkpoint callback for stage '%s'", stage)
         callback_func()
 


Test patch:
diff --git a/tests/func/experiments/test_checkpoints.py b/tests/func/experiments/test_checkpoints.py
--- a/tests/func/experiments/test_checkpoints.py
+++ b/tests/func/experiments/test_checkpoints.py
@@ -7,9 +7,13 @@
 
 
 @pytest.mark.parametrize("workspace", [True, False])
+@pytest.mark.parametrize("links", ["reflink,copy", "hardlink,symlink"])
 def test_new_checkpoint(
-    tmp_dir, scm, dvc, checkpoint_stage, mocker, workspace
+    tmp_dir, scm, dvc, checkpoint_stage, mocker, workspace, links
 ):
+    with dvc.config.edit() as conf:
+        conf["cache"]["type"] = links
+
     new_mock = mocker.spy(dvc.experiments, "new")
     results = dvc.experiments.run(
         checkpoint_stage.addressing, params=["foo=2"], tmp_dir=not workspace