Repository: iterative/dvc

Issue:
dvc pull: ERROR: unexpected error - [Errno 1] Operation not permitted: 
# Bug Report
ERROR: unexpected error - [Errno 1] Operation not permitted: <model_path>

## Description

With versions 2.0.0 and 2.0.1, executing the dvc pull command gives the above error

### Reproduce

The RUN command of the dockerfile is as follows.
```
RUN groupadd --system --gid 500 ${GROUP} && \
    useradd --system --create-home --home-dir /home/${USER} --no-log-init --gid 500 --uid 500 ${USER} && \
    apt update && \
    apt install -y git wget && \
    wget https://dvc.org/deb/dvc.list -O /etc/apt/sources.list.d/dvc.list && \
    apt update && \
    apt install -y dvc && \
    apt purge -y wget && \
    apt autoremove -y && \
    apt clean && \
    rm -rf /var/lib/apt/lists/*
```

### Expected

The models should be pulled successfully.

### Environment information

Docker - Image built from `ubuntu:20.04`

**Output of `dvc version`:**

```console
$ dvc version
2.0.1
```

**Additional Information (if any):**

When a previous DVC version such as `1.11.15` is specified in the docker-file as `apt install -y dvc=1.11.15`, the models are pulled successfully without the above error.


Discussion:
@DuneeshaFernando can you submit the full output of the `dvc pull` command with `--verbose`
> @DuneeshaFernando can you submit the full output of the `dvc pull` command with `--verbose`

```
2021-03-05 13:32:12,246 DEBUG: Check for update is enabled.
2021-03-05 13:32:12,355 DEBUG: Trying to spawn '['daemon', '-q', 'updater']'
2021-03-05 13:32:12,356 DEBUG: Spawned '['daemon', '-q', 'updater']'
[31] Failed to execute script __main__
2021-03-05 13:32:16,355 DEBUG: Preparing to download data from 'azure://models'
2021-03-05 13:32:16,355 DEBUG: Preparing to collect status from azure://models
2021-03-05 13:32:16,356 DEBUG: Collecting information from local cache...
2021-03-05 13:32:16,356 DEBUG: Collecting information from remote cache...
2021-03-05 13:32:16,357 DEBUG: Matched '0' indexed hashes
2021-03-05 13:32:18,270 DEBUG: Estimated remote size: 4096 files
2021-03-05 13:32:18,271 DEBUG: Large remote ('4' hashes < '4.096' traverse weight), using object_exists for remaining hashes
2021-03-05 13:32:18,272 DEBUG: Querying 4 hashes via object_exists
2021-03-05 13:32:20,118 DEBUG: Downloading 'azure://models/b3/98361ddc51495b705ff33579a71e14' to '.dvc/cache/b3/98361ddc51495b705ff33579a71e14'
2021-03-05 13:32:20,139 DEBUG: Downloading 'azure://models/30/2f6aca49d6c4049973532926835730' to '.dvc/cache/30/2f6aca49d6c4049973532926835730'
2021-03-05 13:32:20,159 DEBUG: Downloading 'azure://models/c8/01ea2f11afdf062cc3962625b31877' to '.dvc/cache/c8/01ea2f11afdf062cc3962625b31877'
2021-03-05 13:32:20,161 DEBUG: Downloading 'azure://models/a9/8c552376d7431bc4a3245e85119fb0' to '.dvc/cache/a9/8c552376d7431bc4a3245e85119fb0'
2021-03-05 13:32:33,049 DEBUG: state save (8376853, 1614951142286609152, 373512) b398361ddc51495b705ff33579a71e14
2021-03-05 13:32:33,054 DEBUG: state save (8376855, 1614951142396609024, 373576) 302f6aca49d6c4049973532926835730
2021-03-05 13:32:33,056 DEBUG: state save (8376858, 1614951153046609152, 12035657) c801ea2f11afdf062cc3962625b31877
2021-03-05 13:32:33,056 DEBUG: state save (8376859, 1614951142454609152, 373864) a98c552376d7431bc4a3245e85119fb0
2021-03-05 13:32:33,063 DEBUG: 'models/levelShiftsModel' doesn't exist.
2021-03-05 13:32:33,064 DEBUG: Checking out 'models/levelShiftsModel' with cache 'object md5: 302f6aca49d6c4049973532926835730'.
2021-03-05 13:32:33,144 ERROR: unexpected error - [Errno 1] Operation not permitted: '/home/python-ai/******-ai-anomaly-detector/models'
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/main.py", line 50, in main
  File "dvc/command/data_sync.py", line 40, in run
  File "dvc/repo/__init__.py", line 49, in wrapper
  File "dvc/repo/pull.py", line 44, in pull
  File "dvc/repo/__init__.py", line 49, in wrapper
  File "dvc/repo/checkout.py", line 115, in checkout
  File "site-packages/funcy/decorators.py", line 39, in wrapper
  File "dvc/stage/decorators.py", line 36, in rwlocked
  File "site-packages/funcy/decorators.py", line 60, in __call__
  File "dvc/stage/__init__.py", line 537, in checkout
  File "dvc/stage/__init__.py", line 546, in _checkout
  File "dvc/output/base.py", line 431, in checkout
  File "dvc/checkout.py", line 301, in checkout
  File "dvc/checkout.py", line 237, in _checkout
  File "dvc/checkout.py", line 167, in _checkout_file
  File "dvc/checkout.py", line 121, in _link
  File "dvc/objects/db/local.py", line 56, in makedirs
  File "dvc/utils/fs.py", line 183, in makedirs
PermissionError: [Errno 1] Operation not permitted: '/home/python-ai/******-ai-anomaly-detector/models'
------------------------------------------------------------
2021-03-05 13:32:35,059 DEBUG: Version info for developers:
DVC version: 2.0.1 (deb)
---------------------------------
Platform: Python 3.7.10 on Linux-4.19.114-x86_64-with-debian-bullseye-sid
Supports: All remotes
Cache types: 
Cache directory: overlay on overlay
Caches: local
Remotes: azure
Workspace directory: overlay on overlay
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-03-05 13:32:35,139 DEBUG: Analytics is disabled.
```

@DuneeshaFernando Are you able to `mkdir /home/python-ai/******-ai-anomaly-detector/models` by hand?
> @DuneeshaFernando Are you able to `mkdir /home/python-ai/******-ai-anomaly-detector/models` by hand?

It doesn't allow me to create the directory by hand saying Permission denied
@DuneeshaFernando Then you need to figure out that first, as it seems like a misconfiguration on your end. E.g. maybe https://stackoverflow.com/questions/24288616/permission-denied-on-accessing-host-directory-in-docker ?
> @DuneeshaFernando Then you need to figure out that first, as it seems like a misconfiguration on your end. E.g. maybe https://stackoverflow.com/questions/24288616/permission-denied-on-accessing-host-directory-in-docker ?

Ack. But when we use DVC v1.11.15 the models are loaded fine. 
@DuneeshaFernando Btw, could you show `dvc config --list`, please? On another look, the original error occurs on `chmod` in makedirs, so I suppose you are using `cache.shared=group` config option?
> dvc config --list

core.analytics=false
remote.modelstorage.url=azure://models
remote.modelstorage.connection_string=`credential`
core.remote=modelstorage
@DuneeshaFernando Thanks! So in 1.x we we didn't pass mode to makedirs() at all, but in 2.x we've started doing so, which results in this error. We might consider ignoring that error, since it is not fatal.

Regarding the fact that you are not able to create a directory by hand, I suppose you were maybe trying to do so with a user that has no rights for that?




> @DuneeshaFernando Thanks! So in 1.x we we didn't pass mode to makedirs() at all, but in 2.x we've started doing so, which results in this error. We might consider ignoring that error, since it is not fatal.

Thanks. So can we expect a fix to be available in the next release?

> Regarding the fact that you are not able to create a directory by hand, I suppose you were maybe trying to do so with a user that has no rights for that?

Yes. I guess that is why it was the same for both 1.x and 2.x version.

@DuneeshaFernando Yes, will be sending a quick fix shortly. Stay tuned :) 

Resolving patch:
diff --git a/dvc/utils/fs.py b/dvc/utils/fs.py
--- a/dvc/utils/fs.py
+++ b/dvc/utils/fs.py
@@ -180,7 +180,10 @@ def makedirs(path, exist_ok=False, mode=None):
         if not exist_ok or not os.path.isdir(path):
             raise
 
-    os.chmod(path, mode)
+    try:
+        os.chmod(path, mode)
+    except OSError:
+        logger.trace("failed to chmod '%o' '%s'", mode, path, exc_info=True)
 
 
 def copyfile(src, dest, no_progress_bar=False, name=None):


Test patch: