Repository: iterative/dvc

Issue:
Webdav connection fails under dvc 2.0.x
As long as I am using dvc 1.x I can push to the remote webdav storage.

But after an upgrade to dvc 2.0.4 I receive the following error:
`ERROR: failed to push data to the cloud - Unable to connect to WebDAV http:...`

webdav itself logs
`user <username>: authentication failure for "/...": Password Mismatch`

The user is send correctly, but the password does not seem to be picked up from config.local




Discussion:
@AnnKatrinBecker, could you please post the command that's failing with the verbose output (please add `--verbose` at the end). 

Also, please post the complete `dvc doctor` output. Thanks.
dvc doctor

```
DVC version: 2.0.4 (pip)
---------------------------------
Platform: Python 3.9.0 on Linux-5.4.0-66-generic-x86_64-with-glibc2.31
Supports: http, https, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: btrfs on /dev/nvme0n1p1
Caches: local
Remotes: webdav, webdav, webdav, webdav
Workspace directory: btrfs on /dev/nvme0n1p1
Repo: dvc, git
```


dvc push --verbose

```
2021-03-10 08:50:50,524 DEBUG: Check for update is enabled.
2021-03-10 08:50:50,649 DEBUG: Preparing to upload data to 'http://storage...'
2021-03-10 08:50:50,650 DEBUG: Preparing to collect status from http://storage...
2021-03-10 08:50:50,651 DEBUG: Collecting information from local cache...
2021-03-10 08:50:50,664 DEBUG: Collecting information from remote cache...                                                                                                                         
2021-03-10 08:50:50,664 DEBUG: Querying 1 hashes via object_exists
2021-03-10 08:50:50,678 ERROR: failed to push data to the cloud - Unable to connect to WebDAV http://storage....                                                         
------------------------------------------------------------
Traceback (most recent call last):
  File "/.../.venv/lib/python3.9/site-packages/dvc/command/data_sync.py", line 56, in run
    processed_files_count = self.repo.push(
  File "/.../.venv/lib/python3.9/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/.../.venv/lib/python3.9/site-packages/dvc/repo/push.py", line 41, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/.../.venv/lib/python3.9/site-packages/dvc/data_cloud.py", line 66, in push
    return remote.push(
  File "/.../.venv/lib/python3.9/site-packages/dvc/remote/base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "/.../.venv/lib/python3.9/site-packages/dvc/remote/base.py", line 466, in push
    ret = self._process(
  File "/.../.venv/lib/python3.9/site-packages/dvc/remote/base.py", line 323, in _process
    dir_status, file_status, dir_contents = self._status(
  File "/.../.venv/lib/python3.9/site-packages/dvc/remote/base.py", line 169, in _status
    remote_exists.update(
  File "/.../.venv/lib/python3.9/site-packages/dvc/remote/base.py", line 242, in _indexed_dir_hashes
    dir_exists.update(self.odb.list_hashes_exists(dir_md5s - dir_exists))
  File "/.../.venv/lib/python3.9/site-packages/dvc/objects/db/base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/home/akb/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 600, in result_iterator
    yield fs.pop().result()
  File "/home/akb/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/home/akb/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/akb/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/.../.venv/lib/python3.9/site-packages/dvc/objects/db/base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/mnt/data/ml_projekte/name_address_splitter/name_address_splitter_rawdata/.venv/lib/python3.9/site-packages/dvc/fs/webdav.py", line 147, in exists
    return self._client.check(path_info.path)
  File "/.../.venv/lib/python3.9/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/.../.venv/lib/python3.9/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/.../.venv/lib/python3.9/site-packages/dvc/fs/webdav.py", line 121, in _client
    raise WebDAVConnectionError(self.hostname)
dvc.fs.webdav.WebDAVConnectionError: Unable to connect to WebDAV http://storage...
------------------------------------------------------------
2021-03-10 08:50:50,683 DEBUG: Analytics is enabled.
2021-03-10 08:50:50,728 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp5bwbihg5']'
2021-03-10 08:50:50,730 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp5bwbihg5']
```
@AnnKatrinBecker, how are you adding the URL in the config? Does the URL need to be in https?
For example, my config looks something like following:
```
['remote "myremote"']
    url = webdavs://server.com/remote.php/dav/files/admin/dvc-cache/
```

Have you added it as `webdavs://` or `webdav://` instead of `http://` or `https://`? 
url is added in .dvc/config using `webdav://`

```
['remote "rawdata_remote"']
    url = webdav://storage...:8080/ml/rawdata
    user = username
```
@AnnKatrinBecker, does it need to be `webdavs`? Is it an https connection? Maybe that's the problem?

Also, could you please try sending a curl request and see if it works there?
```console
$ curl -X PROPFIND http://<user>:<password>@storage...:8080/ml/rawdata -v | xmllint -format -
```

should work via `webdav`
curl access is successful
if I downgrade back to dvc 1.11.16 dvc push works again

the password contains an @ might this lead to a resolving issue
@AnnKatrinBecker, it might not be because of the password. 

I am not sure what's going on. It'd be great if you could debug it (with `dvc push --pdb`). Thanks. 🙂 
If I uncomment the following lines in dvc/fs/webdav.py I can push again

```
if not client.check("/"):
            raise WebDAVConnectionError(self.hostname)
```
@AnnKatrinBecker, great find. So, your WebDAV server is not at the root of the URL, which is raising the issue. Our expectation here is wrong.
Indeed, looks like this was introduced in 2.0 on b87e76aa844926a15244066fac4032c66fba22d1.
I think we should be able to just remove that check. @skshetry Do you do something similar in webdav4?

Resolving patch:
diff --git a/dvc/fs/webdav.py b/dvc/fs/webdav.py
--- a/dvc/fs/webdav.py
+++ b/dvc/fs/webdav.py
@@ -117,7 +117,7 @@ def _client(self):
             )
 
         # Check whether connection is valid (root should always exist)
-        if not client.check("/"):
+        if not client.check(self.path_info.path):
             raise WebDAVConnectionError(self.hostname)
 
         return client
@@ -213,7 +213,7 @@ def remove(self, path_info):
     # Creates directories
     def makedirs(self, path_info):
         # Terminate recursion
-        if path_info.path == "/" or self.exists(path_info):
+        if path_info.path == self.path_info.path or self.exists(path_info):
             return
 
         # Recursively descent to root


Test patch: