Repository: iterative/dvc

Issue:
dvc import-url fails for folders in azure remote.
## Bug Report

I've been trying to import a url from a remote azure container using `dvc import-url`. It seems to work fine for single files, but fails for folders. Here is the output, from setting up dvc, importing the single file and finally failing to import a folder.

### Please provide information about your setup

**Output of `dvc version`:**

```console
DVC version: 1.8.1 (brew)
---------------------------------
Platform: Python 3.8.5 on macOS-10.15.6-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Repo: dvc (no_scm)
```

**Additional Information (if any):**

> Context: https://discord.com/channels/485586884165107732/485596304961962003/764182584883937312

```console
~$ dvc remote add dvc_file_import_test azure://dvc-file-import-test/

~$ dvc remote modify --local dvc_file_import_test connection_string $remote_connection_string -v
2020-10-09 10:32:15,453 DEBUG: Writing '/Users/julien/dvc_file_import_test/.dvc/config.local'.
...

~$ dvc import-url remote://dvc_file_import_test/test_file.txt -v
2020-10-09 10:33:20,255 DEBUG: Check for update is enabled.
2020-10-09 10:33:20,257 DEBUG: fetched: [(3,)]
2020-10-09 10:33:20,582 DEBUG: Removing output 'test_file.txt' of stage: 'test_file.txt.dvc'.
Importing 'remote://dvc_file_import_test/test_file.txt' -> 'test_file.txt'
2020-10-09 10:33:20,583 DEBUG: Computed stage: 'test_file.txt.dvc' md5: 'ac01a9a397f504d0298fe8c7e1aef689'
2020-10-09 10:33:20,584 DEBUG: 'md5' of stage: 'test_file.txt.dvc' changed.
2020-10-09 10:33:20,584 DEBUG: URL azure://dvc-file-import-test/
2020-10-09 10:33:20,584 DEBUG: Using connection string 'remote_connection_string'
2020-10-09 10:33:20,592 DEBUG: Container name dvc-file-import-test
2020-10-09 10:33:20,908 DEBUG: Downloading 'azure://dvc-file-import-test/test_file.txt' to 'test_file.txt'
2020-10-09 10:33:21,065 DEBUG: Path '/Users/julien/dvc_file_import_test/test_file.txt' inode '8580528'
2020-10-09 10:33:21,066 DEBUG: fetched: []
2020-10-09 10:33:21,067 DEBUG: Path 'test_file.txt' inode '8580528'
2020-10-09 10:33:21,067 DEBUG: fetched: []
2020-10-09 10:33:21,067 DEBUG: {'test_file.txt': 'modified'}
2020-10-09 10:33:21,068 DEBUG: Path '/Users/julien/dvc_file_import_test/test_file.txt' inode '8580528'
2020-10-09 10:33:21,068 DEBUG: fetched: [('1602264800961265920', '2', 'b026324c6904b2a9cb4b88d6d61c81d1', '1602264801067587072')]
2020-10-09 10:33:21,068 DEBUG: Computed stage: 'test_file.txt.dvc' md5: 'e81fc216a3f58038ede5b08d93fdedf0'
2020-10-09 10:33:21,069 DEBUG: Saving 'test_file.txt' to '.dvc/cache/b0/26324c6904b2a9cb4b88d6d61c81d1'.
2020-10-09 10:33:21,070 DEBUG: Assuming '/Users/julien/dvc_file_import_test/.dvc/cache/b0/26324c6904b2a9cb4b88d6d61c81d1' is unchanged since it is read-only
2020-10-09 10:33:21,071 DEBUG: Created 'reflink': .dvc/cache/.cache_type_test_file -> .3GdfawJ8jYYwCRaeDNTSqh
2020-10-09 10:33:21,071 DEBUG: Removing '/Users/julien/dvc_file_import_test/.3GdfawJ8jYYwCRaeDNTSqh'
2020-10-09 10:33:21,071 DEBUG: Removing '/Users/julien/dvc_file_import_test/.dvc/cache/.cache_type_test_file'
2020-10-09 10:33:21,072 DEBUG: Removing '/Users/julien/dvc_file_import_test/test_file.txt'
2020-10-09 10:33:21,073 DEBUG: Created 'reflink': .dvc/cache/b0/26324c6904b2a9cb4b88d6d61c81d1 -> test_file.txt
2020-10-09 10:33:21,073 DEBUG: Path 'test_file.txt' inode '8580531'
2020-10-09 10:33:21,073 DEBUG: Path 'test_file.txt' inode '8580531'
2020-10-09 10:33:21,073 DEBUG: fetched: []
2020-10-09 10:33:21,073 DEBUG: Path '.dvc/cache/b0/26324c6904b2a9cb4b88d6d61c81d1' inode '8580365'
2020-10-09 10:33:21,074 DEBUG: fetched: [('1602264561767203584', '2', 'b026324c6904b2a9cb4b88d6d61c81d1', '1602264561886681088')]
2020-10-09 10:33:21,075 DEBUG: Saving information to 'test_file.txt.dvc'.
2020-10-09 10:33:21,080 DEBUG: fetched: [(2,)]
2020-10-09 10:33:21,083 DEBUG: Analytics is enabled.
2020-10-09 10:33:21,263 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/v4/m3drx4jj7dgcg94kgkjwh6zc0000gp/T/tmp6ctys6ja']'
2020-10-09 10:33:21,265 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/v4/m3drx4jj7dgcg94kgkjwh6zc0000gp/T/tmp6ctys6ja']'

~$ dvc import-url remote://dvc_file_import_test/test_folder/ -v
2020-10-09 10:35:45,312 DEBUG: Check for update is enabled.
2020-10-09 10:35:45,314 DEBUG: fetched: [(3,)]
2020-10-09 10:35:45,634 DEBUG: Removing output 'test_folder' of stage: 'test_folder.dvc'.
Importing 'remote://dvc_file_import_test/test_folder/' -> 'test_folder'
2020-10-09 10:35:45,635 DEBUG: Computed stage: 'test_folder.dvc' md5: 'e60962bd5c5965cfb3b297a753187e5b'
2020-10-09 10:35:45,636 DEBUG: 'md5' of stage: 'test_folder.dvc' changed.
2020-10-09 10:35:45,636 DEBUG: URL azure://dvc-file-import-test/
2020-10-09 10:35:45,636 DEBUG: Using connection string 'remote_connection_string'
2020-10-09 10:35:45,643 DEBUG: Container name dvc-file-import-test
2020-10-09 10:35:45,858 DEBUG: fetched: [(4,)]
2020-10-09 10:35:45,859 ERROR: failed to import remote://dvc_file_import_test/test_folder/. You could also try downloading it manually, and adding it with `dvc add`. - dependency 'remote://dvc_file_import_test/test_folder/' does not exist
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/command/imp_url.py", line 14, in run
    self.repo.imp_url(
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/repo/__init__.py", line 51, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/repo/imp_url.py", line 54, in imp_url
    stage.run()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/__init__.py", line 429, in run
    sync_import(self, dry, force)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/imports.py", line 29, in sync_import
    stage.save_deps()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/__init__.py", line 392, in save_deps
    dep.save()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/output/base.py", line 254, in save
    raise self.DoesNotExistError(self)
dvc.dependency.base.DependencyDoesNotExistError: dependency 'remote://dvc_file_import_test/test_folder/' does not exist
------------------------------------------------------------
2020-10-09 10:35:45,870 DEBUG: Analytics is enabled.
2020-10-09 10:35:46,049 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/v4/m3drx4jj7dgcg94kgkjwh6zc0000gp/T/tmp3xozxwul']'
2020-10-09 10:35:46,051 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/v4/m3drx4jj7dgcg94kgkjwh6zc0000gp/T/tmp3xozxwul']'
```

Trying without a trailing `/` as suggested by @jorgeorpinel:

```console
~$ dvc import-url remote://dvc_file_import_test/test_folder -v
2020-10-09 10:57:29,184 DEBUG: Check for update is enabled.
2020-10-09 10:57:29,186 DEBUG: fetched: [(3,)]
2020-10-09 10:57:29,536 DEBUG: Removing output 'test_folder' of stage: 'test_folder.dvc'.
Importing 'remote://dvc_file_import_test/test_folder' -> 'test_folder'
2020-10-09 10:57:29,537 DEBUG: Computed stage: 'test_folder.dvc' md5: 'ead4610f755d1be0125823348c9bd4ae'
2020-10-09 10:57:29,537 DEBUG: 'md5' of stage: 'test_folder.dvc' changed.
2020-10-09 10:57:29,537 DEBUG: URL azure://dvc-file-import-test/
2020-10-09 10:57:29,537 DEBUG: Using connection string 'remote_connection_string'
2020-10-09 10:57:29,544 DEBUG: Container name dvc-file-import-test
2020-10-09 10:57:29,784 DEBUG: fetched: [(4,)]
2020-10-09 10:57:29,785 ERROR: failed to import remote://dvc_file_import_test/test_folder. You could also try downloading it manually, and adding it with `dvc add`. - dependency 'remote://dvc_file_import_test/test_folder' does not exist
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/command/imp_url.py", line 14, in run
    self.repo.imp_url(
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/repo/__init__.py", line 51, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/repo/imp_url.py", line 54, in imp_url
    stage.run()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/__init__.py", line 429, in run
    sync_import(self, dry, force)
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/imports.py", line 29, in sync_import
    stage.save_deps()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/stage/__init__.py", line 392, in save_deps
    dep.save()
  File "/usr/local/Cellar/dvc/1.8.1/libexec/lib/python3.8/site-packages/dvc/output/base.py", line 254, in save
    raise self.DoesNotExistError(self)
dvc.dependency.base.DependencyDoesNotExistError: dependency 'remote://dvc_file_import_test/test_folder' does not exist
------------------------------------------------------------
2020-10-09 10:57:29,790 DEBUG: Analytics is enabled.
2020-10-09 10:57:29,966 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/v4/m3drx4jj7dgcg94kgkjwh6zc0000gp/T/tmp64qzqzwm']'
2020-10-09 10:57:29,968 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/v4/m3drx4jj7dgcg94kgkjwh6zc0000gp/T/tmp64qzqzwm']'
```

Confirming that the files are in the remote:

<details>

```console
~$ az storage blob list --account-name remote_storage_account --container dvc-file-import-test
There are no credentials provided in your command and environment, we will query for the account key inside your storage account.
Please provide --connection-string, --account-key or --sas-token as credentials, or use `--auth-mode login` if you have required RBAC roles in your command. For more information about RBAC roles in storage, visit https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli.
Setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information.
[
  {
    "container": "dvc-file-import-test",
    "content": "",
    "deleted": null,
    "encryptedMetadata": null,
    "encryptionKeySha256": null,
    "encryptionScope": null,
    "isAppendBlobSealed": null,
    "isCurrentVersion": null,
    "metadata": {},
    "name": "test_file.txt", ----------------------------------------> This is the test file.
    "objectReplicationDestinationPolicy": null,
    "objectReplicationSourceProperties": [],
    "properties": {
      "appendBlobCommittedBlockCount": null,
      "blobTier": "Hot",
      "blobTierChangeTime": null,
      "blobTierInferred": true,
      "blobType": "BlockBlob",
      "contentLength": 2,
      "contentRange": null,
      "contentSettings": {
        "cacheControl": null,
        "contentDisposition": null,
        "contentEncoding": null,
        "contentLanguage": null,
        "contentMd5": "sCYyTGkEsqnLS4jW1hyB0Q==",
        "contentType": "text/plain"
      },
      "copy": {
        "completionTime": null,
        "destinationSnapshot": null,
        "id": null,
        "incrementalCopy": null,
        "progress": null,
        "source": null,
        "status": null,
        "statusDescription": null
      },
      "creationTime": "2020-10-09T17:28:37+00:00",
      "deletedTime": null,
      "etag": "0x8D86C78C1FE5BF7",
      "lastModified": "2020-10-09T17:28:37+00:00",
      "lease": {
        "duration": null,
        "state": "available",
        "status": "unlocked"
      },
      "pageBlobSequenceNumber": null,
      "pageRanges": null,
      "rehydrationStatus": null,
      "remainingRetentionDays": null,
      "serverEncrypted": true
    },
    "rehydratePriority": null,
    "requestServerEncrypted": null,
    "snapshot": null,
    "tagCount": null,
    "tags": null,
    "versionId": null
  },
  {
    "container": "dvc-file-import-test",
    "content": "",
    "deleted": null,
    "encryptedMetadata": null,
    "encryptionKeySha256": null,
    "encryptionScope": null,
    "isAppendBlobSealed": null,
    "isCurrentVersion": null,
    "metadata": {},
    "name": "test_folder/test.txt", -------------------------------------> This is a file in the test folder
    "objectReplicationDestinationPolicy": null,
    "objectReplicationSourceProperties": [],
    "properties": {
      "appendBlobCommittedBlockCount": null,
      "blobTier": "Hot",
      "blobTierChangeTime": null,
      "blobTierInferred": true,
      "blobType": "BlockBlob",
      "contentLength": 2,
      "contentRange": null,
      "contentSettings": {
        "cacheControl": null,
        "contentDisposition": null,
        "contentEncoding": null,
        "contentLanguage": null,
        "contentMd5": "HcyiM1UnIFbwT+i/IO384A==",
        "contentType": "text/plain"
      },
      "copy": {
        "completionTime": null,
        "destinationSnapshot": null,
        "id": null,
        "incrementalCopy": null,
        "progress": null,
        "source": null,
        "status": null,
        "statusDescription": null
      },
      "creationTime": "2020-10-09T17:28:23+00:00",
      "deletedTime": null,
      "etag": "0x8D86C78B942233A",
      "lastModified": "2020-10-09T17:28:23+00:00",
      "lease": {
        "duration": null,
        "state": "available",
        "status": "unlocked"
      },
      "pageBlobSequenceNumber": null,
      "pageRanges": null,
      "rehydrationStatus": null,
      "remainingRetentionDays": null,
      "serverEncrypted": true
    },
    "rehydratePriority": null,
    "requestServerEncrypted": null,
    "snapshot": null,
    "tagCount": null,
    "tags": null,
    "versionId": null
  },
  {
    "container": "dvc-file-import-test",
    "content": "",
    "deleted": null,
    "encryptedMetadata": null,
    "encryptionKeySha256": null,
    "encryptionScope": null,
    "isAppendBlobSealed": null,
    "isCurrentVersion": null,
    "metadata": {},
    "name": "test_folder/test_2.txt", ----------------------------------> This is the second file in the test folder
    "objectReplicationDestinationPolicy": null,
    "objectReplicationSourceProperties": [],
    "properties": {
      "appendBlobCommittedBlockCount": null,
      "blobTier": "Hot",
      "blobTierChangeTime": null,
      "blobTierInferred": true,
      "blobType": "BlockBlob",
      "contentLength": 2,
      "contentRange": null,
      "contentSettings": {
        "cacheControl": null,
        "contentDisposition": null,
        "contentEncoding": null,
        "contentLanguage": null,
        "contentMd5": "sCYyTGkEsqnLS4jW1hyB0Q==",
        "contentType": "text/plain"
      },
      "copy": {
        "completionTime": null,
        "destinationSnapshot": null,
        "id": null,
        "incrementalCopy": null,
        "progress": null,
        "source": null,
        "status": null,
        "statusDescription": null
      },
      "creationTime": "2020-10-09T17:28:23+00:00",
      "deletedTime": null,
      "etag": "0x8D86C78B93416F5",
      "lastModified": "2020-10-09T17:28:23+00:00",
      "lease": {
        "duration": null,
        "state": "available",
        "status": "unlocked"
      },
      "pageBlobSequenceNumber": null,
      "pageRanges": null,
      "rehydrationStatus": null,
      "remainingRetentionDays": null,
      "serverEncrypted": true
    },
    "rehydratePriority": null,
    "requestServerEncrypted": null,
    "snapshot": null,
    "tagCount": null,
    "tags": null,
    "versionId": null
  }
]

</details>


Discussion:
I suspect it's a bug and hopefully only with `remote://` URL aliases. Please evaluate, engineering team 🙂 
For the record, [original discussion](https://discord.com/channels/485586884165107732/485596304961962003/763833696171458600)
Most likely we are lacking some logic for directories in `AzureTree.exists`, need to take a closer look.
This is finally fixed on upstream adlfs, so we only need to add a test and bump the adlfs requirements. See https://github.com/dask/adlfs/pull/237

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -98,7 +98,7 @@ def run(self):
 gs = ["gcsfs==2021.6.1"]
 gdrive = ["pydrive2>=1.8.1", "six >= 1.13.0"]
 s3 = ["s3fs==2021.6.1", "aiobotocore[boto3]==1.3.0"]
-azure = ["adlfs==0.7.1", "azure-identity>=1.4.0", "knack"]
+azure = ["adlfs==2021.7.0", "azure-identity>=1.4.0", "knack"]
 # https://github.com/Legrandin/pycryptodome/issues/465
 oss = ["oss2==2.6.1", "pycryptodome>=3.10"]
 ssh = ["paramiko[invoke]>=2.7.0"]


Test patch:
diff --git a/tests/docker-compose.yml b/tests/docker-compose.yml
--- a/tests/docker-compose.yml
+++ b/tests/docker-compose.yml
@@ -2,7 +2,7 @@
 version: '3.2'
 services:
   azurite:
-    image: mcr.microsoft.com/azure-storage/azurite:3.10.0
+    image: mcr.microsoft.com/azure-storage/azurite:3.13.1
     command: azurite -L -l /data --blobHost 0.0.0.0 --queueHost 0.0.0.0
     ports:
       - "10000"
diff --git a/tests/func/test_import_url.py b/tests/func/test_import_url.py
--- a/tests/func/test_import_url.py
+++ b/tests/func/test_import_url.py
@@ -161,6 +161,12 @@ def test_import_url(tmp_dir, dvc, workspace):
             "2aa17f8daa26996b3f7a4cf8888ac9ac",
             "ec602a6ba97b2dd07bd6d2cd89674a60.dir",
         ),
+        (
+            pytest.lazy_fixture("azure"),
+            None,  # ETags for azure are not consistent with the actual
+            None,  # content of the object, so they will change every time
+            # we create new objects. We'll skip stage content check for Azure.
+        ),
         (
             pytest.lazy_fixture("hdfs"),
             "ec0943f83357f702033c98e70b853c8c",
@@ -192,23 +198,24 @@ def test_import_url_dir(tmp_dir, dvc, workspace, stage_md5, dir_md5):
     assert list(os.listdir(tmp_dir / "dir" / "subdir")) == ["subfile"]
     assert (tmp_dir / "dir" / "subdir" / "subfile").read_text() == "subfile"
 
-    assert (tmp_dir / "dir.dvc").read_text() == (
-        f"md5: {stage_md5}\n"
-        "frozen: true\n"
-        "deps:\n"
-        f"- md5: {dir_md5}\n"
-        "  size: 11\n"
-        "  nfiles: 2\n"
-        "  path: remote://workspace/dir\n"
-        "outs:\n"
-        "- md5: b6dcab6ccd17ca0a8bf4a215a37d14cc.dir\n"
-        "  size: 11\n"
-        "  nfiles: 2\n"
-        "  path: dir\n"
-    )
-
     assert dvc.status() == {}
 
+    if stage_md5 is not None and dir_md5 is not None:
+        assert (tmp_dir / "dir.dvc").read_text() == (
+            f"md5: {stage_md5}\n"
+            "frozen: true\n"
+            "deps:\n"
+            f"- md5: {dir_md5}\n"
+            "  size: 11\n"
+            "  nfiles: 2\n"
+            "  path: remote://workspace/dir\n"
+            "outs:\n"
+            "- md5: b6dcab6ccd17ca0a8bf4a215a37d14cc.dir\n"
+            "  size: 11\n"
+            "  nfiles: 2\n"
+            "  path: dir\n"
+        )
+
 
 def test_import_url_preserve_meta(tmp_dir, dvc):
     text = textwrap.dedent(