Repository: iterative/dvc

Issue:
Issue with fetch from azure remote "no current event loop in thread"
# Bug Report

I'm getting this error
```
ERROR: failed to download 'azure://container/***' to '.dvc\cache\***' - There is no current event loop in thread 'ThreadPoolExecutor-127_0
...
ERROR: failed to download 'azure://container/***' to '.dvc\cache\***' - There is no current event loop in thread 'ThreadPoolExecutor-127_14
ERROR: failed to fetch data from the cloud - 53 files failed to download
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x000002159F326E88
```
For a number of files (not all as far I can tell) when running 
`dvc fetch [some_files]` on Azure devops windows CI machine with SPA auth.

DVC v2.0.3, everything seems to run fine on my local windows machine

This still fails if I set the number of jobs to 1

This is working fine with DVC v1.11.10


Comments:
@friggog Please post verbose log for that command. E.g. `dvc fetch -v`. Also, please post `dvc doctor` output.
```
DVC version: 2.0.3 (pip)
---------------------------------
Platform: Python 3.7.9 on Windows-10-10.0.14393-SP0
Supports: azure, hdfs, http, https, ssh
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: azure
Workspace directory: NTFS on D:\
Repo: dvc (subdir), git
2021-03-05 16:23:34,513 DEBUG: Check for update is enabled.
2021-03-05 16:23:34,540 DEBUG: Trying to spawn '['daemon', '-q', 'updater']'
2021-03-05 16:23:34,590 DEBUG: Spawned '['daemon', '-q', 'updater']'
2021-03-05 16:23:34,619 DEBUG: Checking if stage 'FILE_1' is in 'dvc.yaml'
2021-03-05 16:23:35,958 DEBUG: Checking if stage 'FILE_2' is in 'dvc.yaml'
2021-03-05 16:23:35,959 DEBUG: Checking if stage 'FILE_3' is in 'dvc.yaml'
2021-03-05 16:23:35,960 DEBUG: Checking if stage 'DIR\FILE_4' is in 'dvc.yaml'
2021-03-05 16:23:36,759 DEBUG: Preparing to download data from 'azure://CONTAINER_NAME'
2021-03-05 16:23:36,760 DEBUG: Preparing to collect status from azure://CONTAINER_NAME
2021-03-05 16:23:36,760 DEBUG: Collecting information from local cache...
2021-03-05 16:23:36,760 DEBUG: Collecting information from remote cache...
2021-03-05 16:23:36,761 DEBUG: Matched '0' indexed hashes
2021-03-05 16:23:36,761 DEBUG: Querying 1 hashes via object_exists
2021-03-05 16:23:36,765 DEBUG: failed to pull cache for 'DIR'
2021-03-05 16:23:35,960 DEBUG: Checking if stage 'DIR\FILE_5' is in 'dvc.yaml'
2021-03-05 16:23:36,759 DEBUG: Preparing to download data from 'azure://CONTAINER_NAME'
2021-03-05 16:23:36,760 DEBUG: Preparing to collect status from azure://CONTAINER_NAME
2021-03-05 16:23:36,760 DEBUG: Collecting information from local cache...
2021-03-05 16:23:36,760 DEBUG: Collecting information from remote cache...
2021-03-05 16:23:36,761 DEBUG: Matched '0' indexed hashes
2021-03-05 16:23:36,761 DEBUG: Querying 1 hashes via object_exists
2021-03-05 16:23:36,765 DEBUG: failed to pull cache for 'DIR'

... many times ....

-----------------------------------------------------------
2021-03-05 16:24:56,944 ERROR: failed to download 'azure://CONTAINER_NAME/e4/deb4d2426b0b5fd5919319e3eca7e7' to '.dvc\cache\e4\deb4d2426b0b5fd5919319e3eca7e7' - There is no current event loop in thread 'ThreadPoolExecutor-127_2'.
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\remote\base.py", line 35, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\fs\base.py", line 280, in download
    return self._download_file(from_info, to_info, name, no_progress_bar,)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\fs\base.py", line 332, in _download_file
    from_info, tmp_file, name=name, no_progress_bar=no_progress_bar
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\fs\fsspec_wrapper.py", line 104, in _download
    with self.open(from_info, "rb") as fobj:
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\fs\fsspec_wrapper.py", line 48, in open
    return self.fs.open(self._with_bucket(path_info), mode=mode)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\fsspec\spec.py", line 943, in open
    **kwargs,
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\adlfs\spec.py", line 1445, in _open
    **kwargs,
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\adlfs\spec.py", line 1509, in __init__
    fs.service_client.get_container_client(self.container_name)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\azure\storage\blob\aio\_blob_service_client_async.py", line 591, in get_container_client
    key_resolver_function=self.key_resolver_function, loop=self._loop)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\azure\storage\blob\aio\_container_client_async.py", line 118, in __init__
    **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\azure\storage\blob\_container_client.py", line 149, in __init__
    super(ContainerClient, self).__init__(parsed_url, service='blob', credential=credential, **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\azure\storage\blob\_shared\base_client.py", line 114, in __init__
    self._config, self._pipeline = self._create_pipeline(self.credential, storage_sdk=service, **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\azure\storage\blob\_shared\base_client_async.py", line 71, in _create_pipeline
    self._credential_policy = AsyncBearerTokenCredentialPolicy(credential, STORAGE_OAUTH_SCOPE)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\azure\core\pipeline\policies\_authentication_async.py", line 24, in __init__
    self._lock = asyncio.Lock()
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\asyncio\locks.py", line 161, in __init__
    self._loop = events.get_event_loop()
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\asyncio\events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-127_2'.

... many times ...

------------------------------------------------------------
2021-03-05 16:24:56,993 ERROR: failed to fetch data from the cloud - 53 files failed to download
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\command\data_sync.py", line 89, in run
    run_cache=self.args.run_cache,
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\dvc\repo\fetch.py", line 77, in fetch
    raise DownloadError(failed)
dvc.exceptions.DownloadError: 53 files failed to download
------------------------------------------------------------
2021-03-05 16:24:56,997 DEBUG: Analytics is enabled.
2021-03-05 16:24:57,000 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\tmpy3i7nnwh']'
2021-03-05 16:24:57,052 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\tmpy3i7nnwh']'
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x0000016651F9DEC8>
```

I'm pretty sure it is only files that are within a directory that is tracked (rather than the file itself being tracked) that are failing.

Also tried a fresh clone on my local machine and I'm getting auth errors with DefaultAzureCredentials :( 
Update that this problem persists on the latest version (2.0.13) and happens both locally and on CI

Here is the traceback (after fixing DefaultAzureCredential see #5725)

```
Traceback (most recent call last):
  File "python_lib\site-packages\adlfs\spec.py", line 483, in do_connect
    for cred in creds
  File "python_lib\site-packages\adlfs\spec.py", line 484, in <listcomp>
    if cred is not None
  File "python_lib\site-packages\azure\storage\blob\aio\_blob_service_client_async.py", line 118, in __init__
    **kwargs)
  File "python_lib\site-packages\azure\storage\blob\_blob_service_client.py", line 126, in __init__
    super(BlobServiceClient, self).__init__(parsed_url, service='blob', credential=credential, **kwargs)
  File "python_lib\site-packages\azure\storage\blob\_shared\base_client.py", line 114, in __init__
    self._config, self._pipeline = self._create_pipeline(self.credential, storage_sdk=service, **kwargs)
  File "python_lib\site-packages\azure\storage\blob\_shared\base_client_async.py", line 71, in _create_pipeline
    self._credential_policy = AsyncBearerTokenCredentialPolicy(credential, STORAGE_OAUTH_SCOPE)
  File "python_lib\site-packages\azure\core\pipeline\policies\_authentication_async.py", line 24, in __init__
    self._lock = asyncio.Lock()
  File "python_lib\asyncio\locks.py", line 161, in __init__
    self._loop = events.get_event_loop()
  File "python_lib\asyncio\events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-0_0'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "python_lib\site-packages\dvc\main.py", line 55, in main
    ret = cmd.run()
  File "python_lib\site-packages\dvc\command\data_sync.py", line 40, in run
    glob=self.args.glob,
  File "python_lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "python_lib\site-packages\dvc\repo\pull.py", line 38, in pull
    run_cache=run_cache,
  File "python_lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "python_lib\site-packages\dvc\repo\fetch.py", line 63, in fetch
    used, jobs, remote=remote, show_checksums=show_checksums,
  File "python_lib\site-packages\dvc\data_cloud.py", line 92, in pull
    show_checksums=show_checksums,
  File "python_lib\site-packages\dvc\remote\base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "python_lib\site-packages\dvc\remote\base.py", line 491, in pull
    download=True,
  File "python_lib\site-packages\dvc\remote\base.py", line 328, in _process
    download=download,
  File "python_lib\site-packages\dvc\remote\base.py", line 176, in _status
    md5s, jobs=jobs, name=str(self.fs.path_info)
  File "python_lib\site-packages\dvc\remote\base.py", line 132, in hashes_exist
    return indexed_hashes + self.odb.hashes_exist(list(hashes), **kwargs)
  File "python_lib\site-packages\dvc\objects\db\base.py", line 380, in hashes_exist
    remote_hashes = self.list_hashes_exists(hashes, jobs, name)
  File "python_lib\site-packages\dvc\objects\db\base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "python_lib\concurrent\futures\_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "python_lib\concurrent\futures\_base.py", line 435, in result
    return self.__get_result()
  File "python_lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "python_lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "python_lib\site-packages\dvc\objects\db\base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "python_lib\site-packages\dvc\fs\fsspec_wrapper.py", line 94, in exists
    return self.fs.exists(self._with_bucket(path_info))
  File "python_lib\site-packages\funcy\objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "python_lib\site-packages\funcy\objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "python_lib\site-packages\dvc\fs\azure.py", line 114, in fs
    file_system = AzureBlobFileSystem(**self.fs_args)
  File "python_lib\site-packages\fsspec\spec.py", line 66, in __call__
    obj = super().__call__(*args, **kwargs)
  File "python_lib\site-packages\adlfs\spec.py", line 403, in __init__
    self.do_connect()
  File "python_lib\site-packages\adlfs\spec.py", line 500, in do_connect
    raise ValueError(f"unable to connect to account for {e}")
ValueError: unable to connect to account for There is no current event loop in thread 'ThreadPoolExecutor-0_0'.
```

```
2021-03-30 08:55:25,226 DEBUG: Version info for developers:
DVC version: 2.0.13 (pip)
---------------------------------
Platform: Python 3.7.10 on Windows-10-10.0.19041-SP0
Supports: azure, http, https
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: azure
Workspace directory: NTFS on C:\
Repo: dvc (subdir), git
```
After some further investigation, this issue seems to arise because `adlfs` uses `azure.identity.aio` and `azure.storage.blob.aio ` which both use `asyncio`. Because DVC uses `ThreadPoolExecutor` at various points (e.g. [here](https://github.com/iterative/dvc/blob/a1aab35f997a1f9175f945f8266cf4038a5715b8/dvc/remote/base.py#L364) and [here](https://github.com/iterative/dvc/blob/a1aab35f997a1f9175f945f8266cf4038a5715b8/dvc/fs/base.py#L306)) without any handling for `asyncio` this causes the failures above where there is no event loop available to `aysncio` within threads of the `ThreadPoolExecutor`. Disabling uses of `ThreadPoolExecutor` within DVC results in pulls from Azure succeeding. 

@isidentical @efiop 
I'm a little confused why nobody else seems to have this issue? Also not sure what is the best path for a reliable fix - I have no experience with `asyncio`...
@friggog, could you please try installing adlfs from the master?  It could have been fixed in upstream.
```console
$ pip install git+https://github.com/dask/adlfs.git
```

Nope same error @skshetry 
@isidentical have you managed to repro this at all?
> @isidentical have you managed to repro this at all?

No, unfortunately, I haven't encounter this error. If you can share your specific conditions (which login method are you using, which operation it is exactly failing etc) I'll try to work on it. 
I'm using the DefaultAzureCredential with cached azure CLI credentials (run `az login` before DVC commands) and also ServicePrincipleAuth on Devops CI. Both have the same error with `dvc fetch some_files`
I've reproduced your issue @friggog via using `DefaultCredentials`. The fix is simple though it is a bit in the gray line on whether we should fix it on the dvc itself or on the adlfs so I've opened a question on the adlfs upstream dask/adlfs#218
@isidentical In the interest of making this functional for DVC users can we fix it here now even if not the perfect solution, then if eventually it gets fixed in adlfs the alterations can be removed from DVC?

Patch:
diff --git a/dvc/fs/azure.py b/dvc/fs/azure.py
--- a/dvc/fs/azure.py
+++ b/dvc/fs/azure.py
@@ -1,6 +1,8 @@
+import asyncio
 import logging
 import os
 import threading
+from contextlib import contextmanager
 
 from funcy import cached_property, wrap_prop
 
@@ -19,6 +21,34 @@
 )
 
 
+@contextmanager
+def _temp_event_loop():
+    """When trying to initalize azure filesystem instances
+    with DefaultCredentials, the authentication process requires
+    to have an access to a separate event loop. The normal calls
+    are run in a separate loop managed by the fsspec, but the
+    DefaultCredentials assumes that the callee is managing their
+    own event loop. This function checks whether is there any
+    event loop set, and if not it creates a temporary event loop
+    and set it. After the context is finalized, it restores the
+    original event loop back (if there is any)."""
+
+    try:
+        original_loop = asyncio.get_event_loop()
+    except RuntimeError:
+        original_loop = None
+
+    loop = original_loop or asyncio.new_event_loop()
+
+    try:
+        asyncio.set_event_loop(loop)
+        yield
+    finally:
+        if original_loop is None:
+            loop.close()
+        asyncio.set_event_loop(original_loop)
+
+
 class AzureAuthError(DvcException):
     pass
 
@@ -120,11 +150,12 @@ def fs(self):
         from azure.core.exceptions import AzureError
 
         try:
-            file_system = AzureBlobFileSystem(**self.fs_args)
-            if self.bucket not in [
-                container.rstrip("/") for container in file_system.ls("/")
-            ]:
-                file_system.mkdir(self.bucket)
+            with _temp_event_loop():
+                file_system = AzureBlobFileSystem(**self.fs_args)
+                if self.bucket not in [
+                    container.rstrip("/") for container in file_system.ls("/")
+                ]:
+                    file_system.mkdir(self.bucket)
         except (ValueError, AzureError) as e:
             raise AzureAuthError(
                 f"Authentication to Azure Blob Storage via {self.login_method}"
@@ -133,3 +164,9 @@ def fs(self):
             ) from e
 
         return file_system
+
+    def open(
+        self, path_info, mode="r", **kwargs
+    ):  # pylint: disable=arguments-differ
+        with _temp_event_loop():
+            return self.fs.open(self._with_bucket(path_info), mode=mode)
diff --git a/dvc/logger.py b/dvc/logger.py
--- a/dvc/logger.py
+++ b/dvc/logger.py
@@ -196,7 +196,17 @@ def disable_other_loggers():
 
 def setup(level=logging.INFO):
     colorama.init()
-    logging.getLogger("asyncio").setLevel(logging.CRITICAL)
+
+    if level >= logging.DEBUG:
+        # Unclosed session errors for asyncio/aiohttp are only available
+        # on the tracing mode for extensive debug purposes. They are really
+        # noisy, and this is potentially somewhere in the client library
+        # not managing their own session. Even though it is the best practice
+        # for them to do so, we can be assured that these errors raised when
+        # the object is getting deallocated, so no need to take any extensive
+        # action.
+        logging.getLogger("asyncio").setLevel(logging.CRITICAL)
+        logging.getLogger("aiohttp").setLevel(logging.CRITICAL)
 
     addLoggingLevel("TRACE", logging.DEBUG - 5)
     logging.config.dictConfig(


Test patch:
diff --git a/tests/unit/remote/test_azure.py b/tests/unit/fs/test_azure.py
similarity index 59%
rename from tests/unit/remote/test_azure.py
rename to tests/unit/fs/test_azure.py
--- a/tests/unit/remote/test_azure.py
+++ b/tests/unit/fs/test_azure.py
@@ -1,4 +1,9 @@
-from dvc.fs.azure import AzureFileSystem
+import asyncio
+from concurrent.futures import ThreadPoolExecutor
+
+import pytest
+
+from dvc.fs.azure import AzureFileSystem, _temp_event_loop
 from dvc.path_info import PathInfo
 
 container_name = "container-name"
@@ -37,3 +42,28 @@ def test_info(tmp_dir, azure):
     hash_ = fs.info(to_info)["etag"]
     assert isinstance(hash_, str)
     assert hash_.strip("'").strip('"') == hash_
+
+
+def test_temp_event_loop():
+    def procedure():
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(asyncio.sleep(0))
+        return "yeey"
+
+    def wrapped_procedure():
+        with _temp_event_loop():
+            return procedure()
+
+        # it should clean the loop after
+        # exitting the context.
+        with pytest.raises(RuntimeError):
+            asyncio.get_event_loop()
+
+    with ThreadPoolExecutor(max_workers=1) as executor:
+        future = executor.submit(procedure)
+
+        with pytest.raises(RuntimeError):
+            future.result()
+
+        future = executor.submit(wrapped_procedure)
+        assert future.result() == "yeey"