Repository: conda/conda

Issue:
Conda 4.7+ extremely slow "Collecting package metadata (repodata.json)" with local (file:///) channels
When installing or updating packages, conda gets stuck for an incredibly long time at the "Collecting package metadata (repodata.json)" step.

This appears to be a regression caused by conda 4.7 (and remains in 4.8) as this operation was quite fast in conda 4.6.

The slowdown only appears to occur when using (file:///) channels on a local or network filesystem. 

To test this I ran `python -m http.server 8000` in the channel parent directory and renamed channel URLs in `.condarc` from `file:///<DRIVE>:/<PATH>` to `http://localhost:8000/`. Commands then run in around the same time as they did in conda 4.6.


<!--
Hi!  Read this; it's important.

Anaconda Community Code of Conduct: https://www.anaconda.com/community-code-of-conduct/

We get a lot of reports on the conda issue tracker about speed. These are mostly
not very helpful, because they only very generally complain about the total time
conda is taking. They often don’t even say what conda is taking so long to do -
just that it’s slow. If you want to file an issue report about conda’s speed, we
ask that you take the time to isolate exactly what part is slow, and what you think
is a reasonable amount of time for that operation to take (preferably by
comparison with another tool that is performing a similar task). Please see some
tips below on how to collect useful information for a bug report.

Complaints that include no useful information will be ignored and closed.

-->

## Steps to Reproduce
`conda update conda` or any other conda update/install command
<!-- Show us some debugging output.  Here we generate conda_debug_output.txt - please upload it with your report.

* On unix (bash shells): 

  CONDA_INSTRUMENTATION_ENABLED=1 <your conda command> -vv | tee conda_debug_output.txt

* On Windows:

  set CONDA_INSTRUMENTATION_ENABLED=1
  powershell "<your conda command> -vv | tee conda_debug_output.txt"

-->
[conda-debug.txt](https://github.com/conda/conda/files/4156708/conda-debug.txt)
(note I exited (Ctrl+c) the command after a few minutes, normally it would run for an hour or more)
```
DEBUG conda.gateways.logging:set_verbosity(231): verbosity set to 2
DEBUG conda.core.solve:solve_final_state(223): solving prefix C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3
  specs_to_remove: frozenset()
  specs_to_add: frozenset({MatchSpec("conda")})
  prune: <auxlib._Null object at 0x0000024AA6792148>
DEBUG conda.core.package_cache_data:_check_writable(259): package cache directory 'C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs' writable: True
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\e8bdfbda.json
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\270cd9d6.json
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/main/win-64/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\a83aae50.json
DEBUG conda.core.subdir_data:_load(241): Local cache timed out for file:///X:/<LOCAL-MIRROR-PATH>/main/noarch/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\ce3ccb3a.json
INFO conda._vendor.auxlib.logz:stringify(144): request is 'None' for Response object with url file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch/repodata.json
DEBUG conda.core.subdir_data:fetch_repodata_remote_request(487):
<<FILE 200 None
< Content-Type: application/json
< Last-Modified: Tue, 13 Nov 2018 03:00:11 GMT
< Content-Length: 103
< Elapsed: 00:00.840189

{ 'info': {'subdir': 'noarch'},
  'packages': {},
  'removed': [],
  'repodata_version': 1}


DEBUG conda.core.subdir_data:_pickle_me(284): Saving pickled state for file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\e8bdfbda.json
INFO conda._vendor.auxlib.logz:stringify(144): request is 'None' for Response object with url file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64/repodata.json
DEBUG conda.core.subdir_data:fetch_repodata_remote_request(487):
<<FILE 200 None
< Content-Type: application/json
< Last-Modified: Thu, 30 Jan 2020 00:18:12 GMT
< Content-Length: 227211
< Elapsed: 00:00.843118

{ 'info': {'subdir': 'win-64'},
  'packages': { 'm2-autoconf-2.69-3.tar.bz2': { 'build': '3',
                                                'build_number': 3,
                                                'depends': [ 'm2-bash',



INFO conda._vendor.auxlib.logz:stringify(144): request is 'None' for Response object with url file:///X:/<LOCAL-MIRROR-PATH>/main/noarch/repodata.json
DEBUG conda.core.subdir_data:_pickle_me(284): Saving pickled state for file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64/repodata.json at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs\cache\270cd9d6.json
DEBUG conda.core.subdir_data:fetch_repodata_remote_request(487):
<<FILE 200 None
< Content-Type: application/json
< Last-Modified: Thu, 30 Jan 2020 00:17:55 GMT
< Content-Length: 704400
< Elapsed: 00:00.841911

{ 'info': {'subdir': 'noarch'},
  'packages': { 'affine-2.1.0-pyh128a3a6_1.tar.bz2': { 'build': 'pyh128a3a6_1',
                                                       'build_number': 1,
                                                       'depends': ['py

```

```
conda.core.prefix_data.PrefixData.load,0.565156
conda.core.solve.Solver._collect_all_metadata,119.613876
conda.core.prefix_data.PrefixData.load,0.163001
conda.core.solve.Solver._collect_all_metadata,9.134097
conda.core.prefix_data.PrefixData.load,0.161001
conda.core.solve.Solver._collect_all_metadata,84.478895
```


## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

```
     active environment : None
            shell level : 0
       user config file : C:\Users\<USERNAME>\.condarc
 populated config files : C:\Users\<USERNAME>\.condarc
          conda version : 4.8.1
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages :
       base environment : C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3  (writable)
           channel URLs : file:///X:/<LOCAL-MIRROR-PATH>/main/win-64
                          file:///X:/<LOCAL-MIRROR-PATH>/main/noarch
                          file:///X:/<LOCAL-MIRROR-PATH>/msys2/win-64
                          file:///X:/<LOCAL-MIRROR-PATH>/msys2/noarch
          package cache : C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\pkgs
                          C:\Users\<USERNAME>\.conda\pkgs
                          C:\Users\<USERNAME>\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3\envs
                          C:\Users\<USERNAME>\.conda\envs
                          C:\Users\<USERNAME>\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.1 requests/2.22.0 CPython/3.7.6 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False
```
</p></details>


<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

```
==> C:\Users\<USERNAME>\.condarc <==
auto_update_conda: False
auto_activate_base: False
channel_alias: file:///X:/<LOCAL-MIRROR-PATH>
channel_priority: disabled
channels:
  - defaults
default_channels:
  - main
  - msys2
show_channel_urls: True
use_only_tar_bz2: True
report_errors: False
```
</p></details>


<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda list --show-channel-urls' -->

```
# packages in environment at C:\Users\<USERNAME>\AppData\Local\Programs\Miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                1.3.0                    py37_0    defaults
backports                 1.0                        py_2    defaults
backports.functools_lru_cache 1.6.1                      py_0    defaults
backports.tempfile        1.0                        py_1    defaults
backports.weakref         1.0.post1                  py_1    defaults
beautifulsoup4            4.8.2                    py37_0    defaults
bzip2                     1.0.8                he774522_0    defaults
ca-certificates           2019.11.27                    0    defaults
certifi                   2019.11.28               py37_0    defaults
cffi                      1.13.2           py37h7a1dbc1_0    defaults
chardet                   3.0.4                 py37_1003    defaults
click                     7.0                        py_0    defaults
conda                     4.8.1                    py37_0    defaults
conda-build               3.18.11                  py37_0    defaults
conda-package-handling    1.6.0            py37h62dcd97_0    defaults
conda-verify              3.4.2                      py_1    defaults
console_shortcut          0.1.1                         3    https://repo.anaconda.com/pkgs/main
constructor               3.0.0                    py37_0    defaults
cryptography              2.8              py37h7a1dbc1_0    defaults
filelock                  3.0.12                     py_0    defaults
freetype                  2.9.1                ha9979f8_1    defaults
future                    0.18.2                   py37_0    defaults
glob2                     0.7                        py_0    defaults
idna                      2.8                      py37_0    https://repo.anaconda.com/pkgs/main
jinja2                    2.10.3                     py_0    defaults
jpeg                      9b                   hb83a4c4_2    defaults
libarchive                3.3.3                h0643e63_5    defaults
libiconv                  1.15                 h1df5818_7    defaults
liblief                   0.9.0                ha925a31_2    defaults
libpng                    1.6.37               h2a8f88b_0    defaults
libtiff                   4.1.0                h56a325e_0    defaults
libxml2                   2.9.9                h464c3ec_0    defaults
lz4-c                     1.8.1.2              h2fa13f4_0    defaults
lzo                       2.10                 h6df0209_2    defaults
m2-msys2-runtime          2.5.0.17080.65c939c               3    defaults
m2-patch                  2.7.5                         2    defaults
markupsafe                1.1.1            py37he774522_0    defaults
menuinst                  1.4.16           py37he774522_0    https://repo.anaconda.com/pkgs/main
msys2-conda-epoch         20160418                      1    defaults
nsis                      3.01                          8    defaults
olefile                   0.46                     py37_0    defaults
openssl                   1.1.1d               he774522_3    defaults
pillow                    7.0.0            py37hcc1f983_0    defaults
pip                       20.0.2                   py37_0    defaults
pkginfo                   1.5.0.1                  py37_0    defaults
powershell_shortcut       0.0.1                         2    https://repo.anaconda.com/pkgs/main
psutil                    5.6.7            py37he774522_0    defaults
py-lief                   0.9.0            py37ha925a31_2    defaults
pycosat                   0.6.3            py37he774522_0    defaults
pycparser                 2.19                       py_0    defaults
pyopenssl                 19.1.0                   py37_0    defaults
pysocks                   1.7.1                    py37_0    defaults
python                    3.7.6                h60c2a47_2    defaults
python-dateutil           2.8.1                      py_0    defaults
python-libarchive-c       2.8                     py37_13    defaults
pytz                      2019.3                     py_0    defaults
pywin32                   227              py37he774522_1    defaults
pyyaml                    5.2              py37he774522_0    defaults
requests                  2.22.0                   py37_1    defaults
ruamel_yaml               0.15.87          py37he774522_0    defaults
setuptools                45.1.0                   py37_0    defaults
six                       1.14.0                   py37_0    defaults
soupsieve                 1.9.5                    py37_0    defaults
sqlite                    3.30.1               he774522_0    defaults
tk                        8.6.8                hfa6e2cd_0    defaults
tqdm                      4.42.0                     py_0    defaults
urllib3                   1.25.8                   py37_0    defaults
vc                        14.1                 h0510ff6_4    https://repo.anaconda.com/pkgs/main
vs2015_runtime            14.16.27012          hf0eaf9b_1    defaults
wheel                     0.33.6                   py37_0    defaults
win_inet_pton             1.1.0                    py37_0    https://repo.anaconda.com/pkgs/main
wincertstore              0.2                      py37_0    https://repo.anaconda.com/pkgs/main
xz                        5.2.4                h2fa13f4_4    defaults
yaml                      0.1.7                hc54c509_2    https://repo.anaconda.com/pkgs/main
zlib                      1.2.11               h62dcd97_3    defaults
zstd                      1.3.7                h508b16e_0    defaults
```
</p></details>



Discussion:
I've seen the same thing, a dramatic slow-down in performance when using file-based channels, especially on slow NFS shares.    I think I've found a solution and hope to submit a proper pull request in a day or two, but for now, this information might be useful...

The new solver seems (under some circumstances) to create many instances of the SubDirData class for the same channel, when the old solver only produced one.   That should be okay, as the class is loaded using a metaclass that implements a cache.   So we should only create at most two instances per channel., one for "current_repodata.json" and (perhaps) one for "repodata.json".

The problem is that this cache is explicitly disabled for file-based channels.   That didn't matter much when we were only requesting each object once, but that's no longer the case.  Each instance has to read the file, parse it, and write a copy back to a disk-based cache.   That takes time on slow NFS drives.

My proposed fix will be to change the if statement [here](https://github.com/conda/conda/blob/c154d2e3c68841a82376633f2ebc9ffc916dbf1c/conda/core/subdir_data.py#L66) so that it doesn't check for urls starting with "file:///".

In my test case, I could add certain packages to an environment in 5.6 seconds using conda 4.6.8.  When I upgraded to conda 4.8.2 that suddenly took 93 seconds.   Hacking that if statement to allow caching for file-based urls brought it back down to 6.5 seconds.

As I said, I hope to submit a proper pull request soon. If anyone else wants to do so, feel free.  :)

Resolving patch:
diff --git a/conda/core/subdir_data.py b/conda/core/subdir_data.py
--- a/conda/core/subdir_data.py
+++ b/conda/core/subdir_data.py
@@ -14,7 +14,7 @@
 import json
 from logging import DEBUG, getLogger
 from mmap import ACCESS_READ, mmap
-from os.path import dirname, isdir, join, splitext
+from os.path import dirname, isdir, join, splitext, exists
 import re
 from time import time
 import warnings
@@ -29,6 +29,7 @@
 from ..common.compat import (ensure_binary, ensure_text_type, ensure_unicode, iteritems, iterkeys,
                              string_types, text_type, with_metaclass)
 from ..common.io import ThreadLimitedThreadPoolExecutor, DummyExecutor, dashlist
+from ..common.path import url_to_path
 from ..common.url import join_url, maybe_unquote
 from ..core.package_cache_data import PackageCacheData
 from ..exceptions import (CondaDependencyError, CondaHTTPError, CondaUpgradeError,
@@ -62,11 +63,19 @@ def __call__(cls, channel, repodata_fn=REPODATA_FN):
         assert channel.subdir
         assert not channel.package_filename
         assert type(channel) is Channel
+        now = time()
         cache_key = channel.url(with_credentials=True), repodata_fn
-        if not cache_key[0].startswith('file://') and cache_key in SubdirData._cache_:
-            return SubdirData._cache_[cache_key]
-
+        if cache_key in SubdirData._cache_:
+            cache_entry = SubdirData._cache_[cache_key]
+            if cache_key[0].startswith('file://'):
+                file_path = url_to_path(channel.url() + '/' + repodata_fn)
+                if exists(file_path):
+                    if cache_entry._mtime > getmtime(file_path):
+                        return cache_entry
+            else:
+                return cache_entry
         subdir_data_instance = super(SubdirDataType, cls).__call__(channel, repodata_fn)
+        subdir_data_instance._mtime = now
         SubdirData._cache_[cache_key] = subdir_data_instance
         return subdir_data_instance
 
@@ -75,6 +84,12 @@ def __call__(cls, channel, repodata_fn=REPODATA_FN):
 class SubdirData(object):
     _cache_ = {}
 
+    @classmethod
+    def clear_cached_local_channel_data(cls):
+        # This should only ever be needed during unit tests, when
+        # CONDA_USE_ONLY_TAR_BZ2 may change during process lifetime.
+        cls._cache_ = {k: v for k, v in cls._cache_.items() if not k[0].startswith('file://')}
+
     @staticmethod
     def query_all(package_ref_or_match_spec, channels=None, subdirs=None,
                   repodata_fn=REPODATA_FN):
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -8,6 +8,7 @@
 
 from conda.common.compat import PY3
 from conda.gateways.disk.create import TemporaryDirectory
+from conda.core.subdir_data import SubdirData
 
 win_default_shells = ["cmd.exe", "powershell", "git_bash", "cygwin"]
 shells = ["bash", "zsh"]
@@ -65,6 +66,11 @@ def set_tmpdir(tmpdir):
     tmpdir_in_use = td
 
 
+@pytest.fixture(autouse=True)
+def clear_subdir_cache(tmpdir):
+    SubdirData.clear_cached_local_channel_data()
+
+
 # From: https://hackebrot.github.io/pytest-tricks/fixtures_as_class_attributes/
 # This allows using pytest fixtures in unittest subclasses, here is how to use it:
 #


Test patch:
diff --git a/tests/core/test_subdir_data.py b/tests/core/test_subdir_data.py
--- a/tests/core/test_subdir_data.py
+++ b/tests/core/test_subdir_data.py
@@ -4,6 +4,7 @@
 from logging import getLogger
 from os.path import dirname, join
 from unittest import TestCase
+from time import sleep
 
 import pytest
 
@@ -201,16 +202,59 @@ def test_subdir_data_prefers_conda_to_tar_bz2():
 
 def test_use_only_tar_bz2():
     channel = Channel(join(dirname(__file__), "..", "data", "conda_format_repo", context.subdir))
+    SubdirData.clear_cached_local_channel_data()
     with env_var('CONDA_USE_ONLY_TAR_BZ2', True, stack_callback=conda_tests_ctxt_mgmt_def_pol):
         sd = SubdirData(channel)
         precs = tuple(sd.query("zlib"))
         assert precs[0].fn.endswith(".tar.bz2")
+    SubdirData.clear_cached_local_channel_data()
     with env_var('CONDA_USE_ONLY_TAR_BZ2', False, stack_callback=conda_tests_ctxt_mgmt_def_pol):
         sd = SubdirData(channel)
         precs = tuple(sd.query("zlib"))
         assert precs[0].fn.endswith(".conda")
 
 
+def test_metadata_cache_works():
+    channel = Channel(join(dirname(__file__), "..", "data", "conda_format_repo", context.subdir))
+    SubdirData.clear_cached_local_channel_data()
+
+    # Sadly, on Windows, st_mtime resolution is limited to 2 seconds. (See note in Python docs
+    # on os.stat_result.)  To ensure that the timestamp on the existing JSON file is safely
+    # in the past before this test starts, we need to wait for more than 2 seconds...
+
+    sleep(3)
+
+    with patch('conda.core.subdir_data.fetch_repodata_remote_request',
+               wraps=fetch_repodata_remote_request) as fetcher:
+        sd_a = SubdirData(channel)
+        precs_a = tuple(sd_a.query("zlib"))
+        assert fetcher.call_count == 1
+
+        sd_b = SubdirData(channel)
+        assert sd_b is sd_a
+        precs_b = tuple(sd_b.query("zlib"))
+        assert fetcher.call_count == 1
+
+
+def test_metadata_cache_clearing():
+    channel = Channel(join(dirname(__file__), "..", "data", "conda_format_repo", context.subdir))
+    SubdirData.clear_cached_local_channel_data()
+
+    with patch('conda.core.subdir_data.fetch_repodata_remote_request',
+               wraps=fetch_repodata_remote_request) as fetcher:
+        sd_a = SubdirData(channel)
+        precs_a = tuple(sd_a.query("zlib"))
+        assert fetcher.call_count == 1
+
+        SubdirData.clear_cached_local_channel_data()
+
+        sd_b = SubdirData(channel)
+        assert sd_b is not sd_a
+        precs_b = tuple(sd_b.query("zlib"))
+        assert fetcher.call_count == 2
+        assert precs_b == precs_a
+
+
 # @pytest.mark.integration
 # class SubdirDataTests(TestCase):
 #