Repository: conda/conda

Issue:
cudatoolkit run constrain on __cuda not used for solve
<!--
Hi!  Read this; it's important.

This is an issue tracker for conda -- the package manager.  File feature requests
for conda here, as well as bug reports about something conda has messed up.

Anaconda Community Code of Conduct: https://www.anaconda.com/community-code-of-conduct/

If your issue is a bug report for:
  * a specific conda package from Anaconda ('defaults' channel):
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * a specific conda package from conda-forge:
      ==> file at the corresponding feedstock under https://github.com/conda-forge
  * repo.anaconda.com access and service:
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * anaconda.org access and service:
      ==> file at https://anaconda.org/contact/report
  * commands under 'conda build':
      ==> file at https://github.com/conda/conda-build
  * commands under 'conda env':
      ==> please file it here!
  * all other conda commands that start with 'conda':
      ==> please file it here!

If you continue on,

   **please include all requested information below.**

If a maintainer determines the information is required to understand 
your issue, and if it is not provided, your issue may be
closed automatically.

-->

## Current Behavior

Recently we added a repodata patch to `cudatoolkit` to constrain its version based on the `__cuda` virtual package's version ( https://github.com/AnacondaRecipes/repodata-hotfixes/pull/81 ) (IOW the max version of CUDA the GPU driver will support). However it seems this isn't respected when installing `cudatoolkit`. Instead one needs to prod `conda` a bit to get it to consider `__cuda`.

### Steps to Reproduce

Running the following does not take into account the `__cuda` version supported.

<details>
<summary>

```
conda create -d -n test_cuda cudatoolkit
```

</summary>

```
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ubuntu/miniforge/envs/test_cuda

  added / updated specs:
    - cudatoolkit


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |             main           3 KB
    cudatoolkit-10.2.89        |       hfd86e86_0       539.3 MB
    libgcc-ng-9.1.0            |       hdf63c60_0         8.1 MB
    libstdcxx-ng-9.1.0         |       hdf63c60_0         4.0 MB
    ------------------------------------------------------------
                                           Total:       551.5 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.2.89-hfd86e86_0
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0



DryRunExit: Dry run. Exiting.
```

</details>

However adding `__cuda` to the install spec does work.

<details>
<summary>

```
conda create -d -n test_cuda __cuda cudatoolkit
```

</summary>

```
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ubuntu/miniforge/envs/test_cuda

  added / updated specs:
    - __cuda
    - cudatoolkit


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-10.1.243       |       h6bb024c_0       513.2 MB
    ------------------------------------------------------------
                                           Total:       513.2 MB

The following NEW packages will be INSTALLED:

  cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.1.243-h6bb024c_0



DryRunExit: Dry run. Exiting.
```

</details>

Interestingly we also observe compiler runtime packages installed in the first case, but not in the second. Unclear on why that happens, but may also be related to the same issue.


## Expected Behavior

When installing `cudatoolkit` or really anything that depends on `cudatoolkit`, `__cuda` should be factored into the solve.

## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

```
     active environment : base
    active env location : /datasets/jkirkham/miniconda
            shell level : 1
       user config file : /home/nfs/jkirkham/.condarc
 populated config files : /home/nfs/jkirkham/.condarc
          conda version : 4.8.3
    conda-build version : 3.19.2
         python version : 3.7.6.final.0
       virtual packages : __cuda=10.1
                          __glibc=2.27
       base environment : /datasets/jkirkham/miniconda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
          package cache : /datasets/jkirkham/miniconda/pkgs
                          /home/nfs/jkirkham/.conda/pkgs
       envs directories : /datasets/jkirkham/miniconda/envs
                          /home/nfs/jkirkham/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.23.0 CPython/3.7.6 Linux/4.15.0-76-generic ubuntu/18.04.4 glibc/2.27
                UID:GID : 10124:10004
             netrc file : None
           offline mode : False
```
</p></details>


<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

```
==> /home/nfs/jkirkham/.condarc <==
channels:
  - conda-forge
  - defaults
  - nvidia
show_channel_urls: True
```
</p></details>


<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda list --show-channel-urls' -->

```
# packages in environment at /datasets/jkirkham/miniconda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       0_gnu    conda-forge
beautifulsoup4            4.9.0            py37hc8dfbb8_0    conda-forge
brotlipy                  0.7.0           py37h8f50634_1000    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
cffi                      1.14.0           py37hd463f26_0    conda-forge
chardet                   3.0.4           py37hc8dfbb8_1006    conda-forge
conda                     4.8.3            py37hc8dfbb8_1    conda-forge
conda-build               3.19.2           py37hc8dfbb8_0    conda-forge
conda-package-handling    1.6.0            py37h8f50634_2    conda-forge
cryptography              2.9.2            py37hb09aad4_0    conda-forge
filelock                  3.0.10                     py_0    conda-forge
glob2                     0.7                        py_0    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
idna                      2.9                        py_1    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
libarchive                3.3.3             h3a8160c_1008    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgomp                   9.2.0                h24d8f2e_2    conda-forge
libiconv                  1.15              h516909a_1006    conda-forge
liblief                   0.9.0                hf8a498c_1    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libxml2                   2.9.10               hee79883_0    conda-forge
lz4-c                     1.9.2                he1b5a44_0    conda-forge
lzo                       2.10              h14c3975_1000    conda-forge
markupsafe                1.1.1            py37h8f50634_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
patchelf                  0.10                 he1b5a44_0    conda-forge
pip                       20.0.2                     py_2    conda-forge
pkginfo                   1.5.0.1                    py_0    conda-forge
psutil                    5.7.0            py37h8f50634_1    conda-forge
py-lief                   0.9.0            py37he1b5a44_1    conda-forge
pycosat                   0.6.3           py37h8f50634_1004    conda-forge
pycparser                 2.20                       py_0    conda-forge
pyopenssl                 19.1.0                     py_1    conda-forge
pysocks                   1.7.1            py37hc8dfbb8_1    conda-forge
python                    3.7.6           h8356626_5_cpython    conda-forge
python-libarchive-c       2.9                      py37_0    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2020.1             pyh9f0ad1d_0    conda-forge
pyyaml                    5.3.1            py37h8f50634_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.23.0             pyh8c360ce_2    conda-forge
ripgrep                   12.0.1               h516909a_1    conda-forge
ruamel_yaml               0.15.80         py37h8f50634_1001    conda-forge
setuptools                46.1.3           py37hc8dfbb8_0    conda-forge
six                       1.14.0                     py_1    conda-forge
soupsieve                 1.9.4            py37hc8dfbb8_1    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
tqdm                      4.45.0             pyh9f0ad1d_1    conda-forge
urllib3                   1.25.9                     py_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.5                h516909a_0    conda-forge
yaml                      0.2.4                h516909a_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h6597ccf_3    conda-forge
```
</p></details>



Discussion:

Resolving patch:
diff --git a/conda/core/solve.py b/conda/core/solve.py
--- a/conda/core/solve.py
+++ b/conda/core/solve.py
@@ -390,6 +390,14 @@ def _collect_all_metadata(self, ssc):
             if pkg_name not in ssc.specs_map and ssc.prefix_data.get(pkg_name, None):
                 ssc.specs_map[pkg_name] = MatchSpec(pkg_name)
 
+        # Add virtual packages so they are taken into account by the solver
+        virtual_pkg_index = {}
+        _supplement_index_with_system(virtual_pkg_index)
+        virtual_pkgs = [p.name for p in virtual_pkg_index.keys()]
+        for virtual_pkgs_name in (virtual_pkgs):
+            if virtual_pkgs_name not in ssc.specs_map:
+                ssc.specs_map[virtual_pkgs_name] = MatchSpec(virtual_pkgs_name)
+
         for prec in ssc.prefix_data.iter_records():
             # first check: add everything if we have no history to work with.
             #    This happens with "update --all", for example.
diff --git a/conda/exceptions.py b/conda/exceptions.py
--- a/conda/exceptions.py
+++ b/conda/exceptions.py
@@ -679,11 +679,11 @@ def __init__(self, bad_deps, chains=True, strict=False):
 
 The following specifications were found to be incompatible with each other:
                     '''),
-                    'cuda': dals('''
+                    'virtual_package': dals('''
 
-The following specifications were found to be incompatible with your CUDA driver:\n{specs}
+The following specifications were found to be incompatible with your system:\n{specs}
 
-Your installed CUDA driver is: {ref}
+Your installed version is: {ref}
 ''')}
 
         msg = ""
diff --git a/conda/resolve.py b/conda/resolve.py
--- a/conda/resolve.py
+++ b/conda/resolve.py
@@ -292,7 +292,8 @@ def _classify_bad_deps(self, bad_deps, specs_to_add, history_specs, strict_chann
         classes = {'python': set(),
                    'request_conflict_with_history': set(),
                    'direct': set(),
-                   'cuda': set(), }
+                   'virtual_package': set(),
+                   }
         specs_to_add = set(MatchSpec(_) for _ in specs_to_add or [])
         history_specs = set(MatchSpec(_) for _ in history_specs or [])
         for chain in bad_deps:
@@ -307,10 +308,10 @@ def _classify_bad_deps(self, bad_deps, specs_to_add, history_specs, strict_chann
                             set(self.find_matches(chain[-1]))):
                         classes['python'].add((tuple([chain[0], chain[-1]]),
                                                str(MatchSpec(python_spec, target=None))))
-            elif chain[-1].name == '__cuda':
-                cuda_version = [_ for _ in self._system_precs if _.name == '__cuda']
-                cuda_version = cuda_version[0].version if cuda_version else "not available"
-                classes['cuda'].add((tuple(chain), cuda_version))
+            elif chain[-1].name.startswith('__'):
+                version = [_ for _ in self._system_precs if _.name == chain[-1].name]
+                virtual_package_version = version[0].version if version else "not available"
+                classes['virtual_package'].add((tuple(chain), virtual_package_version))
             elif chain[0] in specs_to_add:
                 match = False
                 for spec in history_specs:
@@ -444,6 +445,8 @@ def build_conflict_map(self, specs, specs_to_add=None, history_specs=None):
         strict_channel_priority = context.channel_priority == ChannelPriority.STRICT
 
         specs = set(specs) | (specs_to_add or set())
+        # Remove virtual packages
+        specs = set([spec for spec in specs if not spec.name.startswith('__')])
         if len(specs) == 1:
             matches = self.find_matches(next(iter(specs)))
             if len(matches) == 1:


Test patch:
diff --git a/tests/core/test_solve.py b/tests/core/test_solve.py
--- a/tests/core/test_solve.py
+++ b/tests/core/test_solve.py
@@ -236,6 +236,16 @@ def test_solve_2(tmpdir):
         assert len(prec_names) == len(set(prec_names))
 
 
+def test_virtual_package_solver(tmpdir):
+    specs = MatchSpec("cudatoolkit"),
+
+    with env_var('CONDA_OVERRIDE_CUDA', '10.0'):
+        with get_solver_cuda(tmpdir, specs) as solver:
+            final_state = solver.solve_final_state()
+            # Check the cuda virtual package is included in the solver
+            assert '__cuda' in solver.ssc.specs_map.keys()
+
+
 def test_cuda_1(tmpdir):
     specs = MatchSpec("cudatoolkit"),
 
@@ -282,12 +292,13 @@ def test_cuda_fail_1(tmpdir):
             plat = "win-64"
     else:
         plat = "linux-64"
-    assert str(exc.value).strip() == dals("""The following specifications were found to be incompatible with your CUDA driver:
+
+    assert str(exc.value).strip() == dals("""The following specifications were found to be incompatible with your system:
 
   - feature:/{}::__cuda==8.0=0
   - cudatoolkit -> __cuda[version='>=10.0|>=9.0']
 
-Your installed CUDA driver is: 8.0""".format(plat))
+Your installed version is: 8.0""".format(plat))
 
 
 
@@ -299,11 +310,12 @@ def test_cuda_fail_2(tmpdir):
         with get_solver_cuda(tmpdir, specs) as solver:
             with pytest.raises(UnsatisfiableError) as exc:
                 final_state = solver.solve_final_state()
-    assert str(exc.value).strip() == dals("""The following specifications were found to be incompatible with your CUDA driver:
+
+    assert str(exc.value).strip() == dals("""The following specifications were found to be incompatible with your system:
 
   - cudatoolkit -> __cuda[version='>=10.0|>=9.0']
 
-Your installed CUDA driver is: not available""")
+Your installed version is: not available""")
 
 def test_prune_1(tmpdir):
     specs = MatchSpec("numpy=1.6"), MatchSpec("python=2.7.3"), MatchSpec("accelerate"),
diff --git a/tests/test_create.py b/tests/test_create.py
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -2112,34 +2112,23 @@ def test_install_freezes_env_by_default(self):
         # create an initial env
         with make_temp_env("python=2", use_restricted_unicode=on_win, no_capture=True) as prefix:
             assert package_is_installed(prefix, "python=2.7.*")
+            # Install a version older than the last one
+            run_command(Commands.INSTALL, prefix, "setuptools=40.*")
+
             stdout, stderr, _ = run_command(Commands.LIST, prefix, '--json')
+
             pkgs = json.loads(stdout)
-            specs = set()
-            for entry in pkgs:
-                if entry['name'] in DEFAULT_AGGRESSIVE_UPDATE_PACKAGES:
-                    specs.add(MatchSpec(entry['name']))
-                else:
-                    specs.add(MatchSpec(entry['name'],
-                                        version=entry['version'],
-                                        channel=entry['channel'],
-                                        subdir=entry['platform'],
-                                        build=entry['build_string']))
-            ms = MatchSpec('imagesize')
-            specs.add(ms)
-
-            import conda.core.solve
-            r = conda.core.solve.Resolve(get_index())
-            reduced_index = r.get_reduced_index([ms])
-
-            # now add imagesize to that env.  The call to get_reduced_index should include our exact specs
-            #     for the existing env.  doing so will greatly reduce the search space for the initial solve
-            with patch.object(conda.core.solve.Resolve, 'get_reduced_index',
-                              return_value=reduced_index) as mock_method:
-                run_command(Commands.INSTALL, prefix, "imagesize")
-                # TODO: this should match the specs above.  It does, at least as far as I can tell from text
-                #    comparison.   Unfortunately, it doesn't evaluate as a match, even though the text all matches.
-                #    I suspect some strange equality of objects issue.
-                mock_method.assert_called_with(specs, exit_on_conflict=False)
+
+            run_command(Commands.INSTALL, prefix, "imagesize", "--freeze-installed")
+
+            stdout, _, _ = run_command(Commands.LIST, prefix, '--json')
+            pkgs_after_install = json.loads(stdout)
+
+            # Compare before and after installing package
+            for pkg in pkgs:
+                for pkg_after in pkgs_after_install:
+                    if pkg["name"] == pkg_after["name"]:
+                        assert pkg["version"] == pkg_after["version"]
 
     @pytest.mark.skipif(on_win, reason="gawk is a windows only package")
     def test_search_gawk_not_win_filter(self):