Repository: conda/conda

Issue:
Regression in conda 4.3.33 regarding build features
It seems that `conda 4.3.33` installs packages that have a build feature when installing any package that has the same build feature.

From a fresh miniconda, with conda pinned at `4.3.32`:

```sh
scorlay@mbp ~]$ conda install cryptopp -c QuantStack -c conda-forge
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment /Users/scorlay/miniconda3:

The following NEW packages will be INSTALLED:

    cling-patches: 1-0           QuantStack
    cryptopp:      5.6.5-cling_0 QuantStack [cling]
```

With `conda 4.3.33`

```sh
[scorlay@mbp ~]$ conda install cryptopp -c QuantStack -c conda-forge
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment /Users/scorlay/miniconda3:

The following NEW packages will be INSTALLED:

    clangdev:      5.0.0-cling_1 QuantStack  [cling]
    cling-patches: 1-0           QuantStack 
    cryptopp:      5.6.5-cling_0 QuantStack  [cling]
    icu:           58.2-0        conda-forge
    libiconv:      1.15-0        conda-forge
    libxml2:       2.9.7-0       conda-forge
    llvmdev:       5.0.0-cling_1 QuantStack  [cling]
```

The only package tracking the `cling` build feature is `cling-patches`.

Notice in the second case that `clangdev` gets installed while it is not a dependency.


Discussion:
As per @jjhelmus 's comment, this might be due to #6766. cc @mcg1969 .
Cannot reproduce thiswith a fresh `Miniconda3-4.3.31-Linux-x86_64.sh` installation updated to `conda 4.3.32`:
```bash
$ conda --version
conda 4.3.32
$ conda create -qnx --dry-run -c QuantStack -c conda-forge cryptopp cling@


Package plan for installation in environment /home/maba/code/conda/mc4332/envs/x:

The following NEW packages will be INSTALLED:

    cling-patches: 1-0           QuantStack
    cryptopp:      5.6.5-cling_0 QuantStack [cling]
    libgcc-7:      7.2.0-0       QuantStack

$ conda install -qy conda=4.3.33


Package plan for installation in environment /home/maba/code/conda/mc4332:

The following packages will be UPDATED:

    conda: 4.3.32-py36_0 --> 4.3.33-py36_0

$ conda create -qnx --dry-run -c QuantStack -c conda-forge cryptopp cling@


Package plan for installation in environment /home/maba/code/conda/mc4332/envs/x:

The following NEW packages will be INSTALLED:

    cling-patches: 1-0           QuantStack
    cryptopp:      5.6.5-cling_0 QuantStack [cling]
    libgcc-7:      7.2.0-0       QuantStack
```
Can you give more information on your environment? (E.g., fill out the issue template.)
This is something I am seeing on OS X. Locally, and also on Travis. This is impacting our conda-forge style builds.
Features need to die a fiery death. They simply are not SAT representable as conceived :-(

I will see what I can do.
> Features need to die a fiery death. They simply are not SAT representable as conceived :-(
> I will see what I can do.

What should I use instead.
@SylvainCorlay can we get a `conda create` recipe from you that can reproduce the problem, so that it can be reliably reproduced?
> What should I used instead.

It depends exactly what you are trying to do.
> @SylvainCorlay can we get a conda create recipe from you that can reproduce the problem, so that it can be reliably reproduced?

I don't understand what you would like here, but I would be happy to help.
Your examples involve `conda install` commands --- I wonder if you can give us a `conda create` command instead that reveals the issue. With `conda install`, it assumes an existing envrionment with a set of packages that might be influencing the solution.
ok will do. FYI, I produced this from the root environment of a newly installed miniconda.
Note: I am not reproducing with `conda create`. Only `conda install`.
`conda create` reproducer:
```
|$ conda --version
conda 4.3.32
|$ CONDA_SUBDIR=osx-64 conda create -qnx --dry-run -c QuantStack -c conda-forge cryptopp icu


Package plan for installation in environment /home/maba/code/conda/conda-4.3/envs/x:

The following NEW packages will be INSTALLED:

    cling-patches: 1-0           QuantStack 
    cryptopp:      5.6.5-cling_0 QuantStack  [cling]
    icu:           58.2-0        conda-forge

|$ conda install -qy conda=4.3.33


Package plan for installation in environment /home/maba/code/conda/conda-4.3:

The following packages will be UPDATED:

    conda: 4.3.32-py36_0 --> 4.3.33-py36_0

|$ CONDA_SUBDIR=osx-64 conda create -qnx --dry-run -c QuantStack -c conda-forge cryptopp icu


Package plan for installation in environment /home/maba/code/conda/conda-4.3/envs/x:

The following NEW packages will be INSTALLED:

    clangdev:      5.0.0-cling_1 QuantStack  [cling]
    cling-patches: 1-0           QuantStack 
    cryptopp:      5.6.5-cling_0 QuantStack  [cling]
    icu:           58.2-0        conda-forge
    libiconv:      1.15-0        conda-forge
    libxml2:       2.9.7-0       conda-forge
    llvmdev:       5.0.0-cling_1 QuantStack  [cling]
    xz:            5.2.3-0       conda-forge
    zlib:          1.2.11-0      conda-forge
```
(uses `CONDA_SUBDIR=osx-64` as I'm on Linux).
To reproduce it, all parts of `-c QuantStack -c conda-forge cryptopp icu` are needed.
@mcg1969 has told me that he probably supports reverting #6766 in a conda 4.3.34 release.  Do you agree @mbargull?
That breaks my heart 😢. That PR improved the situation on previously failing cases a lot.
Reverting that #6766 would mean we are back to square one for https://github.com/conda/conda/issues/6269#issuecomment-344077631.
I still agree with
https://github.com/conda/conda/issues/6833#issuecomment-362604480:
> Features need to die a fiery death. They simply are not SAT representable as conceived :-(

https://github.com/conda/conda/pull/6296#issuecomment-347257443:
> We have discussed this at length internally. @mcg1969 is very hesitant to change anything about how features work. There's a lot of pain there, and his opinion is that it's very hard to anticipate any potential breakage from this PR.

But still, I had hoped those latest changes were to stay 😞.
Well, for a short time, people who experienced regressions can pin conda to an earlier version. But we need to figure something out soon.
We did just discover something useful that may help us form a solution. It turns out that just using a `track_feature` _with no corresponding `feature` packages_ has a useful interpretation for establishing preferences. I'm trying to see if we can reproduce that useful quality.
conda-forge relies a lot on build featutes for MSVC flavors. We do a lot at QuantStack for flavors of builds of llvm (for cling which requires patches).

In another case, we have used it to overcome problems with channel priorities not being respected, ehich appears to have been solved since but a lot of our builds still have the features.
Maybe a compromise would be to revert it in 4.3 but leave it in 4.4? Idk. 

Sent from my iPhone

> On Feb 7, 2018, at 7:32 AM, Sylvain Corlay <notifications@github.com> wrote:
> 
> conda-forge relies a lot on build featutes for MSVC flavors. We do a lot at QuantStack for flavors of builds of llvm (for cling which requires patches).
> 
> In another case, we have used it to overcome problems with channel priorities not being respected, ehich appears to have been solved since but a lot of our builds still have the features.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 

May be @mwcraig @isuruf @pelson @jakirkham have an opinion on this. If it is a backward incompatible change, it would probably deserve a `5.0`.

Is there a good alternative to build features at the moment?
> Maybe a compromise would be to revert it in 4.3 but leave it in 4.4? Idk.

`conda-forge` and `bioconda` both benefit from `4.3.33` in some cases (and still need some time to update to 4.4)...

> If it is a backward incompatible change, it would probably deserve a `5.0`.

In regards to solver logic, there were already some changes in 4.x (channel priority, timestamps) -- so it's not the case that 4.x was stable in that regard anyway... (That's just mean as an objective and *not* suggestive comment.)
@SylvainCorlay, I haven't yet completely assessed the severity of the issue `4.3.33` introduced for you. Would that be a workflow-breaking change or could it be (in a broader sense) seen as an "inconvenience". I know that the introduction of additional packages (esp. during build) is undesirable, but the question is if that is workflow breaking or not.
This is not to downplay the issue, just to put it into perspective to the issues #6766 helped with (some of them were workflow-breaking), so we can decide, based on severity, what we should do.


Resolving patch:
diff --git a/conda/resolve.py b/conda/resolve.py
--- a/conda/resolve.py
+++ b/conda/resolve.py
@@ -566,13 +566,21 @@ def get_dists_for_spec(self, ms, emptyok=False):
         return sorted(dists, key=self.version_key)
 
     @staticmethod
-    def ms_to_v(ms):
+    def to_ms_clause_id(ms):
         ms = MatchSpec(ms)
         return '@s@' + ms.spec + ('?' if ms.optional else '')
 
+    @staticmethod
+    def to_notd_ms_clause_id(ms):
+        return '!@s@'+ms.name
+
+    @staticmethod
+    def to_feature_metric_id(dist, feat):
+        return '@fm@%s@%s' % (dist, feat)
+
     def push_MatchSpec(self, C, ms):
         ms = MatchSpec(ms)
-        name = self.ms_to_v(ms)
+        name = self.to_ms_clause_id(ms)
         m = C.from_name(name)
         if m is not None:
             return name
@@ -588,7 +596,7 @@ def push_MatchSpec(self, C, ms):
         if m is None:
             libs = [dist.full_name for dist in libs]
             if ms.optional:
-                libs.append('!@s@'+ms.name)
+                libs.append(self.to_notd_ms_clause_id(ms))
             m = C.Any(libs)
         C.name_var(m, name)
         return name
@@ -601,7 +609,7 @@ def gen_clauses(self):
             for fkey in group:
                 C.new_var(fkey)
             # Create one variable for the group
-            m = C.new_var(self.ms_to_v(name))
+            m = C.new_var(self.to_ms_clause_id(name))
             # Exactly one of the package variables, OR
             # the negation of the group variable, is true
             C.Require(C.ExactlyOne, group + [C.Not(m)])
@@ -624,20 +632,22 @@ def generate_update_count(self, C, specs):
 
     def generate_feature_metric(self, C):
         eq = {}  # a C.minimize() objective: Dict[varname, coeff]
-        total = 0
+        # Given a pair (dist, feature), assign a "1" score IF:
+        # - The dist is installed
+        # - The dist does NOT require the feature
+        # - At least one package in the group DOES require the feature
+        # - A package that tracks the feature is installed
         for name, group in iteritems(self.groups):
-            nf = [len(self.features(dist)) for dist in group]
-            maxf = max(nf)
-            if maxf > 0:
-                eq.update({dist.full_name: maxf-fc for dist, fc in zip(group, nf) if maxf > fc})
-                # This entry causes conda to weight the absence of a package the same
-                # as if a non-featured version were instaslled. If this were not here,
-                # conda will sometimes select an earlier build of a requested package
-                # if doing so can eliminate this package as a dependency.
-                # https://github.com/conda/conda/issues/6765
-                eq['!' + self.push_MatchSpec(C, name)] = maxf
-            total += maxf
-        return eq, total
+            dist_feats = {dist.full_name: self.features(dist) for dist in group}
+            active_feats = set.union(*dist_feats.values()).intersection(self.trackers)
+            for feat in active_feats:
+                clause_id_for_feature = self.push_MatchSpec(C, '@' + feat)
+                for dist, features in dist_feats.items():
+                    if feat not in features:
+                        feature_metric_id = self.to_feature_metric_id(dist, feat)
+                        C.name_var(C.And(dist, clause_id_for_feature), feature_metric_id)
+                        eq[feature_metric_id] = 1
+        return eq
 
     def generate_removal_count(self, C, specs):
         return {'!'+self.push_MatchSpec(C, ms.name): 1 for ms in specs}
@@ -935,11 +945,16 @@ def mysat(specs, add_if=False):
             solution, obj1 = C.minimize(eq_feature_count, solution)
             log.debug('Track feature count: %d', obj1)
 
-            # Featured packages: maximize featured package count
-            eq_feature_metric, ftotal = r2.generate_feature_metric(C)
+            # Featured packages: minimize number of featureless packages
+            # installed when a featured alternative is feasible.
+            # For example, package name foo exists with two built packages. One with
+            # 'track_features: 'feat1', and one with 'track_features': 'feat2'.
+            # The previous "Track features" minimization pass has chosen 'feat1' for the
+            # environment, but not 'feat2'. In this case, the 'feat2' version of foo is
+            # considered "featureless."
+            eq_feature_metric = r2.generate_feature_metric(C)
             solution, obj2 = C.minimize(eq_feature_metric, solution)
-            obj2 = ftotal - obj2
-            log.debug('Package feature count: %d', obj2)
+            log.debug('Package misfeature count: %d', obj2)
 
             # Requested packages: maximize builds
             solution, obj4 = C.minimize(eq_req_b, solution)


Test patch:
diff --git a/tests/test_resolve.py b/tests/test_resolve.py
--- a/tests/test_resolve.py
+++ b/tests/test_resolve.py
@@ -1395,3 +1395,77 @@ def test_update_deps():
         'tk-8.5.13-0.tar.bz2',
         'zlib-1.2.7-0.tar.bz2',
     ]]]
+
+
+def test_surplus_features_1():
+    index = {
+        'feature-1.0-0.tar.bz2': IndexRecord(**{
+            'name': 'feature',
+            'version': '1.0',
+            'build': '0',
+            'build_number': 0,
+            'track_features': 'feature',
+        }),
+        'package1-1.0-0.tar.bz2': IndexRecord(**{
+            'name': 'package1',
+            'version': '1.0',
+            'build': '0',
+            'build_number': 0,
+            'features': 'feature',
+        }),
+        'package2-1.0-0.tar.bz2': IndexRecord(**{
+            'name': 'package2',
+            'version': '1.0',
+            'build': '0',
+            'build_number': 0,
+            'depends': ['package1'],
+            'features': 'feature',
+        }),
+        'package2-2.0-0.tar.bz2': IndexRecord(**{
+            'name': 'package2',
+            'version': '2.0',
+            'build': '0',
+            'build_number': 0,
+            'features': 'feature',
+        }),
+    }
+    r = Resolve({Dist(key): value for key, value in iteritems(index)})
+    install = r.install(['package2', 'feature'])
+    assert 'package1' not in set(d.name for d in install)
+
+
+def test_surplus_features_2():
+    index = {
+        'feature-1.0-0.tar.bz2': IndexRecord(**{
+            'name': 'feature',
+            'version': '1.0',
+            'build': '0',
+            'build_number': 0,
+            'track_features': 'feature',
+        }),
+        'package1-1.0-0.tar.bz2': IndexRecord(**{
+            'name': 'package1',
+            'version': '1.0',
+            'build': '0',
+            'build_number': 0,
+            'features': 'feature',
+        }),
+        'package2-1.0-0.tar.bz2': IndexRecord(**{
+            'name': 'package2',
+            'version': '1.0',
+            'build': '0',
+            'build_number': 0,
+            'depends': ['package1'],
+            'features': 'feature',
+        }),
+        'package2-1.0-1.tar.bz2': IndexRecord(**{
+            'name': 'package2',
+            'version': '1.0',
+            'build': '1',
+            'build_number': 1,
+            'features': 'feature',
+        }),
+    }
+    r = Resolve({Dist(key): value for key, value in iteritems(index)})
+    install = r.install(['package2', 'feature'])
+    assert 'package1' not in set(d.name for d in install)