Repository: conda/conda-build

Issue:
Test environment incorrectly prioritizes local channel
Scenario:

- Developer is working on two conda packages, "pkgA" and "pkgB".
- Developer builds pkgA at version 1.0
- Developer uploads pkgA=1.0 to conda channel
- Second developer (or the same developer on a different machine) works on pkgA at version 2.0.
- Second developer uploads pkgA=2.0 to conda channel
- Developer starts work on pkgB, which has a dependency on pkgA, and requires functionality only present in pkgA >= 2.0.
- Developer attempts to build pkgB but unit tests fail.

Actual behavior:

The test environment is pulling pkgA=1.0 from the local conda-bld folder, instead of the newer pkgA=2.0 from their conda channel. This causes tests to fail and the build is broken.

Expected behavior:

The test environment pulls from the user's defined set of channels before it falls back to the local channel. This will cause pkgA=2.0 to be pulled as expected and unit tests to pass.


Discussion:
This is kind of contentious.  It is the way that it is because building has historically not been such a distributed activity.  If remote channels are higher in priority, local packages will almost never be used to fulfill dependencies.  They would only be used if a remote package didn't exist, or was otherwise unsatisfiable.

It is more intuitive to me that local be higher priority by default.  If you'd like to change that, I recommend adding ``-c yourchannel``, which I hope will take priority over local.  Unfortunately, I think local is defined as the top priority, according to https://github.com/conda/conda-build/blob/master/conda_build/index.py#L65

I'm not sure how accurate that comment actually is, though.  Give it a shot?
That comment appears to be correct, i.e. adding "-c mychannel" will NOT
cause mychannel to be prioritized over local packages. Personally, I would
expect it to be the case that "local packages will almost never be used to
fulfill dependencies" during a build (outside of the package actually being
built), but I can also understand not wanting to change the default
behavior. I would be ok with "-c mychannel -c local" as a workaround if
that actually worked. Actually what would be really cool is if we could
provide a custom environment file that conda-build would use (and
additionally install the just built package into) for creating the test
environment.

On Fri, Jul 20, 2018, 2:52 PM Mike Sarahan <notifications@github.com> wrote:

> This is kind of contentious. It is the way that it is because building has
> historically not been such a distributed activity. If remote channels are
> higher in priority, local packages will almost never be used to fulfill
> dependencies. They would only be used if a remote package didn't exist, or
> was otherwise unsatisfiable.
>
> It is more intuitive to me that local be higher priority by default. If
> you'd like to change that, I recommend adding -c yourchannel, which I
> hope will take priority over local. Unfortunately, I think local is defined
> as the top priority, according to
> https://github.com/conda/conda-build/blob/master/conda_build/index.py#L65
>
> I'm not sure how accurate that comment actually is, though. Give it a shot?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/conda/conda-build/issues/3036#issuecomment-406695196>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AGCCk4hdqxfzcYyzzNs0S7uukJum3Oh1ks5uIibQgaJpZM4VYoeE>
> .
>


Resolving patch:
diff --git a/conda_build/index.py b/conda_build/index.py
--- a/conda_build/index.py
+++ b/conda_build/index.py
@@ -62,8 +62,6 @@ def get_build_index(subdir, bldpkgs_dir, output_folder=None, clear_cache=False,
 
         log.debug("Building new index for subdir '{}' with channels {}, condarc channels "
                   "= {}".format(subdir, channel_urls, not omit_defaults))
-        # priority: local by croot (can vary), then channels passed as args,
-        #     then channels from config.
         capture = contextlib.contextmanager(lambda: (yield))
         if debug:
             log_context = partial(utils.LoggingContext, logging.DEBUG)
@@ -73,9 +71,18 @@ def get_build_index(subdir, bldpkgs_dir, output_folder=None, clear_cache=False,
             log_context = partial(utils.LoggingContext, logging.CRITICAL + 1)
             capture = utils.capture
 
+        # priority: (local as either croot or output_folder IF NOT EXPLICITLY IN CHANNEL ARGS),
+        #     then channels passed as args (if local in this, it remains in same order),
+        #     then channels from condarc.
         urls = list(channel_urls)
+
+        # this is where we add the "local" channel.  It's a little smarter than conda, because
+        #     conda does not know about our output_folder when it is not the default setting.
         if os.path.isdir(output_folder):
-            urls.insert(0, url_path(output_folder))
+            if 'local' not in urls:
+                urls.insert(0, url_path(output_folder))
+            # replace local with the appropriate real channel.  Order is maintained.
+            urls = [url if url != "local" else url_path(output_folder) for url in urls]
         _ensure_valid_channel(output_folder, subdir, verbose=verbose, locking=locking,
                               timeout=timeout)
 
diff --git a/conda_build/utils.py b/conda_build/utils.py
--- a/conda_build/utils.py
+++ b/conda_build/utils.py
@@ -29,7 +29,7 @@
 from .conda_interface import hashsum_file, md5_file, unix_path_to_win, win_path_to_unix
 from .conda_interface import PY3, iteritems
 from .conda_interface import root_dir, pkgs_dirs
-from .conda_interface import string_types, url_path, get_rc_urls
+from .conda_interface import string_types
 from .conda_interface import memoized
 from .conda_interface import StringIO
 from .conda_interface import VersionOrder, MatchSpec
@@ -1163,18 +1163,6 @@ def env_var(name, value, callback=None):
             callback()
 
 
-def collect_channels(config, is_host=False):
-    urls = [url_path(config.croot)] + get_rc_urls() + ['local', ]
-    if config.channel_urls:
-        urls.extend(config.channel_urls)
-    # defaults has a very limited set of repo urls.  Omit it from the URL list so
-    #     that it doesn't fail.
-    if config.is_cross and is_host:
-        urls.remove('defaults')
-        urls.remove('local')
-    return urls
-
-
 def trim_empty_keys(dict_):
     to_remove = set()
     negative_means_empty = ('final', 'noarch_python', 'zip_keys')


Test patch: