Repository: conda/conda

Issue:
Having trouble downloading from osx-64 on Travis CI
_From @jakirkham on June 14, 2016 20:46_

Seems we are now having issues downloading on packages on Travis CI. See the snippet below and this [log](https://travis-ci.org/conda-forge/apache-libcloud-feedstock/jobs/137608834#L369-L376). Note none of the packages below are on conda-forge (there is `python`, but not 2.7). So, this seems to be a `defaults` channel issue. Log snippet below.

```
BUILD START: apache-libcloud-0.18.0-py27_1
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata .......
WARNING: Could not find any versions of conda-build in the channels
Solving package specifications ....
Error: Packages missing in current osx-64 channels: 
  - python 2.7*
  - setuptools
```

Also, see these PRs for examples
- https://github.com/conda-forge/apache-libcloud-feedstock/pull/3
- https://github.com/conda-forge/colorlog-feedstock/pull/4
- https://github.com/conda-forge/python-feedstock/pull/20

cc @msarahan @danielfrg @frol @jjhelmus @vamega

_Copied from original issue: Anaconda-Platform/support#45_



Discussion:
_From @jjhelmus on June 14, 2016 20:46_

What happens if you pin conda to version 4.0.8?

_From @jakirkham on June 14, 2016 20:47_

It's a good question. The thing is I can't reproduce this locally (even with `conda` 4.1.0). So, if it is a `conda` bug, it is a pretty special one.

_From @jjhelmus on June 14, 2016 20:50_

I'm trying in the conda-forge/python-feedstock#20.  I think the windows issue that I'm seeing may be related:

```
BUILD END: python-3.5.1-0
TEST START: python-3.5.1-0
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata ....... 
WARNING: Could not find any versions of conda-build in the channels
Solving package specifications .... 
Error: Dependencies missing in current win-32 channels: 
  - python 3.5.1 0 -> pip
  - python 3.5* -> pip 
Command exited with code 1
```

_From @jakirkham on June 14, 2016 20:52_

So, there was an issue like that between `conda-build` and `conda-build-all` that affected staged-recipes, but I don't see why that would affect a feedstock as they don't use `conda-build-all` in the CIs. 😕

_From @mcg1969 on June 14, 2016 20:53_

That's a strange one. Folks are looking into it over here; and if we have to pull 4.1, we will.

_From @jjhelmus on June 14, 2016 20:58_

Seems to be related to 4.1, downgrading to 4.0.8 fixes the osx issue for me.
- [conda 4.0.8 python build](https://travis-ci.org/conda-forge/python-feedstock/builds/137630248).
- [conda 4.1.0 python build](https://travis-ci.org/conda-forge/python-feedstock/builds/137621445).

I don't have a Mac available to try reproducing this locally.

_From @jakirkham on June 14, 2016 21:4_

So, we tried to downgrade, but it appears we [cannot](https://travis-ci.org/conda-forge/apache-libcloud-feedstock/jobs/137631775#L369-L370) if we are already on `conda` 4.1.0. In other words, this doesn't seem to be related to `conda-build`.

_From @jakirkham on June 14, 2016 21:5_

Though I cannot reproduce this locally. So, it is something special about Travis CI and `conda` that we need to disentangle.

_From @mcg1969 on June 14, 2016 21:9_

Thanks for the investigation. I can't reproduce locally, either. @kalefranz

_From @jakirkham on June 14, 2016 21:15_

Can confirm that if we downgrade to 4.0.8 (on Travis CI). We do not experience this problem. See these [logs](https://travis-ci.org/conda-forge/apache-libcloud-feedstock/builds/137634752).

_From @jakirkham on June 14, 2016 21:19_

We downgraded before it was upgraded to `conda` 4.1.0 (was on 4.0.5) and made sure it didn't upgrade to 4.1.0 (kept to 4.0.8). Of course, if it went to `conda` 4.1.0, there was no recourse as shown earlier.

_From @jakirkham on June 14, 2016 21:31_

Sorry, that is probably even less clear. How about a diagram?

![conda](https://cloud.githubusercontent.com/assets/3019665/16060465/d7f630b4-3255-11e6-80c6-e227da5c057e.png)

_From @mcg1969 on June 14, 2016 21:40_

@jakirkham @jjhelmus are you seeing failures on Linux too, or Mac only? Trying to figure out if we should pull both OSs.

_From @jjhelmus on June 14, 2016 21:40_

I'm only seeing issues on Mac and Windows (both win-32 and win-64)

_From @mcg1969 on June 14, 2016 21:42_

Ah, Windows too? Linux seems to be working fine? OK.

https://travis-ci.org/conda-forge/python-feedstock/builds/137621445#L258

What happens if you update conda-build to latest?

_From @jakirkham on June 14, 2016 21:50_

So, it seems unrelated to `conda-build` based on this [find](https://github.com/Anaconda-Platform/support/issues/45#issuecomment-226015810), @kalefranz. Basically, we cannot install things once `conda` 4.1.0 is installed on Travis CI.

Hey guys,  could you try adding this config to your setup...

```
anaconda config --set url https://api.anaconda.org
```

If that fixes things for you, we know where the bug is.

_From @jakirkham on June 14, 2016 21:51_

I don't think so. Do we need to do this now?

_From @mcg1969 on June 14, 2016 21:51_

We found one user that could reproduce a similar bug locally, and this was it.

_From @jjhelmus on June 14, 2016 21:51_

Pulled out the [Python recipe](https://travis-ci.org/jjhelmus/conda_410_issue_debug) to try to make debugging easier

_From @mcg1969 on June 14, 2016 21:52_

But we're going to hotfix this one

_From @mcg1969 on June 14, 2016 21:53_

I don't think that's the problem in this case, @kalefranz, because their logs seem to show that they're already using https.

_From @jakirkham on June 14, 2016 21:55_

Did the setting move locations or something? Is there some reason it might not pick it up after the upgrade?

_From @mcg1969 on June 14, 2016 22:0_

I introduced a regression in https://github.com/conda/conda/pull/2564 when fixing the issues mentioned therein. Our local user was seeing a very similar behavior to yours here, but his `anaconda-client` configuration was pointing to `http://api.anaconda.org` instead of `https://api.anaconda.org`. Fixing his configuration corrected the problem (and we will be patching conda to fix the http/https issue).

But since your logs suggest you're already using `https://api.anaconda.org`, I'm not seeing this as being a problem. I wonder though if you even need to install `anaconda-client`; are you relying on private Anaconda Cloud repos?

_From @jjhelmus on June 14, 2016 22:1_

> https://travis-ci.org/conda-forge/python-feedstock/builds/137621445#L258
> 
> What happens if you update conda-build to latest?

[No luck](https://travis-ci.org/jjhelmus/conda_410_issue_debug/builds/137646381)

_From @mcg1969 on June 14, 2016 22:9_

We're zeroing in on the issue---different than the one above

https://travis-ci.org/conda-forge/python-feedstock/builds/137621445#L256

Can you move that line AFTER the `conda config --add channels conda-forge` line?
https://travis-ci.org/conda-forge/python-feedstock/builds/137621445#L259

So...

```
conda update --yes conda
conda install --yes conda-build=1.20.0 jinja2 anaconda-client
conda config --add channels conda-forge
conda config --set show_channel_urls true
```

Just trying to isolate the problem...

When the ~/.condarc file exists, but there's not a `channels` field, it seems `conda config --add channels conda-forge` _just_ adds the conda-forge channel instead of adding both conda-forge and defaults.

When the file doesn't exist and it's newly created, the behavior is as expected.

_From @jjhelmus on June 14, 2016 22:15_

That changed is queued in the [conda_410_issue_debug](https://travis-ci.org/jjhelmus/conda_410_issue_debug/builds) builds.

_From @jjhelmus on June 14, 2016 22:16_

And that looks to be the [issue](https://travis-ci.org/jjhelmus/conda_410_issue_debug/builds/137650353)

Cool!  We'll get a patch out soon. 4.0.1.  But at least you can go on with your CI with the order switched up if you want.

Or just hold tight for a patch.

_From @jjhelmus on June 14, 2016 22:33_

Yes, looks like if `conda config --set show_channel_urls true` is executed before `conda config --add channels conda-forge` then the resulting .condarc does not include the defaults channel, [build](https://travis-ci.org/jjhelmus/conda_410_issue_debug/builds/137651446):

```
show_channel_urls: true
channels:
  - conda-forge
```

If the channel is added before setting show_channel_urls, then the .condarc does include the default channel, [build](https://travis-ci.org/jjhelmus/conda_410_issue_debug/builds/137651370):

```
channels:
  - conda-forge
  - defaults
show_channel_urls: true
```

conda 4.0.8 did not have this behavior, [build](https://travis-ci.org/jjhelmus/conda_410_issue_debug/builds/137653591)

Looking forward to the fix but glad to know the workaround.

_From @jakirkham on June 14, 2016 22:40_

Great detective work guys. Glad we have narrowed this down. 👍


Resolving patch:
diff --git a/conda/cli/main_config.py b/conda/cli/main_config.py
--- a/conda/cli/main_config.py
+++ b/conda/cli/main_config.py
@@ -218,26 +218,21 @@ def execute_config(args, parser):
     else:
         rc_path = user_rc_path
 
-    # Create the file if it doesn't exist
-    if not os.path.exists(rc_path):
-        has_defaults = ['channels', 'defaults'] in args.add
-        if args.add and 'channels' in list(zip(*args.add))[0] and not has_defaults:
-            # If someone adds a channel and their .condarc doesn't exist, make
-            # sure it includes the defaults channel, or else they will end up
-            # with a broken conda.
-            rc_text = """\
-channels:
-  - defaults
-"""
-        else:
-            rc_text = ""
+    # read existing condarc
+    if os.path.exists(rc_path):
+        with open(rc_path, 'r') as fh:
+            rc_config = yaml_load(fh)
     else:
-        with open(rc_path, 'r') as rc:
-            rc_text = rc.read()
-    rc_config = yaml_load(rc_text)
-    if rc_config is None:
         rc_config = {}
 
+    # add `defaults` channel if creating new condarc file or channel key doesn't exist currently
+    if 'channels' not in rc_config:
+        # now check to see if user wants to modify channels at all
+        if any('channels' in item[0] for item in args.add):
+            # don't need to insert defaults if it's already in args
+            if not ['channels', 'defaults'] in args.add:
+                args.add.insert(0, ['channels', 'defaults'])
+
     # Get
     if args.get is not None:
         if args.get == []:


Test patch: