Repository: conda/conda-build

Issue:
--skip-existing flag not working on linux, version 1.21.3
I am only seeing this issue on Linux (not OS X) and it just appeared when upgrading to conda build version 1.21 (1.21.3 specifically).  Additionally, I am using a S3 bucket as my channel.  'conda search package_name' correctly locates the package, but if I issue a 'conda build --skip-existing path/to/recipe' it always rebuilds.  The recipe version is the same as the existing version.

I will work on trying to get a more concrete example using a public package.



Discussion:
Interesting.  We test that functionality explicitly here: https://github.com/conda/conda-build/blob/1.21.x/tests/test_build_recipes.py#L276-L286

Any info you can provide on why that test isn't catching misbehavior you're seeing would be great.  If possible, could you try to run that unit test from the conda-build source?

To be clear, the S3 bucket works OK on OS X, but not Linux?  Or is the S3 bucket only used on one of the platforms?

Ah.  I did a bit more careful digging.  It seems like its also not working on OS X, I just had a local copy cached when I tested that.  My guess it has something to do with the S3 channel not being searched correctly when using the --skip-existing flag.  As mentioned, packages are resolving correctly in the s3 channel when doing a 'conda search package_name' however.

So overall what ive seen:
1.  --skip-existing works correctly when I have a local build of the recipe to resolve against
2.  --skip-existing does NOT work correctly when resolving against an S3 channel (on both OS X and Linux)

In the test case you pointed it out, it appears to only test against scenario 1, so that is at least consistent.

OK, need more info.  The code in conda-build that does this is:

https://github.com/conda/conda-build/blob/master/conda_build/main_build.py#L246-L251
and
https://github.com/conda/conda-build/blob/master/conda_build/main_build.py#L296-L302

these build the index with conda's code here:
https://github.com/conda/conda/blob/4.1.x/conda/api.py#L10-L46

nothing looks obviously wrong to me here.  I think we need to look at your S3 channel, and make sure that conda is recognizing it properly.  I know you said that search works - that's a good start.  What's the output of `conda info`?  Can you open a Python interpreter and try:

```
import conda.api as api
index = api.get_index(use_local=True)
```

The keys of index are package names.  Some of them may have prefixes like local::packagename.  The bug here (if it isn't in your S3 channel) is probably in a prefix that conda-build does not handle correctly.  Please search through these keys for your package name.  Something like this might work:

```
for key in index.keys():
    if "yourpackagename" in key:
       print("found package: " + key)
```

In the meantime, I will add a test that uploads a package to an anaconda.org channel, and uses skip-existing after deleting that built package locally.

Thanks for looking into this.
Here is my output from conda info on OS X:

```
             platform : osx-64
        conda version : 4.1.6
    conda-env version : 2.5.1
  conda-build version : 1.21.3
       python version : 2.7.12.final.0
     requests version : 2.9.1
     root environment : /Users/anthony/miniconda  (writable)
  default environment : /Users/anthony/miniconda
     envs directories : /Users/anthony/miniconda/envs
        package cache : /Users/anthony/miniconda/pkgs
         channel URLs : s3://conda-packages/osx-64/
                        s3://conda-packages/noarch/
                        https://repo.continuum.io/pkgs/free/osx-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/osx-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : /Users/anthony/.condarc
         offline mode : False
    is foreign system : False
```

Here are the keys in the index for an example package 'pika' (when I also have it built locally):

```
package local::pika-0.10.0-py27_0.tar.bz2
package s3://conda-packages::pika-0.10.0-py27_0.tar.bz2
```

Thanks.  Yes, it does look like the s3://... prefix is not going to work with the code as is.  I need to think about how to generalize this, but will do so soon.


Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -334,12 +334,13 @@ def create_info_files(m, files, include_recipe=True):
                         join(config.info_dir, 'icon.png'))
 
 
-def get_build_index(clear_cache=True):
+def get_build_index(clear_cache=True, arg_channels=None):
     if clear_cache:
         # remove the cache such that a refetch is made,
         # this is necessary because we add the local build repo URL
         fetch_index.cache = {}
-    return get_index(channel_urls=[url_path(config.croot)] + list(channel_urls),
+    arg_channels = [] if not arg_channels else arg_channels
+    return get_index(channel_urls=[url_path(config.croot)] + list(channel_urls) + arg_channels,
                      prepend=not override_channels)
 
 
@@ -475,7 +476,8 @@ def build(m, post=None, include_recipe=True, keep_old_work=False,
                 vcs_source = m.uses_vcs_in_build()
                 if vcs_source and vcs_source not in specs:
                     vcs_executable = "hg" if vcs_source == "mercurial" else vcs_source
-                    has_vcs_available = os.path.isfile(external.find_executable(vcs_executable) or "")
+                    has_vcs_available = os.path.isfile(external.find_executable(vcs_executable) or
+                                                       "")
                     if not has_vcs_available:
                         if (vcs_source != "mercurial" or
                                 not any(spec.startswith('python') and "3." in spec
diff --git a/conda_build/index.py b/conda_build/index.py
--- a/conda_build/index.py
+++ b/conda_build/index.py
@@ -76,7 +76,7 @@ def update_index(dir_path, verbose=False, force=False, check_md5=False, remove=T
         sys.exit("""\
 Error:
     Indexing a copy of the Anaconda conda package channel is neither
-    necessary nor supported.  If you which to add your own packages,
+    necessary nor supported.  If you wish to add your own packages,
     you can do so by adding them to a separate channel.
 """)
     for fn in files:
diff --git a/conda_build/main_build.py b/conda_build/main_build.py
--- a/conda_build/main_build.py
+++ b/conda_build/main_build.py
@@ -149,9 +149,12 @@ def handle_binstar_upload(path, args):
     from conda_build.external import find_executable
 
     upload = False
+    if args.token or args.user:
+        args.yes = True
+        upload = True
     # this is the default, for no explicit argument.
     # remember that args.binstar_upload takes defaults from condarc
-    if args.binstar_upload is None:
+    elif args.binstar_upload is None:
         args.yes = False
         args.dry_run = False
     # rc file has uploading explicitly turned off
@@ -220,6 +223,8 @@ def execute(args, parser):
     from os import makedirs
     from os.path import abspath, isdir, isfile
 
+    import conda.config as cc
+
     import conda_build.build as build
     import conda_build.source as source
     from conda_build.config import config
@@ -252,7 +257,11 @@ def execute(args, parser):
             if not isdir(d):
                 makedirs(d)
             update_index(d)
-        index = build.get_build_index(clear_cache=True)
+        arg_channels = ['local']
+        if args.channel:
+            arg_channels.extend(args.channel)
+        index = build.get_build_index(clear_cache=True,
+                                      arg_channels=arg_channels)
 
     already_built = set()
     to_build_recursive = []
@@ -298,11 +307,14 @@ def execute(args, parser):
                     "configuration." % m.dist())
             continue
         if args.skip_existing:
-            # 'or m.pkg_fn() in index' is for conda <4.1 and could be removed in the future.
-            if ('local::' + m.pkg_fn() in index or
-                    m.pkg_fn() in index or
-                    m.pkg_fn() in already_built):
-                print(m.dist(), "is already built, skipping.")
+            urls = cc.get_rc_urls() + cc.get_local_urls() + ['local', ]
+            if args.channel:
+                urls.extend(args.channel)
+
+            # will be empty if none found, and evalute to False
+            package_exists = [url for url in urls if url + '::' + m.pkg_fn() in index]
+            if (package_exists or m.pkg_fn() in index or m.pkg_fn() in already_built):
+                print(m.dist(), "is already built in {0}, skipping.".format(package_exists))
                 continue
         if args.output:
             print(bldpkg_path(m))


Test patch:
diff --git a/tests/test_build_recipes.py b/tests/test_build_recipes.py
--- a/tests/test_build_recipes.py
+++ b/tests/test_build_recipes.py
@@ -10,13 +10,28 @@
 from conda.compat import PY3, TemporaryDirectory
 from conda.config import subdir
 from conda.fetch import download
+
 from conda_build.source import _guess_patch_strip_level, apply_patch
+import conda_build.config as config
+
+if PY3:
+    import urllib.parse as urlparse
+    import urllib.request as urllib
+else:
+    import urlparse
+    import urllib
 
 thisdir = os.path.dirname(os.path.realpath(__file__))
 metadata_dir = os.path.join(thisdir, 'test-recipes', 'metadata')
 fail_dir = os.path.join(thisdir, 'test-recipes', 'fail')
 
 
+# Used for translating local paths into url (file://) paths
+#   http://stackoverflow.com/a/14298190/1170370
+def path2url(path):
+    return urlparse.urljoin('file:', urllib.pathname2url(path))
+
+
 def is_valid_dir(parent_dir, dirname):
     valid = os.path.isdir(os.path.join(parent_dir, dirname))
     valid &= not dirname.startswith("_")
@@ -297,23 +312,100 @@ def test_skip_existing():
                                                                                     "build_number"))
     process = subprocess.Popen(cmd.split(),
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    output, _ = process.communicate()
+    output, error = process.communicate()
     output = output.decode('utf-8')
-    assert "is already built, skipping." in output
+    error = error.decode('utf-8')
+    assert "is already built" in output, error
+
+
+# def test_skip_existing_anaconda_org():
+#     """This test may give false errors, because multiple tests running in parallel (on different
+#     platforms) will all use the same central anaconda.org account.  Thus, this test is only reliable
+#     if it is being run by one person on one machine at a time."""
+#     # generated with conda_test_account user, command:
+#     #    anaconda auth --create --name CONDA_BUILD_UPLOAD_TEST --scopes 'api repos conda'
+#     token = "co-79de533f-926f-4e5e-a766-d393e33ae98f"
+#     cmd = 'conda build --token {} {}'.format(token, os.path.join(metadata_dir, "empty_sections"))
+#     subprocess.check_call(cmd.split())
+
+#     try:
+#         # ensure that we skip with the package in the anaconda.org channel
+#         cmd = ('conda build --no-anaconda-upload --override-channels '
+#                '-c conda_test_account --skip-existing {}'
+#                 .format(os.path.join(metadata_dir, "empty_sections")))
+#         process = subprocess.Popen(cmd.split(),
+#                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+#         output, error = process.communicate()
+#         output = output.decode('utf-8')
+#         error = error.decode('utf-8')
+#     except:
+#         raise
+#     finally:
+#         # clean up: remove the package
+#         cmd = 'anaconda --token {} remove --force conda_test_account/empty_sections'\
+#             .format(token)
+#         subprocess.check_call(cmd.split())
+
+#     assert "is already built" in output, error
+#     assert "conda_test_account" in output, error
+
+
+def test_skip_existing_url():
+    # make sure that it is built
+    cmd = 'conda build --no-anaconda-upload {}'.format(os.path.join(metadata_dir, "empty_sections"))
+    subprocess.check_call(cmd.split())
+
+    output_file = os.path.join(config.croot, subdir, "empty_sections-0.0-0.tar.bz2")
+
+    with TemporaryDirectory() as tmp:
+        platform = os.path.join(tmp, subdir)
+        os.makedirs(platform)
+        shutil.copy2(output_file, os.path.join(platform, os.path.basename(output_file)))
+
+        # create the index so conda can find the file
+        subprocess.check_call(["conda", "index"], cwd=platform)
+
+        channel_url = path2url(tmp)
+        cmd = ('conda build --override-channels --skip-existing '
+               '--no-anaconda-upload -c {} {}'.format(channel_url,
+                                                      os.path.join(metadata_dir, "empty_sections")))
+        process = subprocess.Popen(cmd.split(),
+                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        output, error = process.communicate()
+        output = output.decode('utf-8')
+        error = error.decode('utf-8')
+
+        assert "is already built" in output, (output + error)
+        assert channel_url in output, (output + error)
 
 
 def test_token_upload():
     # generated with conda_test_account user, command:
     #    anaconda auth --create --name CONDA_BUILD_UPLOAD_TEST --scopes 'api repos conda'
     token = "co-79de533f-926f-4e5e-a766-d393e33ae98f"
+
+    # clean up - we don't actually want this package to exist yet
+    cmd = 'anaconda --token {} remove --force conda_test_account/empty_sections'.format(token)
+    subprocess.check_call(cmd.split())
+
+    show_package = "anaconda show conda_test_account/empty_sections"
+    with pytest.raises(subprocess.CalledProcessError):
+        subprocess.check_call(show_package.split())
     # the folder with the test recipe to upload
     cmd = 'conda build --token {} {}'.format(token, os.path.join(metadata_dir, "empty_sections"))
     subprocess.check_call(cmd.split())
+
+    # make sure that the package is available (should raise CalledProcessError if it doesn't)
+    subprocess.check_call(show_package.split())
+
     # clean up - we don't actually want this package to exist
-    cmd = 'anaconda --token {} remove --force conda_test_account/conda-build-test-empty_sections'\
-    .format(token)
+    cmd = 'anaconda --token {} remove --force conda_test_account/empty_sections'.format(token)
     subprocess.check_call(cmd.split())
 
+    # verify cleanup:
+    with pytest.raises(subprocess.CalledProcessError):
+        subprocess.check_call(show_package.split())
+
 
 @pytest.mark.parametrize("service_name", ["binstar", "anaconda"])
 def test_no_anaconda_upload_condarc(service_name):
@@ -331,7 +423,6 @@ def test_no_anaconda_upload_condarc(service_name):
         output, error = process.communicate()
         output = output.decode('utf-8')
         error = error.decode('utf-8')
-        sys.stderr.write(output)
         assert "Automatic uploading is disabled" in output, error