Repository: conda/conda-build

Issue:
skip-existing still doesn't skip existing packages in 3.0.4
After updating to 3.0.4 the crash documented in #2172 is fixed. Unfortunately the build is still not skipping existing packages, it just starts to build.. that functionality worked fine in 2.1.17.


Discussion:
We do have tests for this, but you're hitting a path that seems like some
parts of a package are built, but others are not. Can you post a minimal
reproducing recipe?

On Jul 12, 2017 2:55 AM, "gabm" <notifications@github.com> wrote:

> After updating to 3.0.4 the crash documented in #2172
> <https://github.com/conda/conda-build/issues/2172> is fixed.
> Unfortunately the build is still not skipping existing packages, it just
> starts to build.. that functionality worked fine in 2.1.17.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/conda/conda-build/issues/2187>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACV-W7XlYhYOjim5zQiSFBJMjDZ2vREks5sNHwFgaJpZM4OVQa6>
> .
>

well its as easy as:

```
package:
  name: "testpkg"
  version: "1.0.0"
```

just build it twice with skip existing... in conda 3.0.4 it says:

```
$ conda build . --skip-existing
source tree in: /home/gabm/miniconda3/conda-bld/testpkg_1499864529470/work
Attempting to finalize metadata for testpkg
INFO:conda_build.metadata:Attempting to finalize metadata for testpkg
BUILD START: ['testpkg-1.0.0-hbf21a9e_0.tar.bz2']
Packaging testpkg
INFO:conda_build.build:Packaging testpkg
Packaging testpkg-1.0.0-hbf21a9e_0
INFO:conda_build.build:Packaging testpkg-1.0.0-hbf21a9e_0
number of files: 0
Fixing permissions
Fixing permissions
updating: testpkg-1.0.0-hbf21a9e_0.tar.bz2
Nothing to test for: /home/gabm/miniconda3/conda-bld/linux-64/testpkg-1.0.0-hbf21a9e_0.tar.bz2
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /home/gabm/miniconda3/conda-bld/linux-64/testpkg-1.0.0-hbf21a9e_0.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []



####################################################################################
Source and build intermediates have been left in /home/gabm/miniconda3/conda-bld.
There are currently 85 accumulated.
To remove them, you can run the ```conda build purge``` command
```

while in 2.1.17 it says:

```
 gabm@pc-klosi /t/test  $ conda build . --skip-existing
updating index in: /home/gabm/miniconda3/conda-bld/linux-64
updating index in: /home/gabm/miniconda3/conda-bld/noarch
Fetching package metadata .................
testpkg-1.0.0-0 is already built in ['local'], skipping.
INFO:conda_build.config:--dirty flag and --keep-old-work not specified.Removing build/test folder after successful build/test.

# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:


# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []



####################################################################################
Source and build intermediates have been left in /home/gabm/miniconda3/conda-bld.
There are currently 85 accumulated.
To remove them, you can run the ```conda build purge``` command
```


I simulated it with local, but in our real usecase its a package in a remote repo - same effect
OK, thanks.  That's basically what our test is, so something is wrong there.  I'll look into it.
I'm sorry @msarahan but this still doesn't work in 3.0.5.. I am not using --croot, just a plain `conda build testpkg --skip-existing` fails...

I just reproduced with a clean install using the latest installer and conda-build in 2.1.17 and 3.0.5.


### 2.1.17
```
root@918c47b21c2d:~/testpkg# rm -rf ~/miniconda3/conda-bld/linux-64/* && conda index ~/miniconda3/conda-bld/linux-64/*
updating index in: /root/miniconda3/conda-bld/linux-64/*


root@918c47b21c2d:~/testpkg# conda install conda-build=2.1.17                                                         
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /root/miniconda3:
#
conda-build               2.1.17                   py36_0  


root@918c47b21c2d:~/testpkg# conda build . --skip-existing
updating index in: /root/miniconda3/conda-bld/linux-64
removing: testpkg-1.0.0-hbf21a9e_0.tar.bz2
removing: testpkg-1.0.0-0.tar.bz2
updating index in: /root/miniconda3/conda-bld/noarch
Fetching package metadata ...........
BUILD START: testpkg-1.0.0-0
Package: testpkg-1.0.0-0
source tree in: /root/miniconda3/conda-bld/testpkg_1500024261080/work
INFO:conda_build.build:Packaging testpkg-1.0.0-0
number of files: 0
Fixing permissions
Fixing permissions
/root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-0.tar.bz2
updating index in: /root/miniconda3/conda-bld
updating index in: /root/miniconda3/conda-bld/noarch
updating index in: /root/miniconda3/conda-bld/linux-64
updating: testpkg-1.0.0-0.tar.bz2
updating index in: /root/miniconda3/conda-bld
Nothing to test for: /root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-0.tar.bz2
INFO:conda_build.config:--dirty flag and --keep-old-work not specified.Removing build/test folder after successful build/test.

# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-0.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []


root@918c47b21c2d:~/testpkg# conda build . --skip-existing
updating index in: /root/miniconda3/conda-bld/linux-64
updating index in: /root/miniconda3/conda-bld/noarch
Fetching package metadata ...........
testpkg-1.0.0-0 is already built in ['local'], skipping.
INFO:conda_build.config:--dirty flag and --keep-old-work not specified.Removing build/test folder after successful build/test.

# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:


# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []
```

### 3.0.5

```
root@918c47b21c2d:~/testpkg# rm -rf ~/miniconda3/conda-bld/linux-64/* && conda index ~/miniconda3/conda-bld/linux-64/*
updating index in: /root/miniconda3/conda-bld/linux-64/*


root@918c47b21c2d:~/testpkg# conda install conda-build=3.0.5                                                          
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /root/miniconda3:

The following packages will be UPDATED:

    conda-build: 2.1.17-py36_0 --> 3.0.5-py36_0

Proceed ([y]/n)? 

conda-build-3. 100% |####################################################################################################################################################################| Time: 0:00:00   1.33 MB/s


root@918c47b21c2d:~/testpkg# conda build . --skip-existing
removing: testpkg-1.0.0-0.tar.bz2
source tree in: /root/miniconda3/conda-bld/testpkg_1500024308722/work
Attempting to finalize metadata for testpkg
INFO:conda_build.metadata:Attempting to finalize metadata for testpkg
BUILD START: ['testpkg-1.0.0-hbf21a9e_0.tar.bz2']
Packaging testpkg
INFO:conda_build.build:Packaging testpkg
Packaging testpkg-1.0.0-hbf21a9e_0
INFO:conda_build.build:Packaging testpkg-1.0.0-hbf21a9e_0
number of files: 0
Fixing permissions
Fixing permissions
updating: testpkg-1.0.0-hbf21a9e_0.tar.bz2
Nothing to test for: /root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-hbf21a9e_0.tar.bz2
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-hbf21a9e_0.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []


root@918c47b21c2d:~/testpkg# conda build . --skip-existing
source tree in: /root/miniconda3/conda-bld/testpkg_1500024313371/work
Attempting to finalize metadata for testpkg
INFO:conda_build.metadata:Attempting to finalize metadata for testpkg
BUILD START: ['testpkg-1.0.0-hbf21a9e_0.tar.bz2']
Packaging testpkg
INFO:conda_build.build:Packaging testpkg
Packaging testpkg-1.0.0-hbf21a9e_0
INFO:conda_build.build:Packaging testpkg-1.0.0-hbf21a9e_0
number of files: 0
Fixing permissions
Fixing permissions
updating: testpkg-1.0.0-hbf21a9e_0.tar.bz2
Nothing to test for: /root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-hbf21a9e_0.tar.bz2
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /root/miniconda3/conda-bld/linux-64/testpkg-1.0.0-hbf21a9e_0.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []
```

Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -1816,7 +1816,7 @@ def is_package_built(metadata, env):
                                       locking=metadata.config.locking,
                                       timeout=metadata.config.timeout,
                                       clear_cache=True)
-    urls = [url_path(metadata.config.output_folder)] + get_rc_urls()
+    urls = [url_path(metadata.config.output_folder), 'local'] + get_rc_urls()
     if metadata.config.channel_urls:
         urls.extend(metadata.config.channel_urls)
 


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -488,12 +488,24 @@ def test_conda_py_no_period(testing_workdir, testing_metadata, monkeypatch):
 
 
 @pytest.mark.serial
-def test_build_skip_existing(testing_workdir, testing_config):
+def test_build_skip_existing(testing_workdir, capfd):
     # build the recipe first
     empty_sections = os.path.join(metadata_dir, "empty_sections")
     args = ['--no-anaconda-upload', empty_sections]
     main_build.execute(args)
     args.insert(0, '--skip-existing')
     main_build.execute(args)
-    # output, error = capfd.readouterr()
-    # assert "are already built" in output
+    output, error = capfd.readouterr()
+    assert "are already built" in output
+
+
+@pytest.mark.serial
+def test_build_skip_existing_croot(testing_workdir, capfd):
+    # build the recipe first
+    empty_sections = os.path.join(metadata_dir, "empty_sections")
+    args = ['--no-anaconda-upload', '--croot', testing_workdir, empty_sections]
+    main_build.execute(args)
+    args.insert(0, '--skip-existing')
+    main_build.execute(args)
+    output, error = capfd.readouterr()
+    assert "are already built" in output