Repository: conda/conda-build

Issue:
conda convert throwing error with specific conda-build versions
### Actual Behavior
Getting the following error while using the conda convert command with osx package.
But able to convert the package with conda-build v3.4.1.

```
conda convert --platform all gsas2pkg-1.0.4-0.tar.bz2 -o outputdir/

Source platform 'osx-64' and target platform 'osx-64' are identical. Skipping conversion.
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-32
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-64
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-ppc64le
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-armv6l
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-armv7l
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-aarch64
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to win-32
Traceback (most recent call last):
  File "/Users/smahalingam/anaconda3/bin/conda-convert", line 11, in <module>
    sys.exit(main())
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_convert.py", line 130, in main
    return execute(sys.argv[1:])
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_convert.py", line 126, in execute
    api.convert(f, **args.__dict__)
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/api.py", line 304, in convert
    dry_run=dry_run, dependencies=dependencies)
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/convert.py", line 783, in conda_convert
    dependencies, verbose)
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/convert.py", line 675, in convert_from_unix_to_windows
    update_paths_file(temp_dir, target_platform='win')
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/convert.py", line 380, in update_paths_file
    path['sha256'] = update_executable_sha(temp_dir, path['_path'])
  File "/Users/smahalingam/anaconda3/lib/python3.6/site-packages/conda_build/convert.py", line 318, in update_executable_sha
    with open(os.path.join(package_directory, executable_path), 'rb') as script_file:
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/0v/tzq_sdr533gdm7vz6pcwk_zr0000gp/T/tmptengal2x/Scripts/.gsas2pkg-post-link-script.py'
```

### Expected Behavior
conda convert should convert OSX package to Linux and Windows package.
### Steps to Reproduce
Build a package on OSX using this [recipe](https://subversion.xray.aps.anl.gov/trac/pyGSAS/browser/install/g2pkg#src) and run conda convert command.


##### `conda list conda`

```
anaconda custom py36ha4fed55_0
conda 4.5.4 py36_0
conda-build 3.10.7 py36_0
conda-env 2.6.0 h36134e3_0
conda-verify 2.0.0 py36he837df3_0
```




Discussion:
FWIW, I tried downgrading conda-build to 3.4.1 and re-built my package and tried to use convert, but it still failed with the above error. It would be good to know exactly what was in the environment that worked. 
@briantoby 
**conda list conda**
```
anaconda                  5.2.0                    py36_3  
anaconda-client           1.6.14                   py36_0  
anaconda-navigator        1.8.7                    py36_0  
anaconda-project          0.8.2            py36h9ee5d53_0  
conda                     4.5.4                    py36_0  
conda-build               3.4.1                    py36_0  
conda-env                 2.6.0                h36134e3_0  
conda-verify              2.0.0            py36he837df3_0  
```
```
conda convert --platform all gsas2pkg-1.0.4-0.tar.bz2 -o outputdir/
Source platform 'osx-64' and target platform 'osx-64' are identical. Skipping conversion.
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-32
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-64
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-ppc64le
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-armv6l
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-armv7l
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to linux-aarch64
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to win-32
Converting gsas2pkg-1.0.4-0.tar.bz2 from osx-64 to win-64
0208-smahalingam:osx-64 smahalingam$ ls
gsas2pkg-1.0.4-0.tar.bz2	outputdir			repodata.json			repodata.json.bz2
0208-smahalingam:osx-64 smahalingam$ cd outputdir/
0208-smahalingam:outputdir smahalingam$ ls
linux-32	linux-64	linux-aarch64	linux-armv6l	linux-armv7l	linux-ppc64le	win-32		win-64
```
Attaching the env as text file,
[env.txt](https://github.com/conda/conda-build/files/2090312/env.txt)
. 

Thanks. I had some interesting results that might be of use in debugging this.

Brian

FWIW, downgrading conda-build in a virtual environment did not work:
(leaving out anaconda)

conda install -n build341 anaconda-client=1.6.14 anaconda-project=0.8.2 conda=4.5.4 conda-build=3.4.1 conda-env=2.6.0 conda-verify=2.0.0
source activate3 build341
conda convert --platform all ./osx-64/gsas2pkg-1.0.5-0.tar.bz2 # fails

conda convert failed with the usual error. I then bit the bullet and added anaconda into my virtual env:

conda install -n anaconda=5.2.0 build341 anaconda-client=1.6.14 anaconda-project=0.8.2 conda=4.5.4 conda-build=3.4.1 conda-env=2.6.0 conda-verify=2.0.0

and the conda convert still failed (same error).

Did work setting up a new Python copy from miniconda3 on my /tmp disk:

conda install anaconda=5.2.0 anaconda-client=1.6.14 anaconda-project=0.8.2 conda=4.5.4 conda-build=3.4.1 conda-env=2.6.0 conda-verify=2.0.0
conda convert --platform all ./osx-64/gsas2pkg-1.0.5-0.tar.bz2 # runs

conda convert worked!

Just to confirm, with another fresh miniconda and omitting anaconda=5.2.0 also works.

With a fresh miniconda and everything updated to the latest, conda convert fails:

bash  ~/Downloads/Miniconda3-latest-MacOSX-x86_64.sh
source /tmp/miniconda3/bin/activate
conda update --all
conda install anaconda-client anaconda-project conda conda-env conda-verify
conda install conda-build
conda convert --platform all ./osx-64/gsas2pkg-1.0.5-0.tar.bz2 # fails

But then only downgrade needed is conda-build to fix the problem.

conda install conda-build=3.4.1
conda convert --platform all ./osx-64/gsas2pkg-1.0.5-0.tar.bz2 # runs

However if I leave the new conda-build in place and create a new environment, it breaks:

conda create -n build341 conda-build=3.4.1
conda activate build341
conda list conda-build
# packages in environment at /tmp/miniconda3/envs/build341:
#
# Name                    Version                   Build  Channel
conda-build               3.4.1                    py36_0
conda convert --platform all ./osx-64/gsas2pkg-1.0.5-0.tar.bz2 # fails

This does fix things:
(base) bht14:~ toby$ conda remove conda-build

While conda convert does work, it creates a package that I can't actually use. The problem seems to be that the platform-specific requirements are not handled properly.  Error below is with OSX build package converted to Win-64, but I see exactly the same problem with the OSX build converted to Linux-64 on that platform. Note that if I do a conda build (& upload) on an actual windows machine, the exact same conda install command does work. 

The error message is a bit strange in that is says it can't find gsas2pkg, but it must have read that in order to find the requirement for python.app

```
(base) C:\>conda install -c briantoby gsas2pkg
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - gsas2pkg
  - python.app

Current channels:

  - https://conda.anaconda.org/briantoby/win-64
  - https://conda.anaconda.org/briantoby/noarch
  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/win-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/win-64
  - https://repo.anaconda.com/pkgs/pro/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.



(base) C:\>conda info

     active environment : base
    active env location : C:\Users\toby\miniconda3-64
            shell level : 1
       user config file : C:\Users\toby\.condarc
 populated config files :
          conda version : 4.5.4
    conda-build version : not installed
         python version : 3.6.5.final.0
       base environment : C:\Users\toby\miniconda3-64  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\toby\miniconda3-64\pkgs
                          C:\Users\toby\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\toby\miniconda3-64\envs
                          C:\Users\toby\AppData\Local\conda\conda\envs
                          C:\Users\toby\.conda\envs
               platform : win-64
             user-agent : conda/4.5.4 requests/2.18.4 CPython/3.6.5 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False
```
the meta.yaml file is this: 

```
package: 
  name: gsas2pkg
  version: "1.0.6"  # remember to change in g2conda/construct.yaml to match this

source:
  path: ./src

about:
  home: https://subversion.xray.aps.anl.gov/trac/pyGSAS

#build:
#  noarch: python

requirements:
  run:
    - python
    - conda
    - numpy
    - scipy
    - matplotlib
    - pillow
    - pyopengl
    - wxpython
    - svn
    - hdf5
    - h5py
    - imageio
    - pywin32                              [win]
    - python.app                           [osx]
```
 cc ing @msarahan 
Thanks for the additional debugging.  I'll try to get to this by the end of this week.
no hurry on my account. I am done with builds for a while (I hope).

Brian

On Jun 12, 2018, at 11:43 AM, Mike Sarahan <notifications@github.com<mailto:notifications@github.com>> wrote:


Thanks for the additional debugging. I'll try to get to this by the end of this week.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/conda/conda-build/issues/2953#issuecomment-396656824>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AI_yaFPlb3U3AFADt_f8uBRwqfn7XaJLks5t7-_EgaJpZM4UczFF>.


Same issue here with conda-build 3.10.9 when converting from linux-64 to win. 

Resolving patch:
diff --git a/conda_build/convert.py b/conda_build/convert.py
--- a/conda_build/convert.py
+++ b/conda_build/convert.py
@@ -286,7 +286,7 @@ def update_lib_contents(lib_directory, temp_dir, target_platform, file_path):
             shutil.move(lib_file, new_lib_file)
 
 
-def update_executable_path(file_path, target_platform):
+def update_executable_path(temp_dir, file_path, target_platform):
     """Update the name of the executable files found in the paths.json file.
 
     When converting from unix to windows, executables are renamed with a '-script.py'
@@ -298,11 +298,14 @@ def update_executable_path(file_path, target_platform):
     target_platform (str) -- the platform to target: 'unix' or 'win'
     """
     if target_platform == 'win':
-        renamed_path = os.path.splitext(re.sub('\Abin', 'Scripts', file_path))[0]
-        renamed_executable_path = '{}-script.py' .format(renamed_path)
+        if os.path.basename(file_path).startswith('.') or is_binary_file(temp_dir, file_path):
+            renamed_executable_path = re.sub('\Abin', 'Scripts', file_path)
+        else:
+            renamed_path = os.path.splitext(re.sub('\Abin', 'Scripts', file_path))[0]
+            renamed_executable_path = '{}-script.py' .format(renamed_path)
 
     elif target_platform == 'unix':
-        renamed_path = os.path.splitext(re.sub('\AScripts', 'bin', file_path))[0]
+        renamed_path = re.sub('\AScripts', 'bin', file_path)
         renamed_executable_path = renamed_path.replace('-script.py', '')
 
     return renamed_executable_path
@@ -376,7 +379,7 @@ def update_paths_file(temp_dir, target_platform):
                     path['_path'] = update_lib_path(path['_path'], 'win')
 
                 elif path['_path'].startswith('bin'):
-                    path['_path'] = update_executable_path(path['_path'], 'win')
+                    path['_path'] = update_executable_path(temp_dir, path['_path'], 'win')
                     path['sha256'] = update_executable_sha(temp_dir, path['_path'])
                     path['size_in_bytes'] = update_executable_size(temp_dir, path['_path'])
 
@@ -394,7 +397,7 @@ def update_paths_file(temp_dir, target_platform):
                     path['_path'] = update_lib_path(path['_path'], 'unix', temp_dir)
 
                 elif path['_path'].startswith('Scripts'):
-                    path['_path'] = update_executable_path(path['_path'], 'unix')
+                    path['_path'] = update_executable_path(temp_dir, path['_path'], 'unix')
                     path['sha256'] = update_executable_sha(temp_dir, path['_path'])
                     path['size_in_bytes'] = update_executable_size(temp_dir, path['_path'])
 


Test patch:
diff --git a/tests/test_api_convert.py b/tests/test_api_convert.py
--- a/tests/test_api_convert.py
+++ b/tests/test_api_convert.py
@@ -305,3 +305,32 @@ def test_skip_conversion(testing_workdir, base_platform, package, capfd):
 
     assert skip_message in output
     assert not os.path.exists(package)
+
+
+@pytest.mark.parametrize('base_platform', ['linux', 'osx'])
+@pytest.mark.parametrize('package', [('sparkmagic-0.12.1', '')])
+def test_renaming_executables(testing_workdir, base_platform, package):
+    """Test that the files in /bin are properly renamed.
+
+    When converting the bin/ directory to Scripts/, only scripts
+    need to be changed. Sometimes the /bin directory contains other
+    files that are not Python scripts such as post-link.sh scripts.
+    This test converts a packaege that contains a post-link.sh script
+    in the bin/ directory and checks to see that its filename remains
+    the same.
+    """
+    package_name, example_file = package
+    subdir = '{}-64'.format(base_platform)
+    f = 'http://repo.continuum.io/pkgs/free/{}/{}-py27_0.tar.bz2'.format(subdir,
+                                                                         package_name)
+    fn = "{}-py27_0.tar.bz2".format(package_name)
+    download(f, fn)
+    expected_paths_json = package_has_file(fn, 'info/paths.json')
+    api.convert(fn, platforms='all', quiet=False, verbose=False)
+    for platform in ['osx-64', 'win-64', 'win-32', 'linux-64', 'linux-32']:
+        if subdir != platform:
+            package = os.path.join(platform, fn)
+
+            if expected_paths_json:
+                assert package_has_file(package, 'info/paths.json')
+                assert_package_paths_matches_files(package)