Repository: conda/conda-build

Issue:
fix_shebang fails when file does not have write permissions
Per @jakirkham advice (https://github.com/conda-forge/pdal-feedstock/issues/1#issuecomment-382759759), opening this issue as the `pdal-feedstock` which had been passing both Linux [1] and macOS [2] in `conda-forge/staged-recipes` prior to the upgrade from `conda-build` 2 to 3, now fail both Linux and macOS as a feedstock, as shown below.

https://circleci.com/gh/conda-forge/pdal-feedstock/6

```
Traceback (most recent call last):
  File "/opt/conda/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main
    execute(sys.argv[1:])
  File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute
    verify=args.verify)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 199, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 2105, in build_tree
    notest=notest,
  File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1515, in build
    built_package = bundlers[output_d.get('type', 'conda')](output_d, m, env, stats)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 917, in bundle_conda
    files = post_process_files(metadata, initial_files)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 780, in post_process_files
    post_build(m, new_files, build_python=python)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/post.py", line 642, in post_build
    osx_is_app=osx_is_app)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/post.py", line 54, in fix_shebang
    with io.open(path, encoding=locale.getpreferredencoding(), mode='r+') as fi:
PermissionError: [Errno 13] Permission denied: '/home/conda/feedstock_root/build_artifacts/pdal_1524146590889/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/pdal-config'
```

https://travis-ci.org/conda-forge/pdal-feedstock/builds/368307455

```
Traceback (most recent call last):
  File "/Users/travis/miniconda3/bin/upload_or_check_non_existence", line 129, in <module>
    main()
  File "/Users/travis/miniconda3/bin/upload_or_check_non_existence", line 119, in main
    upload(cli, fname, owner, channel)
  File "/Users/travis/miniconda3/bin/upload_or_check_non_existence", line 66, in upload
    env=os.environ)
  File "/Users/travis/miniconda3/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['anaconda', '--quiet', '-t', '/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/tmphe4j3uvo/binstar.token', 'upload', '/Users/travis/miniconda3/conda-bld/osx-64/pdal-1.7.1-hf626a3b_0.tar.bz2', '--user=conda-forge', '--channel=main']' returned non-zero exit status 1.
```

[1] https://circleci.com/gh/conda-forge/staged-recipes/25828<br>
[2] https://travis-ci.org/conda-forge/staged-recipes/builds/365092079


Discussion:
FTR the second traceback is probably caused by the package not existing. IOW since the `conda build` step failed there is no package to upload. So that part we can ignore for this discussion.
Yes, I grabbed the incorrect lines from the Travis logs. Should be [here](https://travis-ci.org/conda-forge/pdal-feedstock/builds/368307455#L1130-L1151) and nearly identical to what is shown on CircleCI.
@msarahan, do you have any ideas for us here?
not really off the top of my head.  Permission stuff hasn't really changed that I know of.  Try to reproduce in a docker container locally, and take a look at that file in particular in the host prefix.  Does it exist?  What are its permissions?
@msarahan It does exist, but without write permissions.

```
-r-xr-xr-x 1 conda conda 3362 Apr 23 22:15 /opt/conda/conda-bld/pdal_1524521105777/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/bin/pdal-config
```

What is `fix_shebang` doing that it needs write access?
Hmm...interesting. Why doesn't it have write permissions?
Here is where it's set (https://github.com/PDAL/PDAL/blob/master/apps/CMakeLists.txt#L78-L83).
Ouch! Thanks for that info.

The point of `fix_shebang` and other similar things in `conda-build` is to detect hard coded paths in scripts, etc. and correct them during packaging so they can be changed later. This is the magic ingredient that allows `conda` packages to be installed in other locations or on different people's machines without issues. Namely `conda` is overwriting these paths based on your prefix on the fly during installation. As the PDAL build is blocking write permissions on this script, PDAL is making it harder for `conda-build` to do this correctly with this file. At this point, you have a few options to solve this.

1. Raise an issue with PDAL devs and see if they are willing to relax the permissions to allow writing.
2. Patch the build scripts to allow writing.
3. Disable prefix detection on this file (leaving it broken, but including it in the package).
4. Disable installation of this script entirely (it's not broken, but it isn't there if things need it).

My recommendation regardless of the other options you might choose is to do 1 (raise with PDAL devs). Resolving at that level may take some time. However if they are open to correcting this, they will start looking into ways to fix it that you can use down the road (or perhaps more immediately as part of 2, patching the build scripts). As patching does look pretty easy in this case, could even try that and submit the patch upstream as well to get it integrated (avoiding long term maintenance burden at the packaging level). Would only go with 3 or 4 as last resorts, but they are on the table if everything else is too challenging. If you go with either 3 or 4, would recommend raising an issue on the feedstock to track this. Of 3 or 4, would argue 4 is preferable as this script seems considerably less useful if it can't correctly identify its install location.
Thanks @jakirkham. I'll work 1 and 2 concurrently. I work closely with the devs and it shouldn't be a problem. Closing this as a non-issue with `conda-build`.
Sounds good. Let us know how that goes.
Independently there is a question of how `conda-build` should handle this. Some thoughts that come to mind are...

1. Do nothing (current behavior is in some sense correct)
2. Make the file writable anyways and proceed
3. Basically 2, but mark the file read only afterwards (conda will have to resolve this somehow)

Of course there could be other ideas or tweaks on these not listed. Suggestions welcome.
At the very least, it seems like we could show a better message.  Either keep it as an error, or make it warning, but either way, explain what is happening and what the effect might be.
Should we reopen to track that?
Sure.  I'll edit the title.
As you can see from the commits and PRs linked above, we are seeing this issue quite often recently in Bioconda for macOS builds, which we have to workaround on each affected (Perl) package.
Please consider submitting a PR.  As this is so strongly affecting bioconda but not many others, I feel it is reasonable to ask that community to address the problem.

Resolving patch:
diff --git a/conda_build/post.py b/conda_build/post.py
--- a/conda_build/post.py
+++ b/conda_build/post.py
@@ -52,7 +52,8 @@ def fix_shebang(f, prefix, build_python, osx_is_app=False):
 
     bytes_ = False
 
-    with io.open(path, encoding=locale.getpreferredencoding(), mode='r+') as fi:
+    os.chmod(path, 0o775)
+    with io.open(path, mode='r+', encoding=locale.getpreferredencoding()) as fi:
         try:
             data = fi.read(100)
             fi.seek(0)
@@ -98,7 +99,6 @@ def fix_shebang(f, prefix, build_python, osx_is_app=False):
             fo.write(new_data)
         except TypeError:
             fo.write(new_data.decode())
-    os.chmod(path, 0o775)
 
 
 def write_pth(egg_path, config):
diff --git a/conda_build/utils.py b/conda_build/utils.py
--- a/conda_build/utils.py
+++ b/conda_build/utils.py
@@ -1452,8 +1452,8 @@ def mmap_mmap(fileno, length, tagname=None, flags=0, prot=mmap_PROT_READ | mmap_
     Hides the differences between mmap.mmap on Windows and Unix.
     Windows has `tagname`.
     Unix does not, but makes up for it with `flags` and `prot`.
-    On both, the defaule value for `access` is determined from how the file
-    was opened so must not be passed in at all to get this default behaviour
+    On both, the default value for `access` is determined from how the file
+    was opened so must not be passed in at all to get this default behaviour.
     '''
     if on_win:
         if access:


Test patch:
diff --git a/tests/test_post.py b/tests/test_post.py
--- a/tests/test_post.py
+++ b/tests/test_post.py
@@ -50,6 +50,16 @@ def test_postbuild_files_raise(testing_metadata, testing_workdir):
         assert f in str(exc)
 
 
+@pytest.mark.skipif(on_win, reason="fix_shebang is not executed on win32")
+def test_fix_shebang(testing_config):
+    fname = 'test1'
+    with open(fname, 'w') as f:
+        f.write("\n")
+    os.chmod(fname, 0o000)
+    post.fix_shebang(fname, '.', '/test/python')
+    assert os.stat(fname).st_mode == 33277  # file with permissions 0o775
+
+
 def test_postlink_script_in_output_explicit(testing_config):
     recipe = os.path.join(metadata_dir, '_post_link_in_output')
     pkg = api.build(recipe, config=testing_config, notest=True)[0]