Repository: pypa/setuptools

Issue:
Packaged script execute permissions lost from v46.1.0 onwards
## Issue

We use pipenv and, as a result, are forced to use the latest version of setuptools as it internally pins to the latest version.

We observed that executable scripts that are part of python packages have lost their execute flag from setuptools v46.1.0 onwards.  The example to demonstrate this bug uses pyspark which includes a number of executable scripts in its package.

The issue was introduced by commit https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f where the `copy_file()` function is now called with `preserve_mode=False`. The change log states the reason for the change as:

> Prevent keeping files mode for package_data build. It may break a build if user's package data has read only flag.

Unfortunately, this has the side effect of stripping all execute permissions from files, meaning users can't use the scripts "out of the box" - they have to set the execute permissions manually.

## Demonstration Script

```
#!/bin/bash
set -eu

wget -nc https://files.pythonhosted.org/packages/9a/5a/271c416c1c2185b6cb0151b29a91fff6fcaed80173c8584ff6d20e46b465/pyspark-2.4.5.tar.gz

for version in "46.0.0" "46.1.0"; do
    rm -rf .venv pyspark-2.4.5
    tar xzf pyspark-2.4.5.tar.gz
    virtualenv -q -p /usr/bin/python3.7 .venv
    . .venv/bin/activate
    python3 -m pip install --upgrade setuptools="==${version}" wheel
    pushd pyspark-2.4.5
    python3 setup.py -q bdist_wheel
    pushd dist
    unzip -q pyspark-2.4.5-py2.py3-none-any.whl
    echo -e "\n\n${version}: Here are the permissions for spark-submit:\n"
    ls -l ./pyspark/bin/spark-submit
    echo -e "\n\n"
    popd
    popd
done
```

## Expected result

```
-rwxr-xr-x 1 dave dave 1040 Feb  2 19:35 ./pyspark/bin/spark-submit
```

## Actual Result

```
-rw-rw-r-- 1 dave dave 1040 Feb  2 19:35 ./pyspark/bin/spark-submit
```
Packaged script execute permissions lost from v46.1.0 onwards
## Issue

We use pipenv and, as a result, are forced to use the latest version of setuptools as it internally pins to the latest version.

We observed that executable scripts that are part of python packages have lost their execute flag from setuptools v46.1.0 onwards.  The example to demonstrate this bug uses pyspark which includes a number of executable scripts in its package.

The issue was introduced by commit https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f where the `copy_file()` function is now called with `preserve_mode=False`. The change log states the reason for the change as:

> Prevent keeping files mode for package_data build. It may break a build if user's package data has read only flag.

Unfortunately, this has the side effect of stripping all execute permissions from files, meaning users can't use the scripts "out of the box" - they have to set the execute permissions manually.

## Demonstration Script

```
#!/bin/bash
set -eu

wget -nc https://files.pythonhosted.org/packages/9a/5a/271c416c1c2185b6cb0151b29a91fff6fcaed80173c8584ff6d20e46b465/pyspark-2.4.5.tar.gz

for version in "46.0.0" "46.1.0"; do
    rm -rf .venv pyspark-2.4.5
    tar xzf pyspark-2.4.5.tar.gz
    virtualenv -q -p /usr/bin/python3.7 .venv
    . .venv/bin/activate
    python3 -m pip install --upgrade setuptools="==${version}" wheel
    pushd pyspark-2.4.5
    python3 setup.py -q bdist_wheel
    pushd dist
    unzip -q pyspark-2.4.5-py2.py3-none-any.whl
    echo -e "\n\n${version}: Here are the permissions for spark-submit:\n"
    ls -l ./pyspark/bin/spark-submit
    echo -e "\n\n"
    popd
    popd
done
```

## Expected result

```
-rwxr-xr-x 1 dave dave 1040 Feb  2 19:35 ./pyspark/bin/spark-submit
```

## Actual Result

```
-rw-rw-r-- 1 dave dave 1040 Feb  2 19:35 ./pyspark/bin/spark-submit
```


Discussion:
I'm facing this issue too on macOS 10.15.3 and Pipenv 2018.11.26. This code was working fine in 46.0.0  but started failing in 46.1.0:

```python
from pyspark.sql import SparkSession # pyspark==2.4.5

spark = SparkSession.builder.appName("Test").getOrCreate()
```
In the Ansible community we've started seeing this issue testing Ansible Collections, since `ansible-test` depends on internal executable scripts which are not entry points.

See https://github.com/pypa/setuptools/pull/1607#issuecomment-602922992
The bug seems to have been introduced in the last 3 days, when we started seeing permission problems, most notably the lack of execution permissions in scripts.

Someone briefly mentioned 512565e, but that seems to only be whitespace changes, and the original change in 7843688 goes back to 2018, which was merged a long time ago.

We can workaround this issue by changing the permissions of installed scripts, but I'm not sure that this project is the root cause of this recent regression.

@jaraco @KOLANICH 
@leekillough https://github.com/pypa/setuptools/pull/1607 (which contained https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f)  was merged into `master` only 2 days ago with the change. The change can be seen in the release history for version v46.1.0, released on March 21:

https://setuptools.readthedocs.io/en/latest/history.html#v46-1-0

> #1424: Prevent keeping files mode for package_data build. It may break a build if user’s package data has read only flag.
> 
> 
> @leekillough #1607 (which contained [7843688](https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f)) was merged into `master` only 2 days ago with the change. The change can be seen in the release history for version v46.1.0, released on March 21:
> 
> https://setuptools.readthedocs.io/en/latest/history.html#v46-1-0
> 
> > #1424: Prevent keeping files mode for package_data build. It may break a build if user’s package data has read only flag.

Sorry, I missed that PR. I thought I went through them all, but for some reason I could never find commit [7843688](https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f).
Hi @jaraco and @jorikdima, Is it going to be fixed?

Apache Spark community faced this issue and CI started to break. We're setting the upper bound of setuptools to work around CI break for now.


I also filed this issue in Spark side to track [SPARK-31231](https://issues.apache.org/jira/browse/SPARK-31231).
I'm facing this issue too on macOS 10.15.3 and Pipenv 2018.11.26. This code was working fine in 46.0.0  but started failing in 46.1.0:

```python
from pyspark.sql import SparkSession # pyspark==2.4.5

spark = SparkSession.builder.appName("Test").getOrCreate()
```
In the Ansible community we've started seeing this issue testing Ansible Collections, since `ansible-test` depends on internal executable scripts which are not entry points.

See https://github.com/pypa/setuptools/pull/1607#issuecomment-602922992
The bug seems to have been introduced in the last 3 days, when we started seeing permission problems, most notably the lack of execution permissions in scripts.

Someone briefly mentioned 512565e, but that seems to only be whitespace changes, and the original change in 7843688 goes back to 2018, which was merged a long time ago.

We can workaround this issue by changing the permissions of installed scripts, but I'm not sure that this project is the root cause of this recent regression.

@jaraco @KOLANICH 
@leekillough https://github.com/pypa/setuptools/pull/1607 (which contained https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f)  was merged into `master` only 2 days ago with the change. The change can be seen in the release history for version v46.1.0, released on March 21:

https://setuptools.readthedocs.io/en/latest/history.html#v46-1-0

> #1424: Prevent keeping files mode for package_data build. It may break a build if user’s package data has read only flag.
> 
> 
> @leekillough #1607 (which contained [7843688](https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f)) was merged into `master` only 2 days ago with the change. The change can be seen in the release history for version v46.1.0, released on March 21:
> 
> https://setuptools.readthedocs.io/en/latest/history.html#v46-1-0
> 
> > #1424: Prevent keeping files mode for package_data build. It may break a build if user’s package data has read only flag.

Sorry, I missed that PR. I thought I went through them all, but for some reason I could never find commit [7843688](https://github.com/pypa/setuptools/commit/7843688bc33dd4e13e10130bc49da4c290fe7d7f).
Hi @jaraco and @jorikdima, Is it going to be fixed?

Apache Spark community faced this issue and CI started to break. We're setting the upper bound of setuptools to work around CI break for now.


I also filed this issue in Spark side to track [SPARK-31231](https://issues.apache.org/jira/browse/SPARK-31231).

Resolving patch:
diff --git a/setuptools/command/build_py.py b/setuptools/command/build_py.py
--- a/setuptools/command/build_py.py
+++ b/setuptools/command/build_py.py
@@ -7,6 +7,7 @@
 import io
 import distutils.errors
 import itertools
+import stat
 
 from setuptools.extern import six
 from setuptools.extern.six.moves import map, filter, filterfalse
@@ -20,6 +21,10 @@ def run_2to3(self, files, doctests=True):
             "do nothing"
 
 
+def make_writable(target):
+    os.chmod(target, os.stat(target).st_mode | stat.S_IWRITE)
+
+
 class build_py(orig.build_py, Mixin2to3):
     """Enhanced 'build_py' command that includes data files with packages
 
@@ -120,8 +125,8 @@ def build_package_data(self):
                 target = os.path.join(build_dir, filename)
                 self.mkpath(os.path.dirname(target))
                 srcfile = os.path.join(src_dir, filename)
-                outf, copied = self.copy_file(
-                    srcfile, target, preserve_mode=False)
+                outf, copied = self.copy_file(srcfile, target)
+                make_writable(target)
                 srcfile = os.path.abspath(srcfile)
                 if (copied and
                         srcfile in self.distribution.convert_2to3_doctests):


Test patch:
diff --git a/setuptools/tests/test_build_py.py b/setuptools/tests/test_build_py.py
--- a/setuptools/tests/test_build_py.py
+++ b/setuptools/tests/test_build_py.py
@@ -2,6 +2,8 @@
 import stat
 import shutil
 
+import pytest
+
 from setuptools.dist import Distribution
 
 
@@ -26,9 +28,10 @@ def test_directories_in_package_data_glob(tmpdir_cwd):
 
 def test_read_only(tmpdir_cwd):
     """
-    Ensure mode is not preserved in copy for package modules
-    and package data, as that causes problems
-    with deleting read-only files on Windows.
+    Ensure read-only flag is not preserved in copy
+    for package modules and package data, as that
+    causes problems with deleting read-only files on
+    Windows.
 
     #1451
     """
@@ -47,3 +50,35 @@ def test_read_only(tmpdir_cwd):
     dist.parse_command_line()
     dist.run_commands()
     shutil.rmtree('build')
+
+
+@pytest.mark.xfail(
+    'platform.system() == "Windows"',
+    reason="On Windows, files do not have executable bits",
+    raises=AssertionError,
+    strict=True,
+)
+def test_executable_data(tmpdir_cwd):
+    """
+    Ensure executable bit is preserved in copy for
+    package data, as users rely on it for scripts.
+
+    #2041
+    """
+    dist = Distribution(dict(
+        script_name='setup.py',
+        script_args=['build_py'],
+        packages=['pkg'],
+        package_data={'pkg': ['run-me']},
+        name='pkg',
+    ))
+    os.makedirs('pkg')
+    open('pkg/__init__.py', 'w').close()
+    open('pkg/run-me', 'w').close()
+    os.chmod('pkg/run-me', 0o700)
+
+    dist.parse_command_line()
+    dist.run_commands()
+
+    assert os.stat('build/lib/pkg/run-me').st_mode & stat.S_IEXEC, \
+        "Script is not executable"