Repository: tox-dev/tox

Issue:
Python bytes path is not decoded to string while using TOX_LIMITED_SHEBANG and Python 3.4
- [ ] Minimal reproducible example or detailed description, assign "bug"

```
(.venv) [vlotorev@cygnus-pc infra-ci-scripts]$ export TOX_LIMITED_SHEBANG=1
(.venv) [vlotorev@cygnus-pc infra-ci-scripts]$ tox
ERROR: invocation failed (errno 2), args: [b'/homefs/vlotorev/projects/infra-ci-scripts/.tox/py34/bin/python3.4', '/homefs/vlotorev/projects/infra-ci-scripts/.tox/py34/bin/pip', 'freeze'], cwd: /homefs/vlotorev/projects/infra-ci-scripts
Traceback (most recent call last):
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 39, in cmdline
    main(args)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 45, in main
    retcode = Session(config).runcommand()
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 422, in runcommand
    return self.subcommand_test()
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 622, in subcommand_test
    self.runenvreport(venv)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 633, in runenvreport
    packages = self.hook.tox_runenvreport(venv=venv, action=action)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/pluggy/hooks.py", line 258, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/pluggy/manager.py", line 67, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/pluggy/manager.py", line 61, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/venv.py", line 499, in tox_runenvreport
    output = venv._pcall(args, cwd=venv.envconfig.config.toxinidir, action=action)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/venv.py", line 427, in _pcall
    return action.popen(args, cwd=cwd, env=env, redirect=redirect, ignore_ret=ignore_ret)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 153, in popen
    popen = self._popen(args, cwd, env=env, stdout=stdout, stderr=subprocess.STDOUT)
  File "/homefs/vlotorev/projects/infra-ci-scripts/.venv/lib64/python3.4/site-packages/tox/session.py", line 248, in _popen
    env=env,
  File "/usr/lib64/python3.4/subprocess.py", line 856, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.4/subprocess.py", line 1460, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: "b'/homefs/vlotorev/projects/infra-ci-scripts/.tox/py34/bin/python3.4'"

```
Note the path exists:
```
(.venv) [vlotorev@cygnus-pc infra-ci-scripts]$ file /homefs/vlotorev/projects/infra-ci-scripts/.tox/py34/bin/python3.4
/homefs/vlotorev/projects/infra-ci-scripts/.tox/py34/bin/python3.4: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=ec3bcf7075f305062a6cab344ddc86ff72d5bbee, stripped
```

Note the path in exception message has b' prefix: this is because `return interp_args + args` (see https://github.com/tox-dev/tox/blob/master/src/tox/venv.py#L463) return bytes variable, which isn't handled properly later.

- [ ] OS and `pip list` output
```
(.venv) [vlotorev@cygnus-pc infra-ci-scripts]$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
(.venv) [vlotorev@cygnus-pc infra-ci-scripts]$ pip list
Package    Version
---------- -------
packaging  17.1
pip        18.0
pluggy     0.7.1
py         1.5.4
pyparsing  2.2.0
setuptools 40.0.0
six        1.11.0
tox        3.1.2
virtualenv 16.0.0
wheel      0.31.1
(.venv) [vlotorev@cygnus-pc infra-ci-scripts]$ python --version
Python 3.4.5
```

In case I create the same environment (pip list) with Python 2.7 then issue is not reproducible.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=h1) Report
> Merging [#946](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=desc) into [master](https://codecov.io/gh/tox-dev/tox/commit/37e2b53fcd9651cab0c786eca469e3e4c5ad9006?src=pr&el=desc) will **decrease** coverage by `1.34%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/tox-dev/tox/pull/946/graphs/tree.svg?src=pr&token=DYodAwDCZ5&width=650&height=150)](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #946      +/-   ##
==========================================
- Coverage   92.77%   91.42%   -1.35%     
==========================================
  Files          12       12              
  Lines        2379     2379              
  Branches      413      413              
==========================================
- Hits         2207     2175      -32     
- Misses        109      139      +30     
- Partials       63       65       +2
```


| [Impacted Files](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/tox/venv.py](https://codecov.io/gh/tox-dev/tox/pull/946/diff?src=pr&el=tree#diff-c3JjL3RveC92ZW52LnB5) | `92.03% <100%> (-0.89%)` | :arrow_down: |
| [src/tox/interpreters.py](https://codecov.io/gh/tox-dev/tox/pull/946/diff?src=pr&el=tree#diff-c3JjL3RveC9pbnRlcnByZXRlcnMucHk=) | `69.84% <0%> (-19.05%)` | :arrow_down: |
| [src/tox/\_pytestplugin.py](https://codecov.io/gh/tox-dev/tox/pull/946/diff?src=pr&el=tree#diff-c3JjL3RveC9fcHl0ZXN0cGx1Z2luLnB5) | `92.24% <0%> (-0.41%)` | :arrow_down: |
| [src/tox/session.py](https://codecov.io/gh/tox-dev/tox/pull/946/diff?src=pr&el=tree#diff-c3JjL3RveC9zZXNzaW9uLnB5) | `91.57% <0%> (-0.36%)` | :arrow_down: |
| [src/tox/config.py](https://codecov.io/gh/tox-dev/tox/pull/946/diff?src=pr&el=tree#diff-c3JjL3RveC9jb25maWcucHk=) | `95.54% <0%> (-0.25%)` | :arrow_down: |
| [src/tox/\_quickstart.py](https://codecov.io/gh/tox-dev/tox/pull/946/diff?src=pr&el=tree#diff-c3JjL3RveC9fcXVpY2tzdGFydC5weQ==) | `76.74% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=footer). Last update [37e2b53...98eaecf](https://codecov.io/gh/tox-dev/tox/pull/946?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/changelog/931.bugfix.rst b/changelog/931.bugfix.rst
new file mode 100644
index 00000000..060a2902
--- /dev/null
+++ b/changelog/931.bugfix.rst
@@ -0,0 +1,1 @@
+fix ``TOX_LIMITED_SHEBANG`` when running under python3 - by :user:`asottile`
diff --git a/src/tox/venv.py b/src/tox/venv.py
index b4e70984..f93b9445 100755
--- a/src/tox/venv.py
+++ b/src/tox/venv.py
@@ -459,10 +459,10 @@ def prepend_shebang_interpreter(args):
         with open(args[0], "rb") as f:
             if f.read(1) == b"#" and f.read(1) == b"!":
                 MAXINTERP = 2048
-                interp = f.readline(MAXINTERP).rstrip()
+                interp = f.readline(MAXINTERP).rstrip().decode("UTF-8")
                 interp_args = interp.split(None, 1)[:2]
                 return interp_args + args
-    except IOError:
+    except (UnicodeDecodeError, IOError):
         pass
     return args
 


Test patch:
diff --git a/tests/test_venv.py b/tests/test_venv.py
index 0ba6d27c..c5673c3a 100644
--- a/tests/test_venv.py
+++ b/tests/test_venv.py
@@ -903,6 +903,17 @@ def test_tox_testenv_interpret_shebang_empty_interpreter_ws(tmpdir):
     assert args == base_args
 
 
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_non_utf8(tmpdir):
+    testfile = tmpdir.join("check_non_utf8.py")
+    base_args = [str(testfile), "arg1", "arg2", "arg3"]
+
+    # empty interpreter (whitespaces)
+    testfile.write_binary(b"#!\x9a\xef\x12\xaf\n")
+    args = prepend_shebang_interpreter(base_args)
+    assert args == base_args
+
+
 @pytest.mark.skipif("sys.platform == 'win32'")
 def test_tox_testenv_interpret_shebang_interpreter_simple(tmpdir):
     testfile = tmpdir.join("check_shebang_interpreter_simple.py")
@@ -911,7 +922,7 @@ def test_tox_testenv_interpret_shebang_interpreter_simple(tmpdir):
     # interpreter (simple)
     testfile.write("#!interpreter")
     args = prepend_shebang_interpreter(base_args)
-    assert args == [b"interpreter"] + base_args
+    assert args == ["interpreter"] + base_args
 
 
 @pytest.mark.skipif("sys.platform == 'win32'")
@@ -922,7 +933,7 @@ def test_tox_testenv_interpret_shebang_interpreter_ws(tmpdir):
     # interpreter (whitespaces)
     testfile.write("#!  interpreter  \n\n")
     args = prepend_shebang_interpreter(base_args)
-    assert args == [b"interpreter"] + base_args
+    assert args == ["interpreter"] + base_args
 
 
 @pytest.mark.skipif("sys.platform == 'win32'")
@@ -933,7 +944,7 @@ def test_tox_testenv_interpret_shebang_interpreter_arg(tmpdir):
     # interpreter with argument
     testfile.write("#!interpreter argx\n")
     args = prepend_shebang_interpreter(base_args)
-    assert args == [b"interpreter", b"argx"] + base_args
+    assert args == ["interpreter", "argx"] + base_args
 
 
 @pytest.mark.skipif("sys.platform == 'win32'")
@@ -944,7 +955,7 @@ def test_tox_testenv_interpret_shebang_interpreter_args(tmpdir):
     # interpreter with argument (ensure single argument)
     testfile.write("#!interpreter argx argx-part2\n")
     args = prepend_shebang_interpreter(base_args)
-    assert args == [b"interpreter", b"argx argx-part2"] + base_args
+    assert args == ["interpreter", "argx argx-part2"] + base_args
 
 
 @pytest.mark.skipif("sys.platform == 'win32'")
@@ -955,7 +966,7 @@ def test_tox_testenv_interpret_shebang_real(tmpdir):
     # interpreter (real example)
     testfile.write("#!/usr/bin/env python\n")
     args = prepend_shebang_interpreter(base_args)
-    assert args == [b"/usr/bin/env", b"python"] + base_args
+    assert args == ["/usr/bin/env", "python"] + base_args
 
 
 @pytest.mark.skipif("sys.platform == 'win32'")
@@ -971,9 +982,9 @@ def test_tox_testenv_interpret_shebang_long_example(tmpdir):
     )
     args = prepend_shebang_interpreter(base_args)
     expected = [
-        b"this-is-an-example-of-a-very-long-interpret-directive-what-should-be-"
-        b"directly-invoked-when-tox-needs-to-invoked-the-provided-script-name-"
-        b"in-the-argument-list"
+        "this-is-an-example-of-a-very-long-interpret-directive-what-should-be-"
+        "directly-invoked-when-tox-needs-to-invoked-the-provided-script-name-"
+        "in-the-argument-list"
     ]
 
     assert args == expected + base_args