Repository: tox-dev/tox
Pull request #812 (2018-05-09 21:10:44)

Remove old config-v2 docs page
## Thanks for contributing a pull request!

If you are contributing for the first time or provide a trivial fix don't worry too
much about the checklist - we will help you get started.

## Contribution checklist:

(also see [CONTRIBUTING.rst](https://github.com/tox-dev/tox/tree/master/CONTRIBUTING.rst) for details)

- [ ] wrote descriptive pull request text
- [ ] added/updated test(s)
- [x] updated/extended the documentation
- [ ] added relevant [issue keyword](https://help.github.com/articles/closing-issues-using-keywords/)
      in message body
- [ ] added news fragment in [changelog folder](https://github.com/tox-dev/tox/tree/master/changelog)
  * fragment name: `<issue number>.<type>.rst` for example (588.bugfix.rst)
  * `<type>` is must be one of `bugfix`, `feature`, `deprecation`,`breaking`, `doc`, `misc`
  * if pr has no issue: consider creating one first or change it to the pr number after creating the pr
  * "sign" fragment with "by @<your username>"
  * please use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files - by @superuser."
  * also see [examples](https://github.com/tox-dev/tox/tree/master/changelog/examples.rst)
- [ ] added yourself to `CONTRIBUTORS` (preserving alphabetical order)


codecov[bot]: # [Codecov](https://codecov.io/gh/tox-dev/tox/pull/812?src=pr&el=h1) Report
> Merging [#812](https://codecov.io/gh/tox-dev/tox/pull/812?src=pr&el=desc) into [master](https://codecov.io/gh/tox-dev/tox/commit/ba51cfccbb80bec0c8e6896429a0993d2e62e524?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.


```diff
@@          Coverage Diff          @@
##           master   #812   +/-   ##
=====================================
  Coverage      95%    95%           
=====================================
  Files          11     11           
  Lines        2307   2307           
=====================================
  Hits         2202   2202           
  Misses        105    105
```


Patch:
diff --git a/changelog/794.feature.rst b/changelog/794.feature.rst
new file mode 100644
index 00000000..085334b9
--- /dev/null
+++ b/changelog/794.feature.rst
@@ -0,0 +1,5 @@
+Add support to explicitly invoke interpreter directives for environments with
+long path lengths. In the event that ``tox`` cannot invoke scripts with a
+system-limited shebang (e.x. a Linux host running a Jenkins Pipeline), a user
+can set the environment variable ``TOX_LIMITED_SHEBANG`` to workaround the
+system's limitation (e.x. ``export TOX_LIMITED_SHEBANG=1``) - by @jdknight
diff --git a/doc/config-v2.rst b/doc/config-v2.rst
deleted file mode 100644
index cfb24dc2..00000000
--- a/doc/config-v2.rst
+++ /dev/null
@@ -1,290 +0,0 @@
-V2: new tox multi-dimensional, platform-specific configuration
---------------------------------------------------------------------
-
-.. note::
-
-   This is a draft document sketching a to-be-done implementation.
-   It does not fully specify each change yet but should give a good
-   idea of where things are heading.  For feedback, mail the
-   testing-in-python mailing list or open a pull request at
-   https://github.com/tox-dev/tox.
-
-**Abstract**: Adding multi-dimensional configuration, platform-specification
-and multiple installers to tox.ini.
-
-**Target audience**: Developers using or wanting to use tox for testing
-their python projects.
-
-Issues with current tox (1.4) configuration
-------------------------------------------------
-
-tox is used as a tool for creating and managing virtualenv environments
-and running tests in them. As of tox-1.4 there are some issues frequently
-coming up with its configuration language:
-
-- there is no way to instruct tox to parametrize testenv specifications
-  other than to list all combinations by specifying a ``[testenv:...]``
-  section for each combination. Examples of real life situations
-  arising from this:
-
-  * https://github.com/encode/django-rest-framework/blob/b001a146d73348af18cfc4c943d87f2f389349c9/tox.ini
-
-  * https://bitbucket.org/tabo/django-treebeard/src/93b579395a9c/tox.ini
-
-- there is no way to have platform specific settings other than to
-  define specific testenvs and invoke tox with a platform-specific
-  testenv list.
-
-- there is no way to specify the platforms against which a project
-  shall successfully run.
-
-- tox always uses pip for installing packages currently.  This has
-  several issues:
-
-  - no way to check if installing via easy_install works
-  - no installs of packages with compiled c-extensions (win32 standard)
-
-
-Goals, resolving those issues
-------------------------------------
-
-This document discusses a possible solution for each of these issues,
-namely these goals:
-
-- allow to more easily define and run dependency/interpreter variants
-  with testenvs
-- allow platform-specific settings
-- allow to specify platforms against which tests should run
-- allow to run installer-variants (easy_install or pip, xxx)
-- try to mimick/re-use bash-style syntax to ease learning curve.
-
-
-Example: Generating and selecting variants
-----------------------------------------------
-
-Suppose you want to test your package against python3.6, python2.7 and on the
-windows and linux platforms.  Today you would have to
-write down 2*2 = 4 ``[testenv:*]`` sections and then instruct
-tox to run a specific list of environments on each platform.
-
-With tox-1.X you can directly specify combinations::
-
-    # combination syntax gives 2 * 2 = 4 testenv names
-    #
-    envlist = {py27,py36}-{win,linux}
-
-    [testenv]
-    deps = pytest
-    platform =
-           win: windows
-           linux: linux
-    basepython =
-           py27: python2.7
-           py36: python3.6
-    commands = pytest
-
-Let's go through this step by step::
-
-    envlist = {py27,py36}-{windows,linux}
-
-This is bash-style syntax and will create ``2*2=4`` environment names
-like this::
-
-    py27-windows
-    py27-linux
-    py36-windows
-    py36-linux
-
-Our ``[testenv]`` uses a new templating style for the ``platform`` definition::
-
-    platform=
-           windows: windows
-           linux: linux
-
-These two conditional settings will lead to either ``windows`` or
-``linux`` as the platform string.  When the test environment is run,
-its platform string needs to be contained in the string returned
-from ``platform.platform()``. Otherwise the environment will be skipped.
-
-The next configuration item in the ``testenv`` section deals with
-the python interpreter::
-
-    basepython =
-           py27: python2.7
-           py36: python3.6
-
-This defines a python executable, depending on if ``py36`` or ``py27``
-appears in the environment name.
-
-The last config item is simply the invocation of the test runner::
-
-    commands = pytest
-
-Nothing special here :)
-
-.. note::
-
-    tox provides good defaults for platform and basepython
-    settings, so the above ini-file can be further reduced::
-
-        [tox]
-        envlist = {py27,py36}-{win,linux}
-
-        [testenv]
-        deps = pytest
-        commands = pytest
-
-    Voila, this multi-dimensional ``tox.ini`` configuration
-    defines 2*2=4 environments.
-
-
-The new "platform" setting
---------------------------------------
-
-A testenv can define a new ``platform`` setting.  If its value
-is not contained in the string obtained from calling
-``sys.platform`` the environment will be skipped.
-
-Expanding the ``envlist`` setting
-----------------------------------------------------------
-
-The new ``envlist`` setting allows to use ``{}`` bash-style
-expressions.  XXX explanation or pointer to bash-docs
-
-Templating based on environment names
--------------------------------------------------
-
-For a given environment name, all lines in a testenv section which
-start with "NAME: ..." will be checked for being part in the environment
-name.  If they are part of it, the remainder will be the new line.
-If they are not part of it, the whole line will be left out.
-Parts of an environment name are obtained by ``-``-splitting it.
-
-Variant specification with [variant:VARNAME]
-
-Showing all expanded sections
--------------------------------
-
-To help with understanding how the variants will produce section values,
-you can ask tox to show their expansion with a new option::
-
-    $ tox -l [XXX output omitted for now]
-
-Making sure your package installs with easy_install
-------------------------------------------------------
-
-The new "installer" testenv setting allows to specify the tool for
-installation in a given test environment::
-
-    [testenv]
-    installer =
-        easy: easy_install
-        pip: pip
-
-If you want to have your package installed with both easy_install
-and pip, you can list them in your envlist like this::
-
-    [tox]
-    envlist = py[27,35,36]-django[13,14]-[easy,pip]
-
-If no installer is specified, ``pip`` will be used.
-
-Default settings related to environment names/variants
----------------------------------------------------------------
-
-tox comes with predefined settings for certain variants, namely:
-
-* ``{easy,pip}`` use easy_install or pip respectively
-* ``{py27,py34,py35,py36,pypy19}`` use the respective
-  pythonNN or PyPy interpreter
-* ``{win32,linux,darwin}`` defines the according ``platform``.
-
-You can use those in your “envlist” specification
-without the need to define them yourself.
-
-
-Use more bash-style syntax
---------------------------------------
-
-tox leverages bash-style syntax if you specify mintoxversion = 1.4:
-
-- $VARNAME or ${...} syntax instead of the older {} substitution.
-- XXX go through config.rst and see how it would need to be changed
-
-
-Transforming the examples: django-rest
-------------------------------------------------
-
-The original `django-rest-framework tox.ini
-<https://github.com/encode/django-rest-framework/blob/b001a146d73348af18cfc4c943d87f2f389349c9/tox.ini>`_
-file has 159 lines and a lot of repetition, the new one would have ``20+``
-lines and almost no repetition::
-
-     [tox]
-     envlist = {py27,py35,py36}-{django12,django13}{,-example}
-
-     [testenv]
-     deps=
-         coverage==3.4
-         unittest-xml-reporting==1.2
-         Pyyaml==3.10
-         django12: django==1.2.4
-         django13: django==1.3.1
-         # some more deps for running examples
-         example: wsgiref==0.1.2
-         example: Pygments==1.4
-         example: httplib2==0.6.0
-         example: Markdown==2.0.3
-
-     commands =
-        !example: python setup.py test
-        example: python examples/runtests.py
-
-
-Note that ``{,-example}`` in the envlist denotes two values, an empty
-one and a ``example`` one.  The empty value means that there are no specific
-settings and thus no need to define a variant name.
-
-Transforming the examples: django-treebeard
-------------------------------------------------
-
-Another `tox.ini
-<https://bitbucket.org/tabo/django-treebeard/raw/93b579395a9c/tox.ini>`_
-has 233 lines and runs tests against multiple Postgres and Mysql
-engines.  It also performs backend-specific test commands, passing
-different command line options to the test script.  With the new tox-1.X
-we not only can do the same with 32 non-repetive configuration lines but
-we also produce 36 specific testenvs with specific dependencies and test
-commands::
-
-    [tox]
-    envlist =
-     {py27,py34,py35,py36}-{django11,django12,django13}-{nodb,pg,mysql}, docs
-
-    [testenv:docs]
-    changedir = docs
-    deps =
-        Sphinx
-        Django
-    commands =
-        make clean
-        make html
-
-     [testenv]
-     deps=
-           coverage
-           pysqlite
-           django11: django==1.1.4
-           django12: django==1.2.7
-           django13: django==1.3.1
-           django14: django==1.4
-           nodb: pysqlite
-           pg: psycopg2
-           mysql: MySQL-python
-
-     commands =
-         nodb: {envpython} runtests.py {posargs}
-         pg: {envpython} runtests.py --DATABASE_ENGINE=postgresql_psycopg2 \
-                                     --DATABASE_USER=postgres {posargs}
-         mysql: {envpython} runtests.py --DATABASE_ENGINE=mysql \
-                                        --DATABASE_USER=root {posargs}
diff --git a/doc/config.rst b/doc/config.rst
index cf1d9ec4..5a40420d 100644
--- a/doc/config.rst
+++ b/doc/config.rst
@@ -699,6 +699,26 @@ With the previous configuration, it will install:
 - ``flake8`` package for ``py36`` environment.
 - ``flake8`` and ``coverage`` packages for ``coverage`` environment.
 
+Advanced settings
+-----------------
+
+Handle interpreter directives with long lengths
++++++++++++++++++++++++++++++++++++++++++++++++
+
+For systems supporting executable text files (scripts with a shebang), the
+system will attempt to parse the interpreter directive to determine the program
+to execute on the target text file. When ``tox`` prepares a virtual environment
+in a file container which has a large length (e.x. using Jenkins Pipelines), the
+system might not be able to invoke shebang scripts which define interpreters
+beyond system limits (e.x. Linux as a limit of 128; ``BINPRM_BUF_SIZE``). To
+workaround an environment which suffers from an interpreter directive limit, a
+user can bypass the system's interpreter parser by defining the
+``TOX_LIMITED_SHEBANG`` environment variable before invoking ``tox``::
+
+    export TOX_LIMITED_SHEBANG=1
+
+When the workaround is enabled, all tox-invoked text file executables will have
+their interpreter directive parsed by and explicitly executed by ``tox``.
 
 Other Rules and notes
 =====================
diff --git a/doc/index.rst b/doc/index.rst
index 1ffe3ab3..59697fce 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -100,7 +100,6 @@ Current features
    install
    examples
    config
-   config-v2
    support
    changelog
    links
diff --git a/tox/venv.py b/tox/venv.py
index 1d52e521..1f05c3a1 100755
--- a/tox/venv.py
+++ b/tox/venv.py
@@ -400,6 +400,8 @@ class VirtualEnv(object):
 
         cwd.ensure(dir=1)
         args[0] = self.getcommandpath(args[0], venv, cwd)
+        if sys.platform != 'win32' and 'TOX_LIMITED_SHEBANG' in os.environ:
+            args = prepend_shebang_interpreter(args)
         env = self._getenv(testcommand=testcommand)
         bindir = str(self.envconfig.envbindir)
         env['PATH'] = p = os.pathsep.join([bindir, os.environ["PATH"]])
@@ -415,6 +417,30 @@ def getdigest(path):
     return path.computehash()
 
 
+def prepend_shebang_interpreter(args):
+    # prepend interpreter directive (if any) to argument list
+    #
+    # When preparing virtual environments in a file container which has large
+    # length, the system might not be able to invoke shebang scripts which
+    # define interpreters beyond system limits (e.x. Linux as a limit of 128;
+    # BINPRM_BUF_SIZE). This method can be used to check if the executable is
+    # a script containing a shebang line. If so, extract the interpreter (and
+    # possible optional argument) and prepend the values to the provided
+    # argument list. tox will only attempt to read an interpreter directive of
+    # a maximum size of 2048 bytes to limit excessive reading and support UNIX
+    # systems which may support a longer interpret length.
+    try:
+        with open(args[0], 'rb') as f:
+            if f.read(1) == b'#' and f.read(1) == b'!':
+                MAXINTERP = 2048
+                interp = f.readline(MAXINTERP).rstrip()
+                interp_args = interp.split(None, 1)[:2]
+                return interp_args + args
+    except IOError:
+        pass
+    return args
+
+
 @tox.hookimpl
 def tox_testenv_create(venv, action):
     config_interpreter = venv.getsupportedinterpreter()


Test patch:
diff --git a/tests/test_venv.py b/tests/test_venv.py
index 5ff601b0..912f03f6 100644
--- a/tests/test_venv.py
+++ b/tests/test_venv.py
@@ -7,7 +7,8 @@ import pytest
 import tox
 from tox.interpreters import NoInterpreterInfo
 from tox.venv import (
-    CreationConfig, VirtualEnv, getdigest, tox_testenv_create, tox_testenv_install_deps)
+    CreationConfig, VirtualEnv, getdigest, prepend_shebang_interpreter,
+    tox_testenv_create, tox_testenv_install_deps)
 
 
 def test_getdigest(tmpdir):
@@ -761,3 +762,108 @@ def test_tox_testenv_pre_post(newmocksession):
     assert log == []
     mocksession.runtestenv(venv)
     assert log == ['started', 'finished']
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_empty_instance(tmpdir):
+    testfile = tmpdir.join('check_shebang_empty_instance.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # empty instance
+    testfile.write('')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_empty_interpreter(tmpdir):
+    testfile = tmpdir.join('check_shebang_empty_interpreter.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # empty interpreter
+    testfile.write('#!')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_empty_interpreter_ws(tmpdir):
+    testfile = tmpdir.join('check_shebang_empty_interpreter_ws.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # empty interpreter (whitespaces)
+    testfile.write('#!    \n')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_interpreter_simple(tmpdir):
+    testfile = tmpdir.join('check_shebang_interpreter_simple.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # interpreter (simple)
+    testfile.write('#!interpreter')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == [b'interpreter'] + base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_interpreter_ws(tmpdir):
+    testfile = tmpdir.join('check_shebang_interpreter_ws.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # interpreter (whitespaces)
+    testfile.write('#!  interpreter  \n\n')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == [b'interpreter'] + base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_interpreter_arg(tmpdir):
+    testfile = tmpdir.join('check_shebang_interpreter_arg.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # interpreter with argument
+    testfile.write('#!interpreter argx\n')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == [b'interpreter', b'argx'] + base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_interpreter_args(tmpdir):
+    testfile = tmpdir.join('check_shebang_interpreter_args.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # interpreter with argument (ensure single argument)
+    testfile.write('#!interpreter argx argx-part2\n')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == [b'interpreter', b'argx argx-part2'] + base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_real(tmpdir):
+    testfile = tmpdir.join('check_shebang_real.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # interpreter (real example)
+    testfile.write('#!/usr/bin/env python\n')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == [b'/usr/bin/env', b'python'] + base_args
+
+
+@pytest.mark.skipif("sys.platform == 'win32'")
+def test_tox_testenv_interpret_shebang_long_example(tmpdir):
+    testfile = tmpdir.join('check_shebang_long_example.py')
+    base_args = [str(testfile), 'arg1', 'arg2', 'arg3']
+
+    # interpreter (long example)
+    testfile.write(
+        '#!this-is-an-example-of-a-very-long-interpret-directive-what-should-'
+        'be-directly-invoked-when-tox-needs-to-invoked-the-provided-script-'
+        'name-in-the-argument-list')
+    args = prepend_shebang_interpreter(base_args)
+    assert args == [
+        b'this-is-an-example-of-a-very-long-interpret-directive-what-should-be-'
+        b'directly-invoked-when-tox-needs-to-invoked-the-provided-script-name-'
+        b'in-the-argument-list'] + base_args