Repository: tox-dev/tox

Issue:
tox 3.7.0 doesn't fail on undefined factor (where 3.6 did)
In tox 3.6 and earlier, if your tox.ini delcares an `envlist`, running `tox -e py36-someUndefinedFactor` would exit with "ERROR: unknown environment 'py36-someUndefinedFactor'" (and without creating a venv).

In tox 3.7.0, the same command seems to ignore "someUndefinedFactor", and instead creates a venv and attempts to run the commands in it.

This makes it difficult to catch typos in commands or tox.ini.

## Reproduction

(I'm on Python 3.7.1, but don't think this matters. Full system info below.)

1. Create an empty project with this tox.ini. It *must* include an envlist to reproduce this bug. Notice the envlist *doesn't* cover "django22":
```ini
[tox]
envlist =
    py{36,37}-django{20,21}

[testenv]
deps =
    django20: django~=2.0.0
    django21: django~=2.1.0
skip_install = true
commands =
    python -c 'import django; print(django.__version__)'
```

2. `pip install tox==3.6.1`

3. Run `tox -e py36-django22` (django22 is *not* in the envlist).
   Results: "ERROR: unknown environment 'py36-django22'" (as expected)

4. `pip install tox==3.7.0`

5. Run `tox -e py36-django22` again.
   Expected: same unknown environment error as with tox 3.6.1.
   Actual results: tox attempts to create the environment and run the commands in it:
```terminal
py36-django22 create: .../.tox/py36-django22
py36-django22 run-test-pre: PYTHONHASHSEED='1187325010'
py36-django22 runtests: commands[0] | python -c 'import django; print(django.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'django'
ERROR: InvocationError for command '.../.tox/py36-django22/bin/python -c import django; print(django.__version__)' (exited with code 1)
```

## System info

macOS 10.14.2 
Python 3.7.1
pip list
```terminal
Package    Version
---------- -------
filelock   3.0.10
pip        19.0.2
pluggy     0.8.1
py         1.7.0
setuptools 40.8.0
six        1.12.0
toml       0.10.0
tox        3.6.1
virtualenv 16.4.0
wheel      0.32.3
```



Discussion:
medmunds: Heh. I *did* run `tox -e fix-lint` before committing, and it didn't make any changes or complain. (Because that env is actually spelled `fix_lint`, and there's [this bug](#1160), you see, where typos in envs aren't caught.)

But @codeclimate I stand by the readability of the test case I added. 😛 

Patch:
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index ce823339..7c9d371a 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -91,12 +91,12 @@ Short version
 #. `Fork the repository <https://help.github.com/articles/fork-a-repo/>`_.
 #. Make your changes.
 #. open a `pull request <https://help.github.com/articles/about-pull-requests/>`_ targeting the ``master`` branch.
-#. Follow **PEP-8**. There's a ``tox`` command to help fixing it: ``tox -e fix-lint``.
+#. Follow **PEP-8**. There's a ``tox`` command to help fixing it: ``tox -e fix_lint``.
    You can also add a pre commit hook to your local clone to run the style checks and fixes
-   (see hint after running ``tox -e fix-lint``)
+   (see hint after running ``tox -e fix_lint``)
 #. Tests for tox are (obviously) run using ``tox``::
 
-    tox -e fix-lint,py27,py36
+    tox -e fix_lint,py27,py36
 
    The test environments above are usually enough to cover most cases locally.
 
@@ -149,7 +149,7 @@ Here is a simple overview, with tox-specific bits:
    You need to have Python 2.7 and 3.6 available in your system.  Now
    running tests is as simple as issuing this command::
 
-    $ tox -e fix-lint,py27,py36
+    $ tox -e fix_lint,py27,py36
 
    This command will run tests via the "tox" tool against Python 2.7 and 3.6
    and also perform style checks with some automatic fixes.
diff --git a/docs/changelog/1160.bugfix.rst b/docs/changelog/1160.bugfix.rst
new file mode 100644
index 00000000..a207ef0b
--- /dev/null
+++ b/docs/changelog/1160.bugfix.rst
@@ -0,0 +1,1 @@
+Fix missing error for ``tox -e unknown`` when tox.ini declares ``envlist``. - by :user:`medmunds`
diff --git a/src/tox/config/__init__.py b/src/tox/config/__init__.py
index fe4f4bb0..ba013d9e 100644
--- a/src/tox/config/__init__.py
+++ b/src/tox/config/__init__.py
@@ -1053,7 +1053,7 @@ class ParseIni(object):
         # factors stated in config envlist
         stated_envlist = reader.getstring("envlist", replace=False)
         if stated_envlist:
-            for env in config.envlist:
+            for env in _split_env(stated_envlist):
                 known_factors.update(env.split("-"))
 
         # configure testenvs


Test patch:
diff --git a/tests/unit/config/test_config.py b/tests/unit/config/test_config.py
index 7cecd418..ed29a5e2 100644
--- a/tests/unit/config/test_config.py
+++ b/tests/unit/config/test_config.py
@@ -2147,6 +2147,15 @@ class TestGlobalOptions:
         env = config.envconfigs["py"]
         assert str(env.basepython) == sys.executable
 
+    def test_no_implicit_venv_from_cli_with_envlist(self, newconfig):
+        # See issue 1160.
+        inisource = """
+            [tox]
+            envlist = stated-factors
+        """
+        config = newconfig(["-etypo-factor"], inisource)
+        assert "typo-factor" not in config.envconfigs
+
     def test_correct_basepython_chosen_from_default_factors(self, newconfig):
         envlist = list(tox.PYTHON.DEFAULT_FACTORS.keys())
         config = newconfig([], "[tox]\nenvlist={}".format(", ".join(envlist)))
diff --git a/tests/unit/test_z_cmdline.py b/tests/unit/test_z_cmdline.py
index 2e29f1b7..7ade17aa 100644
--- a/tests/unit/test_z_cmdline.py
+++ b/tests/unit/test_z_cmdline.py
@@ -280,6 +280,21 @@ def test_unknown_environment(cmd, initproj):
     assert result.out == "ERROR: unknown environment 'qpwoei'\n"
 
 
+def test_unknown_environment_with_envlist(cmd, initproj):
+    initproj(
+        "pkg123",
+        filedefs={
+            "tox.ini": """
+            [tox]
+            envlist = py{36,37}-django{20,21}
+        """
+        },
+    )
+    result = cmd("-e", "py36-djagno21")
+    assert result.ret, "{}\n{}".format(result.err, result.out)
+    assert result.out == "ERROR: unknown environment 'py36-djagno21'\n"
+
+
 def test_minimal_setup_py_empty(cmd, initproj):
     initproj(
         "pkg123-0.7",