Repository: smarie/python-pytest-cases (language: python)

Issue:
AUTO cases in conftest: AssertionError
Thanks for the great work! I just stumbled across a corner case.

Example to reproduce:

***conftest.py***
```python
from pytest_cases import fixture, parametrize_with_cases

@fixture
@parametrize_with_cases('arg')
def auto_cases(arg):
    return arg
```

***test_cases.py***
```python
def case():
    return 1
```

***test.py***
```python
def test(auto_cases):
    assert auto_cases == 1
```

Gives
```
ImportError while loading conftest
pytest_cases/common_pytest.py:925: in apply
    return apply_decorator(test_or_fixture_func, container)
pytest_cases/case_parametrizer_new.py:143: in _apply_parametrization
    cases_funs = get_all_cases(f, cases=cases, prefix=prefix, glob=glob, has_tag=has_tag, filter=filter)
pytest_cases/case_parametrizer_new.py:303: in get_all_cases
    c = import_default_cases_module(caller_module_name)
pytest_cases/case_parametrizer_new.py:670: in import_default_cases_module
    assert parts[-1][0:5] == 'test_'
E   AssertionError: assert 'conft' == 'test_'
```

Seems very reasonable to me to not allow `AUTO` use in a `conftest.py` as it can be ambiguous which cases are to be used for parametrization. However, it probably would be nicer to explicitly check for this and raise a clearer exception? E.g., at the beginning of
https://github.com/smarie/python-pytest-cases/blob/ab3b7190d728b18512141b9f5f3a1c3dfc7cedf2/src/pytest_cases/case_parametrizer_new.py#L650


As a side note, I realized a possible inconsistency in the doc of the `@parametrize_with_cases` API: it looks like the "second alternative" for looking up cases in `AUTO` mode is `cases_<name>.py` (plural), and not `case_<name>.py`. See
https://github.com/smarie/python-pytest-cases/blob/ab3b7190d728b18512141b9f5f3a1c3dfc7cedf2/src/pytest_cases/case_parametrizer_new.py#L671


Discussion / hints:
Thanks @michele-riva  ! Indeed I confirm

 - The docstring and API reference page state `By default (cases=AUTO) the list of test cases is automatically drawn from the python module file named test_<name>_cases.py or if not found, case_<name>.py, where test_<name> is the current module name.` while the correct alternate pattern is `cases_<name>.py`. Note that a few comments here and there in the code have to be fixed too as they mention 'case_'.
 - It would be beneficial to raise an explicit `ValueError` in `get_all_cases` just before `import_default_cases_module` is called ([here](https://github.com/smarie/python-pytest-cases/blob/ab3b7190d728b18512141b9f5f3a1c3dfc7cedf2/src/pytest_cases/case_parametrizer_new.py#L302)) if `test_module_name.split('.')[-1]` does not start with `test_`, stating that `AUTO` is not allowed outside of normal test files.

Would you like to propose a PR for these two mods ?

Sure, I can draft one right away.

Fix patch:
diff --git a/docs/api_reference.md b/docs/api_reference.md
index ded231dc..3a2f5db8 100644
--- a/docs/api_reference.md
+++ b/docs/api_reference.md
@@ -274,7 +274,7 @@ CaseType = Union[Callable, Type, ModuleRef]
 
 A decorator for test functions or fixtures, to parametrize them based on test cases. It works similarly to [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html): argnames represent a coma-separated string of arguments to inject in the decorated test function or fixture. The argument values (`argvalues` in [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html)) are collected from the various case functions found according to `cases`, and injected as lazy values so that the case functions are called just before the test or fixture is executed.
 
-By default (`cases=AUTO`) the list of test cases is automatically drawn from the python module file named `test_<name>_cases.py` or if not found, `case_<name>.py`,  where `test_<name>` is the current module name.
+By default (`cases=AUTO`) the list of test cases is automatically drawn from the python module file named `test_<name>_cases.py` or if not found, `cases_<name>.py`,  where `test_<name>` is the current module name.
 
 Finally, the `cases` argument also accepts an explicit case function, cases-containing class, module or module name; or a list containing any mix of these elements. Note that both absolute and relative module names are supported.
 
@@ -293,7 +293,7 @@ argvalues = get_parametrize_args(host_class_or_module_of_f, cases_funs)
 
  - `argnames`: same than in `@pytest.mark.parametrize`
  
- - `cases`: a case function, a class containing cases, a module object or a module name string (relative module names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module. `AUTO` (default) means that the module named `test_<name>_cases.py` or if not found, `case_<name>.py`, will be loaded, where `test_<name>.py` is the module file of the decorated function. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag` are selected, including those functions nested in classes following naming pattern `*Case*`. Nested subclasses are taken into account, as long as they follow the `*Case*` naming pattern. When classes are explicitly provided in the list, they can have any name and do not need to follow this `*Case*` pattern.
+ - `cases`: a case function, a class containing cases, a module object or a module name string (relative module names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module. `AUTO` (default) means that the module named `test_<name>_cases.py` or if not found, `cases_<name>.py`, will be loaded, where `test_<name>.py` is the module file of the decorated function. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag` are selected, including those functions nested in classes following naming pattern `*Case*`. Nested subclasses are taken into account, as long as they follow the `*Case*` naming pattern. When classes are explicitly provided in the list, they can have any name and do not need to follow this `*Case*` pattern.
 
  - `prefix`: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to denote different kind of cases, for example 'data_', 'algo_', 'user_', etc.
 
diff --git a/docs/changelog.md b/docs/changelog.md
index b583a8f2..205aec23 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -1,5 +1,16 @@
 # Changelog
 
+### 3.8.2 (in progress) - bugfixes
+
+ - Corrected API documentation (and comments) for the second file-name
+   pattern for `AUTO`-cases lookup (`cases_<name>.py` instead of
+   `case_<name>.py`). PR [#320](https://github.com/smarie/python-pytest-cases/pull/320)
+   by [@michele-riva](https://github.com/michele-riva).
+ - Fixed `AssertionError` on `AUTO` cases outside a 'normal' test module.
+   Fixes [#309](https://github.com/smarie/python-pytest-cases/issues/309). PR
+   [#320](https://github.com/smarie/python-pytest-cases/pull/320) by
+   [@michele-riva](https://github.com/michele-riva).
+
 ### 3.8.1 - bugfixes
 
  - Fixed `ScopeMismatch` with parametrized cases in non-trivial test


Test patch:
diff --git a/src/pytest_cases/case_parametrizer_new.py b/src/pytest_cases/case_parametrizer_new.py
index 63135a06..52d10116 100644
--- a/src/pytest_cases/case_parametrizer_new.py
+++ b/src/pytest_cases/case_parametrizer_new.py
@@ -106,7 +106,7 @@ def parametrize_with_cases(argnames,                # type: Union[str, List[str]
     :param argnames: same than in @pytest.mark.parametrize
     :param cases: a case function, a class containing cases, a module object or a module name string (relative module
         names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module.
-        `AUTO` (default) means that the module named `test_<name>_cases.py` or if not found, `case_<name>.py`, will be
+        `AUTO` (default) means that the module named `test_<name>_cases.py` or if not found, `cases_<name>.py`, will be
         loaded, where `test_<name>.py` is the module file of the decorated function. When a module is listed, all of
         its functions matching the `prefix`, `filter` and `has_tag` are selected, including those functions nested in
         classes following naming pattern `*Case*`. Nested subclasses are taken into account, as long as they follow the
@@ -224,7 +224,7 @@ def get_all_cases(parametrization_target=None,  # type: Callable
         names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module.
         `AUTO` (default) means that the module named `test_<name>_cases.py` will be loaded, where `test_<name>.py` is
         the module file of the decorated function. `AUTO2` allows you to use the alternative naming scheme
-        `case_<name>.py`. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag`
+        `cases_<name>.py`. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag`
         are selected, including those functions nested in classes following naming pattern `*Case*`. When classes are
         explicitly provided in the list, they can have any name and do not need to follow this `*Case*` pattern.
     :param prefix: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to
@@ -299,7 +299,18 @@ def get_all_cases(parametrization_target=None,  # type: Callable
         else:
             # module
             if c is AUTO:
-                # First try `test_<name>_cases.py` Then `case_<name>.py`
+                # Make sure we're in a test_<xxx>.py-like module.
+                # We cannot accept AUTO cases in, e.g., conftest.py
+                # as we don't know what to look for. We complain here
+                # rather than raising AssertionError in the call to
+                # import_default_cases_module. See #309.
+                if not caller_module_name.split('.')[-1].startswith('test_'):
+                    raise ValueError(
+                        'Cannot use `cases=AUTO` in file "%s". `cases=AUTO` is '
+                        'only allowed in files whose name starts with "test_" '
+                        % caller_module_name
+                        )
+                # First try `test_<name>_cases.py` Then `cases_<name>.py`
                 c = import_default_cases_module(caller_module_name)
 
             elif c is THIS_MODULE or c == '.':
@@ -680,7 +691,7 @@ def import_default_cases_module(test_module_name):
     try:
         cases_module = import_module(cases_module_name1)
     except ModuleNotFoundError:
-        # Then try `case_<name>.py`
+        # Then try `cases_<name>.py`
         parts = test_module_name.split('.')
         assert parts[-1][0:5] == 'test_'
         cases_module_name2 = "%s.cases_%s" % ('.'.join(parts[:-1]), parts[-1][5:])
diff --git a/tests/cases/issues/issue_309/conftest.py b/tests/cases/issues/issue_309/conftest.py
new file mode 100644
index 00000000..9e3dc2e4
--- /dev/null
+++ b/tests/cases/issues/issue_309/conftest.py
@@ -0,0 +1,14 @@
+from pytest_cases import fixture, get_all_cases
+from pytest_cases.common_others import AUTO
+
+
+def mock_parameterization_target():
+    """A callable to use as parametrization target."""
+
+
+@fixture
+def get_all_cases_auto_fails():
+    """Fail because we ask for AUTO cases in a non-'test_<...>' file."""
+    def _fail():
+        get_all_cases(mock_parameterization_target, cases=AUTO)
+    return _fail
diff --git a/tests/cases/issues/issue_309/test_issue_309.py b/tests/cases/issues/issue_309/test_issue_309.py
new file mode 100644
index 00000000..9eba22ce
--- /dev/null
+++ b/tests/cases/issues/issue_309/test_issue_309.py
@@ -0,0 +1,7 @@
+
+import pytest
+
+
+def test_get_all_cases_auto_raises(get_all_cases_auto_fails):
+    with pytest.raises(ValueError):
+        get_all_cases_auto_fails()