Repository: pytest-dev/pytest
Pull request: Fix Module.name from full path without drive letter
Fix #7628


Problem statement:
pytest-6.0.1 does not find custom fixtures, test filename incorrect (just ..\..\)
```
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- c:\prog
ram files (x86)\python37-32\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.7.0', 'Platform': 'Windows-2008ServerR2-6.1.7601-SP1', '
Packages': {'pytest': '6.0.1', 'py': '1.9.0', 'pluggy': '0.13.1'}, 'Plugins': {'
html': '2.1.1', 'metadata': '1.10.0'}}
rootdir: C:\
plugins: html-2.1.1, metadata-1.10.0
collected 2 items

..\..\::test_install[uninstall_S0_0] ERROR (0 secs)

..\..\::test_install[ver_1.6.1_S0_1] ERROR (0 secs)


=================================== ERRORS ====================================
________________ ERROR at setup of test_install[uninstall_S0_0] _________________
file C:\install\tests\install_test.py, line 77
  @pytest.mark.parametrize("rel_info", release_dirs, ids=test_ids)
  def test_install(install_fixture, rel_info):
file C:\install\tests\install_test.py, line 26
  @pytest.fixture(scope='session')
  def module_init_fixture(session_fixture):
E       fixture 'session_fixture' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbin
ary, doctest_namespace, extra, install_fixture, metadata, module_init_fixture, m
onkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml
_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.
```

- [ ] output of `pip list` from the virtual environment you are using
```
Package            Version
------------------ -------
atomicwrites       1.4.0
attrs              19.3.0
bcrypt             3.1.7
cffi               1.14.1
colorama           0.4.3
cryptography       3.0
importlib-metadata 1.7.0
iniconfig          1.0.1
json-flatten       0.1
more-itertools     8.4.0
packaging          20.4
paramiko           2.7.1
pip                10.0.1
pluggy             0.13.1
py                 1.9.0
pycparser          2.20
PyNaCl             1.4.0
pyparsing          2.4.7
pytest             6.0.1
pytest-html        2.1.1
pytest-metadata    1.10.0
scp                0.13.2
selenium           3.141.0
setuptools         39.0.1
six                1.15.0
toml               0.10.1
urllib3            1.25.10
zipp               3.1.0
```

 Python 3.7.0, pytest-6.0.1
OS Name:                   Microsoft Windows Server 2008 R2 Enterprise
OS Version:                6.1.7601 Service Pack 1 Build 7601

The same test work with pytest-5.3.2

Hints from issue discussion:
Hi @naugustin - can you share a reproducing example that we can run?  The test output helps, but being able to replay your problem locally is a lot better :slightly_smiling_face: 
Thank you.

I mimicked my fixture order with simple script. I am able to reproduce it. please find attached the script. Simply unzip under c:\ and run below command.
[test.zip](https://github.com/pytest-dev/pytest/files/5053060/test.zip)

```
pytest -s -v --show-capture=no \test\test\tests\dummy_test.py
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- c:\prog
ram files (x86)\python37-32\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.7.0', 'Platform': 'Windows-2008ServerR2-6.1.7601-SP1', '
Packages': {'pytest': '6.0.1', 'py': '1.9.0', 'pluggy': '0.13.1'}, 'Plugins': {'
html': '2.1.1', 'metadata': '1.10.0'}}
rootdir: C:\
plugins: html-2.1.1, metadata-1.10.0
collected 1 item

..\..\::test_dummy ERROR

=================================== ERRORS ====================================
________________________ ERROR at setup of test_dummy _________________________
file C:\test\test\tests\dummy_test.py, line 9
  def test_dummy(session_fixture2):
E       fixture 'session_fixture2' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbin
ary, doctest_namespace, extra, metadata, module_fixture, monkeypatch, pytestconf
ig, record_property, record_testsuite_property, record_xml_attribute, recwarn, t
mp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

C:\test\test\tests\dummy_test.py:9
=========================== short test summary info ===========================
ERROR ..\..\::test_dummy
============================== 1 error in 0.03s ===============================
```
can you try with pytest master, i recall that @bluetech just recently landed some enhancements for path handling that might help that (i don't have a windows box)
I can reproduce the problem even with current `master`. The key to make this fail is to use an absolute path but without the drive letter:

```
d:\projects\pytest (HEAD detached at 92af2e2 -> origin)
(.env37) λ pytest \projects\pytest\.tmp\drive\tests\dummy_test.py
==fails==
```

Including the drive letter the test passes:

```
d:\projects\pytest (master -> origin)
(.env37) λ pytest d:\projects\pytest\.tmp\drive\tests\dummy_test.py
==passes==
```

I can investigate this later. 👍 
Quick hunch in case somebody wants to investigate this before me:

```python
>>> from pathlib import Path
>>> Path(r'd:\projects\pytest\.tmp\drive\tests\dummy_test.py') == Path(r'\projects\pytest\.tmp\drive\tests\dummy_test.py')
False
>>> Path(r'd:\projects\pytest\.tmp\drive\tests\dummy_test.py') == Path(r'\projects\pytest\.tmp\drive\tests\dummy_test.py').absolute()
True
```
@naugustin meanwhile a workaround is to pass the full path including the drive letter.
Thank you.

Yes. it works if i specify the test file starting with C:\
I ran the code from above in Python 3.8.5 and could not reproduce the results. Both comparisons result in False. I ran this in the context of my C: drive.
```
>>> from pathlib import Path
>>> Path(r'd:\projects\pytest\.tmp\drive\tests\dummy_test.py') == Path(r'\projects\pytest\.tmp\drive\tests\dummy_test.py')
False
>>> Path(r'd:\projects\pytest\.tmp\drive\tests\dummy_test.py') == Path(r'\projects\pytest\.tmp\drive\tests\dummy_test.py').absolute()
False
>>> Path(r'd:\projects\pytest\.tmp\drive\tests\dummy_test.py')
WindowsPath('d:/projects/pytest/.tmp/drive/tests/dummy_test.py')
>>>Path(r'\projects\pytest\.tmp\drive\tests\dummy_test.py')
WindowsPath('/projects/pytest/.tmp/drive/tests/dummy_test.py')
>>> Path(r'\projects\pytest\.tmp\drive\tests\dummy_test.py').absolute()
WindowsPath('C:/projects/pytest/.tmp/drive/tests/dummy_test.py')
```
Also running `pytest` in the directory of the extracted `test.zip` provided above discovers a test in `tests\dummy_test.py` which passes.
> . Both comparisons result in False. I ran this in the context of my C: drive.

You need to be in the same drive as the drives of the paths being used. If you are on the C: drive, you should use `C:` paths, because `absolute()` will use the current directory to construct the full path. In the examples above, just replace `d:` by `c:` and you will see the same results.

> Also running pytest in the directory of the extracted test.zip provided above discovers a test in tests\dummy_test.py which passes.

It is important to pass the *full path without a drive letter* to pytest, so if you extracted that zip to `C:\temp`, you need to execute pytest from the C: drive and pass `\temp\tests` to it.

Commit message: Fix Module.name from full path without drive letter

Fix #7628

Patch to review:
diff --git a/changelog/7628.bugfix.rst b/changelog/7628.bugfix.rst
new file mode 100644
index 00000000000..9f3480aaa65
--- /dev/null
+++ b/changelog/7628.bugfix.rst
@@ -0,0 +1 @@
+Fix test collection when a full path without a drive letter was passed to pytest on Windows (for example ``\projects\tests\test.py`` instead of ``c:\projects\tests\pytest.py``).
diff --git a/src/_pytest/main.py b/src/_pytest/main.py
index c0e1c9d076e..e0ca1c0b58a 100644
--- a/src/_pytest/main.py
+++ b/src/_pytest/main.py
@@ -31,6 +31,7 @@
 from _pytest.config.argparsing import Parser
 from _pytest.fixtures import FixtureManager
 from _pytest.outcomes import exit
+from _pytest.pathlib import absolutepath
 from _pytest.pathlib import Path
 from _pytest.pathlib import visit
 from _pytest.reports import CollectReport
@@ -693,15 +694,15 @@ def _parsearg(self, arg: str) -> Tuple[py.path.local, List[str]]:
         strpath, *parts = str(arg).split("::")
         if self.config.option.pyargs:
             strpath = self._tryconvertpyarg(strpath)
-        relpath = strpath.replace("/", os.sep)
-        fspath = self.config.invocation_dir.join(relpath, abs=True)
-        if not fspath.check():
+        fspath = Path(str(self.config.invocation_dir), strpath)
+        fspath = absolutepath(fspath)
+        if not fspath.exists():
             if self.config.option.pyargs:
                 raise UsageError(
                     "file or package not found: " + arg + " (missing __init__.py?)"
                 )
             raise UsageError("file not found: " + arg)
-        return (fspath, parts)
+        return py.path.local(str(fspath)), parts
 
     def matchnodes(
         self, matching: Sequence[Union[nodes.Item, nodes.Collector]], names: List[str],
diff --git a/testing/test_collection.py b/testing/test_collection.py
index 9f22f3ee038..283df082f7b 100644
--- a/testing/test_collection.py
+++ b/testing/test_collection.py
@@ -1426,3 +1426,42 @@ def test_modules_not_importable_as_side_effect(self, testdir):
                 "* 1 failed in *",
             ]
         )
+
+
+def test_module_full_path_without_drive(testdir):
+    """Collect and run test using full path except for the drive letter (#7628)
+
+    Passing a full path without a drive letter would trigger a bug in py.path.local
+    where it would keep the full path without the drive letter around, instead of resolving
+    to the full path, resulting in fixtures node ids not matching against test node ids correctly.
+    """
+    testdir.makepyfile(
+        **{
+            "project/conftest.py": """
+                import pytest
+                @pytest.fixture
+                def fix(): return 1
+            """,
+        }
+    )
+
+    testdir.makepyfile(
+        **{
+            "project/tests/dummy_test.py": """
+                def test(fix):
+                    assert fix == 1
+            """
+        }
+    )
+    fn = testdir.tmpdir.join("project/tests/dummy_test.py")
+    assert fn.isfile()
+
+    drive, path = os.path.splitdrive(str(fn))
+
+    result = testdir.runpytest(path, "-v")
+    result.stdout.fnmatch_lines(
+        [
+            os.path.join("project", "tests", "dummy_test.py") + "::test PASSED *",
+            "* 1 passed in *",
+        ]
+    )


Code review comments:
diff_hunk: @@ -770,3 +741,59 @@ def genitems(
                 for subnode in rep.result:
                     yield from self.genitems(subnode)
             node.ihook.pytest_collectreport(report=rep)
+
+
+def search_pypath(module_name: str) -> str:
+    """Search sys.path for the given a dotted module name, and return its file system path."""
+    try:
+        spec = importlib.util.find_spec(module_name)
+    # AttributeError: looks like package module, but actually filename
+    # ImportError: module does not exist
+    # ValueError: not a module name
+    except (AttributeError, ImportError, ValueError):
+        return module_name
+    if spec is None or spec.origin is None or spec.origin == "namespace":
+        return module_name
+    elif spec.submodule_search_locations:
+        return os.path.dirname(spec.origin)
+    else:
+        return spec.origin
+
+
+def resolve_path_argument(
path: src/_pytest/main.py
text: @user1:
I think "path argument" is perhaps *slightly* misleading given that it can be a dotted python path and the selection part.

Maybe `resolve_selection_argument`? `resolve_collection_argument`?

diff_hunk: @@ -693,15 +694,15 @@ def _parsearg(self, arg: str) -> Tuple[py.path.local, List[str]]:
         strpath, *parts = str(arg).split("::")
         if self.config.option.pyargs:
             strpath = self._tryconvertpyarg(strpath)
-        relpath = strpath.replace("/", os.sep)
-        fspath = self.config.invocation_dir.join(relpath, abs=True)
-        if not fspath.check():
+        fspath = Path(str(self.config.invocation_dir), strpath)
path: src/_pytest/main.py
text: @user1:
```suggestion
        fspath = self.config.invocation_params.dir / strpath
```

diff_hunk: @@ -770,3 +741,59 @@ def genitems(
                 for subnode in rep.result:
                     yield from self.genitems(subnode)
             node.ihook.pytest_collectreport(report=rep)
+
+
+def search_pypath(module_name: str) -> str:
+    """Search sys.path for the given a dotted module name, and return its file system path."""
+    try:
+        spec = importlib.util.find_spec(module_name)
+    # AttributeError: looks like package module, but actually filename
+    # ImportError: module does not exist
+    # ValueError: not a module name
+    except (AttributeError, ImportError, ValueError):
+        return module_name
+    if spec is None or spec.origin is None or spec.origin == "namespace":
+        return module_name
+    elif spec.submodule_search_locations:
+        return os.path.dirname(spec.origin)
+    else:
+        return spec.origin
+
+
+def resolve_path_argument(
+    invocation_dir: py.path.local, arg: str, *, as_pypath=False
path: src/_pytest/main.py
text: @user1:
```suggestion
    invocation_dir: py.path.local, arg: str, *, as_pypath: bool = False
```