Repository: ansible/ansible
Pull request: ansible-test - Add work-around for pytest>=8 errors
##### SUMMARY

Resolves https://github.com/ansible/ansible/issues/82713

Adding the `--confcutdir` option to the `pytest` invocation prevents pytest >= 8 from searching for `conftest.py` in unwanted locations, such as all directories under `/home`. This can occur when running ansible-test from an install, where it sources a pytest config file from a path symlinked under `/tmp`.

Thanks to @sivel for digging into this and figuring out the `--confcutdir` option prevents the unwanted behavior.

##### ISSUE TYPE

Bugfix Pull Request


Problem statement:
ansible-test unit tests fail with pytest>=8
### Summary

When running ansible-test unit tests with pytest>=8 it fails with the following trace. Pinning pytest<8 in requirements.txt fixes the issue.
```
________________________ ERROR collecting test session _________________________
tests/output/.tmp/delegation/python3.9/lib/python3.9/site-packages/pluggy/_hooks.py:501: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
tests/output/.tmp/delegation/python3.9/lib/python3.9/site-packages/pluggy/_manager.py:119: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
tests/output/.tmp/delegation/python3.9/lib/python3.9/site-packages/_pytest/python.py:211: in pytest_collect_directory
    if pkginit.is_file():
/usr/lib/python3.9/pathlib.py:1456: in is_file
    return S_ISREG(self.stat().st_mode)
/usr/lib/python3.9/pathlib.py:1232: in stat
    return self._accessor.stat(self)
```

### Issue Type

Bug Report

### Component Name

ansible-test

### Ansible Version

```console
$ ansible --version
ansible [core 2.15.9]
  config file = None
  configured module search path = ['/home/ttuffin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ttuffin/tmp/venv/lib/python3.9/site-packages/ansible
  ansible collection location = /home/ttuffin/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ttuffin/tmp/venv/bin/ansible
  python version = 3.9.18 (main, Dec 14 2023, 16:09:27) [GCC 13.2.1 20230801] (/home/ttuffin/tmp/venv/bin/python3.9)
  jinja version = 3.1.3
  libyaml = True
$ ansible-test --version
ansible-test version 2.15.9
```


### Configuration

```console
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CONFIG_FILE() = None
EDITOR(env: EDITOR) = vim
PAGER(env: PAGER) = less -R
```


### OS / Environment

Arch Linux 

### Steps to Reproduce

1. Install a collection with tests that use the pytest library, e.g. `ansible.eda`:
```
$ ansible-galaxy collection install ansible.eda
```
2. CD to the collection dir:
```
$ cd ~/.ansible/collections/ansible_collections/ansible/eda
```
3. Run ansible-test:
```
$ ansible-test units --venv
```
4. It should fail with the above trace. Now pin pytest<8:
```
$ echo 'pytest<8' >> tests/unit/requirements.txt
```
5. Run ansible-test again. That trace should no longer be present

### Expected Results

ansible-test works with pytest>=8

### Actual Results

ansible-test fails with pytest>=8:
```console
________________________ ERROR collecting test session _________________________
tests/output/.tmp/delegation/python3.9/lib/python3.9/site-packages/pluggy/_hooks.py:501: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
tests/output/.tmp/delegation/python3.9/lib/python3.9/site-packages/pluggy/_manager.py:119: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
tests/output/.tmp/delegation/python3.9/lib/python3.9/site-packages/_pytest/python.py:211: in pytest_collect_directory
    if pkginit.is_file():
/usr/lib/python3.9/pathlib.py:1456: in is_file
    return S_ISREG(self.stat().st_mode)
/usr/lib/python3.9/pathlib.py:1232: in stat
    return self._accessor.stat(self)
E   PermissionError: [Errno 13] Permission denied: '/home/aur_builder/__init__.py'
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct

Hints from issue discussion:
Files identified in the description:

* [`bin/ansible-test`](https://github.com/ansible/ansible/blob/devel/bin/ansible-test)

If these files are incorrect, please update the `component name` section of the description or use the [component](https://github.com/ansible/ansibotmini#commands) bot command.

<!--- boilerplate: components_banner --->

I'm not sure this is a problem with ansible-test necessarily.  It looks like you have a pytest plugin installed that is attempting to read `/home/aur_builder/__init__.py` for some reason, and getting a permission error.  `pytest_collect_directory` takes an explicit directory, and one of the plugins is being pointed to scan `/home/aur_builder`.

Maybe it is impacted by your environment variables?  I'd check to see if you have anything in the output of `env` that references that other homedir.
@ttuffin So far I've been unable to reproduce this issue. Are there any additional details you can provide about the test environment that may help in reproducing this issue?
I'm able to reproduce the issue now.
This appears to be a bug in pytest 8. I'm able to reproduce the issue with only `pytest` -- no collection or ansible-test involved. The problem seems to be due to using `-c` when the config file resides in `/tmp`. So far I've only been able to reproduce it on a remote system using unprivileged users. I haven't yet figured out why I can't get the same reproducer to work locally.

There's at least one open pytest issue which may be related: https://github.com/pytest-dev/pytest/issues/11781
@mattclay what command did you run to reproduce it with pytest? Following my steps I was not able to reproduce this on a RHEL8.9 VM, as a privileged or unprivileged user. We did see this in our CI though (on Ubuntu) and on my local system (Arch).
`pytest -c /tmp/demo.ini` with an empty `/tmp/demo.ini` fails. However, there are other condition(s) required, which I haven't narrowed down to anything simple yet.
I finally figured how to reproduce it on a fresh system. As root:

1) `touch /tmp/demo.ini`
2) `mkdir /home/aaa`
3) `chmod 700 /home/aaa`

Then as a non-root user, install `pytest==8.0.1` in a venv and run `pytest -c /tmp/demo.ini`.

That will fail with: `PermissionError: [Errno 13] Permission denied: '/home/aaa/__init__.py'`

The issue seems to be that pytest wants access to other user's home directories when the config file is in `/tmp`.


If there are no home directories which are inaccessible, then the issue doesn't appear. That was the key part I was missing when trying to reproduce the issue earlier.

Commit message: ansible-test - Add work-around for pytest>=8 errors

Patch to review:
diff --git a/changelogs/fragments/ansible-test-pytest-8.yml b/changelogs/fragments/ansible-test-pytest-8.yml
new file mode 100644
index 00000000000000..be307bca06eb11
--- /dev/null
+++ b/changelogs/fragments/ansible-test-pytest-8.yml
@@ -0,0 +1,2 @@
+minor_changes:
+  - ansible-test - Add a work-around for permission denied errors when using pytest >= 8 on multi-user systems with an installed version of ansible-test.
diff --git a/test/lib/ansible_test/_internal/commands/units/__init__.py b/test/lib/ansible_test/_internal/commands/units/__init__.py
index 7656eebf5b4840..94272627cc9850 100644
--- a/test/lib/ansible_test/_internal/commands/units/__init__.py
+++ b/test/lib/ansible_test/_internal/commands/units/__init__.py
@@ -261,6 +261,7 @@ def command_units(args: UnitsConfig) -> None:
             '--junit-xml', os.path.join(ResultType.JUNIT.path, 'python%s-%s-units.xml' % (python.version, test_context)),
             '--strict-markers',  # added in pytest 4.5.0
             '--rootdir', data_context().content.root,
+            '--confcutdir', data_context().content.root,  # avoid permission errors when running from an installed version and using pytest >= 8
         ]  # fmt:skip
 
         if not data_context().content.collection:


Code review comments:
diff_hunk: @@ -0,0 +1,2 @@
+minor_changes:
+  - ansible-test - Add a work-around for permission denied errors when using pytest >= 8 on multi-user systems with an installed version of ansible-test.
path: changelogs/fragments/ansible-test-pytest-8.yml
text: @user1:
```suggestion
  - ansible-test - Add a work-around for permission denied errors when using ``pytest >= 8`` on multi-user systems with an installed version of ``ansible-test``.
```