Repository: spyder-ide/spyder

Issue:
runtests.py attempts to collect tests from subrepos if extra arguments are given
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

## Problem Description

Running pytests with extra arguments results in collection error.
Recommend updating runtests.py to add `'spyder'` in front of `extra_args`

### What steps reproduce the problem?

1. run `python runtests.py -k 'some_single_test'`

### What is the expected output? What do you see instead?

Should run single test, but instead attempts to load tests from subrepos in 'external-deps'.

### Paste Traceback/Error Below (if applicable)
<!--- Copy from error dialog or View > Panes > Internal Console --->

```
_________________________________________________ ERROR collecting test session __________________________________________________
Defining 'pytest_plugins' in a non-top-level conftest is no longer supported because it affects the entire directory tree in a non-explicit way.
  /Users/rclary/Documents/Python.git/developer/spyder/external-deps/python-language-server/test/conftest.py
Please move it to a top level conftest file at the rootdir:
  /Users/rclary/Documents/Python.git/developer/spyder
For more information, visit:
  https://docs.pytest.org/en/latest/deprecations.html#pytest-plugins-in-non-top-level-conftest-files
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================== 1 error in 0.38 seconds =====================================================
```




Discussion:
Thanks for the report @mrclary, indeed something we need to address.


Thanks for noticing @mrclary! This is very odd indeed.

@CAM-Gerlach, any ideas on how we could fix this?
It works fine if I include 'spyder' in the arguments, i.e.
```
$ python runtests.py 'spyder' -k 'some_single_test'
```
Could it be as simple as always including 'spyder' in the pytest_args?
```python
def run_pytest(run_slow=False, extra_args=None):
    """Run pytest tests for Spyder."""
    pytest_args = ['-vv', '-rw', '--durations=10']
    
    if CI:
        # Exit on first failure and show coverage
        pytest_args += ['-x', '--cov=spyder', '--no-cov-on-fail']

        # To display nice tests resume in Azure's web page
        if os.environ.get('AZURE', None) is not None:
            pytest_args += ['--cache-clear', '--junitxml=result.xml']
    if run_slow or RUN_SLOW:
        pytest_args += ['--run-slow']
    
    pytest_args += ['spyder']  # added line
    # Allow user to pass a custom test path to pytest to e.g. run just one test
    if extra_args:
        pytest_args += extra_args
    # else:
    #     pytest_args += ['spyder']
    print("Pytest Arguments: " + str(pytest_args))
    errno = pytest.main(pytest_args)

    # sys.exit doesn't work here because some things could be running in the
    # background (e.g. closing the main window) when this point is reached.
    # If that's the case, sys.exit doesn't stop the script as you would expect.
    if errno != 0:
        raise SystemExit(errno)
```
Perhaps we should simply remove the `-k` flag when passing it to `runtests.py`?
@mrclary This was the way it was originally, before ##11704, but unless there is something I'm missing (which there might be) this prevents the user from specifying a particular spyder subpackage or module to run tests in (as I often do, when I just want to collect and test the module I'm working on rather than everything). That was one of the motivating reasons for the change.

The presumption here is that if a developer is passing custom pytest args beyond what Spyder defines, they are responsible for passing the appropriate collection directory as well...we could parse the argv and determine if a positional arg is passed and if not insert `spyder`, but that seems like overkill (and a chance for yet more edge case bugs) just for functionality in a script used by a small proportion of active Spyder developers.
@CAM-Gerlach , I see. The real issue here is that pytest attempts to collect tests in 'external-deps', which it should not. Is there a way to just exclude that directory? Then 'spyder' wouldn't be needed at all.
> Perhaps we should simply remove the -k flag when passing it to runtests.py?

I'm not sure how I understand how stripping the user's explicitly passed arguments is intended to fix the problem, sorry.
@mrclary  We could use Pytest's `--ignore=path` option; see the [docs](https://docs.pytest.org/en/latest/example/pythoncollection.html). Does that fix the problem for you?
Aahh, yes, of course. There is an `--ignore=<path>`; we can just add that. It can be used multiple time, so if the user supplies one, its okay, but always ignore the 'external-deps' directory.
Great minds think alike? At the same time?
> There is an `--ignore=<path>`

This is a great idea! We could simply inject that option like this `--ignore=./external-deps`.

Who's interested in giving us a hand and creating a PR for that?
I can do that.
Great! Thanks a lot for your help Ryan!
Thanks so much for all your help @mrclary !

Resolving patch:
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -31,7 +31,8 @@
 
 def run_pytest(run_slow=False, extra_args=None):
     """Run pytest tests for Spyder."""
-    pytest_args = ['-vv', '-rw', '--durations=10']
+    # Be sure to ignore subrepos
+    pytest_args = ['-vv', '-rw', '--durations=10', '--ignore=./external-deps']
 
     if CI:
         # Exit on first failure and show coverage
@@ -45,8 +46,6 @@ def run_pytest(run_slow=False, extra_args=None):
     # Allow user to pass a custom test path to pytest to e.g. run just one test
     if extra_args:
         pytest_args += extra_args
-    else:
-        pytest_args += ['spyder']
 
     print("Pytest Arguments: " + str(pytest_args))
     errno = pytest.main(pytest_args)


Test patch: