Repository: pandas-dev/pandas
Pull request: TST: Skip db tests unless explicitly specified in -m pattern
- [x] closes #24485

Since #24450 we don't skip the db tests if the connection fails. That's useful, because the CI won't fail silently if there is any problem with the databases. But as reported in #24485, it's common to have `pymysql` or `psycopg2` installed locally, but no database connection. This is making the tests fail locally, and the agreed solution is to skip the db tests unless they are explicitly requested. 

But pytest doesn't make it easy to skip tests by default when using `-m`, but this solution should work. Another option would be an `--include-db` option, but I think it makes reading the expressions in the build very difficult (like, what `--only-slow --include-db` runs?). With `-m` the expressions are much clearer (e.g. `slow and db`), so I think this option is better.

The `eval` is how `pytest` itself evaluates the value of the `-m` parameter.


Problem statement:
TST: psycopg2 errors when running tests locally
I just started noticing these (167 of them) in the logs from `python -m pytest pandas/tests --skip-slow`

```
_____________________________ ERROR at setup of TestPostgreSQLAlchemyConn.test_read_sql[load_iris_data0] ______________________________

cls = <pandas.tests.io.test_sql.TestPostgreSQLAlchemyConn object at 0x7f2843ab4a50>

    @pytest.fixture(autouse=True, scope='class')
    def setup_class(cls):
        cls.setup_import()
        cls.setup_driver()
        conn = cls.connect()
>       conn.connect()

[...]
>       conn = _connect(dsn, connection_factory=connection_factory, async=async)
E       OperationalError: (psycopg2.OperationalError) fe_sendauth: no password supplied

../../.local/lib/python2.7/site-packages/psycopg2/__init__.py:164: OperationalError
```

Hints from issue discussion:
Looking at setup.cfg, I guess I'm supposed to start specifying `--skip-db` too?
Skipping should probably be the default, right @datapythonista?
I thought about it, but as `pymysql` and `psycopg` are not in `environment.yml`, I didn't see a difference, as the tests should be skipped anyway.

Happy to change it, I don't have a preference.
I'd prefer skipping but default (but failing if the db tests are requested
and don't get setup correctly). I know that I have pyscopg2 installed in my
dev env, but don't have the expected databases setup right now.

On Sat, Dec 29, 2018 at 2:19 PM Marc Garcia <notifications@github.com>
wrote:

> I thought about it, but as pymysql and psycopg are not in environment.yml,
> I didn't see a difference, as the tests should be skipped anyway.
>
> Happy to change it, I don't have a preference.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/24485#issuecomment-450517431>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIrNIswmqe_-tEAL2uxwR70aJbeW5ks5u985dgaJpZM4Zkpg3>
> .
>

ok, will open a PR

Commit message: TST: Skip db tests unless explicitly specified in -m pattern

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 28b57cc750190..9f6a5f0c5d9aa 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,11 +34,11 @@ matrix:
     include:
     - dist: trusty
       env:
-        - JOB="3.7" ENV_FILE="ci/deps/travis-37.yaml" PATTERN="not slow and not network and not db"
+        - JOB="3.7" ENV_FILE="ci/deps/travis-37.yaml" PATTERN="not slow and not network"
 
     - dist: trusty
       env:
-        - JOB="2.7" ENV_FILE="ci/deps/travis-27.yaml" PATTERN="not slow"
+        - JOB="2.7" ENV_FILE="ci/deps/travis-27.yaml" PATTERN="not slow and db"
       addons:
         apt:
           packages:
@@ -46,11 +46,11 @@ matrix:
 
     - dist: trusty
       env:
-        - JOB="3.6, locale" ENV_FILE="ci/deps/travis-36-locale.yaml" PATTERN="not slow and not network" LOCALE_OVERRIDE="zh_CN.UTF-8"
+        - JOB="3.6, locale" ENV_FILE="ci/deps/travis-36-locale.yaml" PATTERN="not slow and not network and db" LOCALE_OVERRIDE="zh_CN.UTF-8"
 
     - dist: trusty
       env:
-        - JOB="3.6, coverage" ENV_FILE="ci/deps/travis-36.yaml" PATTERN="not slow and not network" PANDAS_TESTING_MODE="deprecate" COVERAGE=true
+        - JOB="3.6, coverage" ENV_FILE="ci/deps/travis-36.yaml" PATTERN="not slow and not network and db" PANDAS_TESTING_MODE="deprecate" COVERAGE=true
 
     # In allow_failures
     - dist: trusty
diff --git a/ci/azure/posix.yml b/ci/azure/posix.yml
index c0c4fb924a605..b9e0cd0b9258c 100644
--- a/ci/azure/posix.yml
+++ b/ci/azure/posix.yml
@@ -12,37 +12,37 @@ jobs:
         py35_np_120:
           ENV_FILE: ci/deps/azure-macos-35.yaml
           CONDA_PY: "35"
-          PATTERN: "not slow and not network and not db"
+          PATTERN: "not slow and not network"
 
       ${{ if eq(parameters.name, 'Linux') }}:
         py27_np_120:
           ENV_FILE: ci/deps/azure-27-compat.yaml
           CONDA_PY: "27"
-          PATTERN: "not slow and not network and not db"
+          PATTERN: "not slow and not network"
 
         py27_locale_slow_old_np:
           ENV_FILE: ci/deps/azure-27-locale.yaml
           CONDA_PY: "27"
-          PATTERN: "slow and not db"
+          PATTERN: "slow"
           LOCALE_OVERRIDE: "zh_CN.UTF-8"
           EXTRA_APT: "language-pack-zh-hans"
 
         py36_locale_slow:
           ENV_FILE: ci/deps/azure-36-locale_slow.yaml
           CONDA_PY: "36"
-          PATTERN: "not slow and not network and not db"
+          PATTERN: "not slow and not network"
           LOCALE_OVERRIDE: "it_IT.UTF-8"
 
         py37_locale:
           ENV_FILE: ci/deps/azure-37-locale.yaml
           CONDA_PY: "37"
-          PATTERN: "not slow and not network and not db"
+          PATTERN: "not slow and not network"
           LOCALE_OVERRIDE: "zh_CN.UTF-8"
 
         py37_np_dev:
           ENV_FILE: ci/deps/azure-37-numpydev.yaml
           CONDA_PY: "37"
-          PATTERN: "not slow and not network and not db"
+          PATTERN: "not slow and not network"
           TEST_ARGS: "-W error"
           PANDAS_TESTING_MODE: "deprecate"
           EXTRA_APT: "xsel"
diff --git a/ci/azure/windows.yml b/ci/azure/windows.yml
index f06b229bb2656..cece002024936 100644
--- a/ci/azure/windows.yml
+++ b/ci/azure/windows.yml
@@ -38,7 +38,7 @@ jobs:
       displayName: 'Build'
     - script: |
         call activate pandas-dev
-        pytest -m "not slow and not network and not db" --junitxml=test-data.xml pandas -n 2 -r sxX --strict --durations=10 %*
+        pytest -m "not slow and not network" --junitxml=test-data.xml pandas -n 2 -r sxX --strict --durations=10 %*
       displayName: 'Test'
     - task: PublishTestResults@2
       inputs:
diff --git a/pandas/conftest.py b/pandas/conftest.py
index f383fb32810e7..66f021ac201ca 100644
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -54,14 +54,23 @@ def pytest_runtest_setup(item):
     if 'network' in item.keywords and item.config.getoption("--skip-network"):
         pytest.skip("skipping due to --skip-network")
 
-    if 'db' in item.keywords and item.config.getoption("--skip-db"):
-        pytest.skip("skipping due to --skip-db")
-
     if 'high_memory' in item.keywords and not item.config.getoption(
             "--run-high-memory"):
         pytest.skip(
             "skipping high memory test since --run-high-memory was not set")
 
+    # if "db" not explicitly set in the -m pattern, we skip the db tests
+    import collections
+    if 'db' in item.keywords:
+        pattern = item.config.getoption('-m')
+        markers = collections.defaultdict(bool)
+        for marker in item.iter_markers():
+            markers[marker.name] = True
+        markers['db'] = False
+        db_in_pattern = not eval(pattern, {}, markers)
+        if not db_in_pattern:
+            pytest.skip('skipping db unless -m "db" is specified')
+
 
 # Configurations for all tests and all test modules
 
diff --git a/pandas/util/_tester.py b/pandas/util/_tester.py
index 18e8d415459fd..aad2f00fa0478 100644
--- a/pandas/util/_tester.py
+++ b/pandas/util/_tester.py
@@ -16,7 +16,7 @@ def test(extra_args=None):
         import hypothesis  # noqa
     except ImportError:
         raise ImportError("Need hypothesis>=3.58 to run tests")
-    cmd = ['--skip-slow', '--skip-network', '--skip-db']
+    cmd = ['--skip-slow', '--skip-network']
     if extra_args:
         if not isinstance(extra_args, list):
             extra_args = [extra_args]


Code review comments:
diff_hunk: @@ -54,14 +54,23 @@ def pytest_runtest_setup(item):
     if 'network' in item.keywords and item.config.getoption("--skip-network"):
         pytest.skip("skipping due to --skip-network")
 
-    if 'db' in item.keywords and item.config.getoption("--skip-db"):
-        pytest.skip("skipping due to --skip-db")
-
     if 'high_memory' in item.keywords and not item.config.getoption(
             "--run-high-memory"):
         pytest.skip(
             "skipping high memory test since --run-high-memory was not set")
 
+    # if "db" not explicitly set in the -m pattern, we skip the db tests
+    import collections
path: pandas/conftest.py
text: @user1:
can you move the import to the top