Repository: scipy/oldest-supported-numpy (language: python)

Issue:
Ideas for improving testing to catch potential issues
> I have no idea how to test for such conflicts before releasing the package

_Originally posted by @rgommers in https://github.com/scipy/oldest-supported-numpy/issues/33#issuecomment-990030204_

One option here would be to parse the requirements of the package, and run them through a sanity check for a certain set of supported environments to ensure that it produces a correct result (i.e. exactly one pin).

The sanity check would be parsing each requirement declaration using `packaging.requirements.Requirement`, filtering with `req.marker.evaluate(...)` . The environment can be specified as a matrix of potential values for various things from the table in https://www.python.org/dev/peps/pep-0508/#environment-markers that you'd actually care about.

This should be feasible as a parameterized test in pytest.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 78099de..e6487ec 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -3,33 +3,10 @@ name: Basic tests
 on: [push, pull_request]
 
 jobs:
-  build:
-
+  test:
     runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version:
-        - 3.5
-        - 3.6
-        - 3.7
-        - 3.8
-        - 3.9
-        - pypy3
-        - '3.10.0-rc.1 - 3.10.0'
-
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Upgrade pip and setuptools
-      run: python -m pip install pip setuptools --upgrade
-    - name: Install wheel
-      if: matrix.python-version == '3.5'
-      run: python -m pip install wheel
-    - name: Install oldest-supported-numpy
-      run: python -m pip install -v .
-    - name: Check that numpy was installed
-      run: python -c 'import numpy'
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+      - run: pip install -r tests/requirements.txt
+      - run: pytest
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..2fb97ee
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,2 @@
+prune .github
+prune tests
diff --git a/setup.cfg b/setup.cfg
index f571905..1cec784 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -18,9 +18,8 @@ version = 0.14
 [options]
 python_requires = >=3.5
 install_requires =
-    # NOTE: platform_system 'AIX' and platform_machine 'aarch64' are mutually
-    #       exclusive so they are specified by themself when setting their
-    #       numpy version.
+    # NOTE: Any platform-related assumptions made here are encoded in the
+    #       tests for this package.
 
     # AIX requires fixes contained in numpy 1.16
     numpy==1.16.0; python_version=='3.5' and platform_system=='AIX'


Test patch:
diff --git a/tests/requirements.txt b/tests/requirements.txt
new file mode 100644
index 0000000..b6ca80d
--- /dev/null
+++ b/tests/requirements.txt
@@ -0,0 +1,2 @@
+pytest
+packaging
diff --git a/tests/test_dependencies.py b/tests/test_dependencies.py
new file mode 100644
index 0000000..e1c9191
--- /dev/null
+++ b/tests/test_dependencies.py
@@ -0,0 +1,73 @@
+"""Tests to ensure that the dependency declarations are sane.
+"""
+
+import configparser
+import pprint
+from functools import cache
+from pathlib import Path
+
+import pytest
+from packaging.requirements import Requirement
+
+SETUP_CFG_FILE = Path(__file__).parent.parent / "setup.cfg"
+
+
+def is_pinned(requirement: Requirement) -> bool:
+    return "==" in str(requirement.specifier)
+
+
+@cache
+def get_package_dependencies() -> list[Requirement]:
+    """A cached reader for getting dependencies of this package."""
+    parser = configparser.ConfigParser()
+    parser.read(SETUP_CFG_FILE)
+
+    return [
+        Requirement(line)
+        for line in parser.get("options", "install_requires").splitlines()
+        if line
+    ]
+
+
+# The ordering of these markers is important, and is used in test names.
+# The tests, when run, look like: PyPy-3.6-Linux-aarch64` (bottom-first)
+@pytest.mark.parametrize("platform_machine", ["x86_64", "aarch64", "s390x", "arm64"])
+@pytest.mark.parametrize("platform_system", ["Linux", "Windows", "Darwin", "AIX"])
+@pytest.mark.parametrize("python_version", ["3.6", "3.7", "3.8", "3.9", "3.10", "3.11"])
+@pytest.mark.parametrize("platform_python_implementation", ["CPython", "PyPy"])
+def test_has_at_most_one_pinned_dependency(
+    platform_machine,
+    platform_system,
+    python_version,
+    platform_python_implementation,
+):
+    # These are known to be platforms that are not valid / possible at this time.
+    if platform_system == "AIX":
+        if platform_machine == "aarch64":
+            pytest.skip("AIX and aarch64 are mutually exclusive.")
+        if platform_python_implementation == "PyPy":
+            pytest.skip("AIX and PyPy are mutually exclusive.")
+
+    if platform_python_implementation == "PyPy" and (platform_machine != "x86_64"):
+        pytest.skip(f"PyPy is not supported on {platform_machine}.")
+
+    environment = {
+        "python_version": python_version,
+        "platform_python_implementation": platform_python_implementation,
+        "platform_machine": platform_machine,
+        "platform_system": platform_system,
+    }
+
+    filtered_requirements = []
+    for req in get_package_dependencies():
+        assert req.marker
+        if not req.marker.evaluate(environment):
+            continue
+        if not is_pinned(req):
+            continue
+
+        filtered_requirements.append(req)
+
+    assert (
+        len(filtered_requirements) <= 1
+    ), f"Expected no more than one pin.\n{pprint.pformat(environment)}"