Repository: strawberry-graphql/strawberry

Issue:
Use Tox for Matrix Testing
<!--- Provide a general summary of your changes in the title above. -->

<!--- This template is entirely optional and can be removed, but is here to help both you and us. -->
<!--- Anything on lines wrapped in comments like these will not show up in the final text. -->

## Description

* Refactor testing to use `tox` for dispatching tests.
  * Added `tox` as a dev dependency in multiple places.
  * Convert existing matrix testing for `starlette` and `django` to `tox` matrixes.
* Reorganized `tests/` subfolders by dependencies required, using those as delineators for `tox` environments.
* Added pytest hook to set markers for each `tests/` subfolder (AKA `tox` environment), removing existing ones that were just for `starlette` and `django`.

The above changes SHOULD allow one to continue to use `poetry` based virtual environments for ease of local testing, but also allows the use of `tox` locally to test against multiple versions of frameworks or python quickly and easily. I highly recommend using [pyenv](https://github.com/pyenv/pyenv) to set up and maintain an array of python versions for use with tox if you want to test with it locally.

## Types of Changes

<!--- What types of changes does your pull request introduce? Put an `x` in all the boxes that apply. -->
- [ ] Core
- [ ] Bugfix
- [ ] New feature
- [x] Enhancement/optimization
- [ ] Documentation

## Issues Fixed or Closed by This PR

* Closes #1917

## Checklist

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the CONTRIBUTING document.
- [x] I have added tests to cover my changes.
- [x] I have tested the changes and verified that they work and don't break anything (as well as I can manage).



Discussion:
# [Codecov](https://codecov.io/gh/strawberry-graphql/strawberry/pull/2247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=strawberry-graphql) Report
> Merging [#2247](https://codecov.io/gh/strawberry-graphql/strawberry/pull/2247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=strawberry-graphql) (8c92a75) into [main](https://codecov.io/gh/strawberry-graphql/strawberry/commit/e04038cf5d46e0d9a0de6771711baa7241245849?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=strawberry-graphql) (e04038c) will **decrease** coverage by `42.24%`.
> The diff coverage is `100.00%`.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##             main    #2247       +/-   ##
===========================================
- Coverage   98.15%   55.91%   -42.25%     
===========================================
  Files         162      157        -5     
  Lines        6521     6263      -258     
  Branches     1456     1210      -246     
===========================================
- Hits         6401     3502     -2899     
- Misses         60     2557     +2497     
- Partials       60      204      +144     
```

</details>
Hi, thanks for contributing to Strawberry 🍓!

We noticed that this PR is missing a `RELEASE.md` file. We use that to automatically do releases here on GitHub and, most importantly, to PyPI!

So as soon as this PR is merged, a release will be made 🚀.

Here's an example of `RELEASE.md`:

```markdown
Release type: patch

Description of the changes, ideally with some examples, if adding a new feature.
```

Release type can be one of patch, minor or major. We use [semver](https://semver.org/), so make sure to pick the appropriate type. If in doubt feel free to ask :)


Here's the tweet text: 
```
🆕 Release (next) is out! Thanks to Timothy Pansino for the PR 👏

Get it here 👉 https://github.com/strawberry-graphql/strawberry/releases/tag/(next)
```
<!-- action-check: release-file -->
@patrick91 

On my fork I was able to run and pass tests, but I suspect there's still a code coverage issue. Likely each successive run replaces the existing coverage file, but as I don't have codecov setup on my end I was unable to confirm. There are solutions to that but I didn't want to implement without being able to debug by looking at codecov.
Hi @TimPansino thanks for much for starting this!

I have two questions:

1. would you be interested in trying out nox? That's the tool I wanted to try personally
2. can we keep the tests in the same folders they are now? I have a PR that removes a lot of them and I think it will be a lot of work to rebase that PR :)
Hi @TimPansino I went on and created #2275, I really wanted to check nox, plus I think I might need to a new a matrix for testing sanic (seems like there have been some breaking changes there)
I might close this PR if I'm happy with nox. Hope that's ok 😊
If you'd rather use nox for creating venvs that's fine by me. I'm not personally a fan of nox, I like the more declarative style of tox and like to let it do the entire job of managing environments rather than mixing tools with things like poetry being run by nox. Not using a lock file based installer for testing is convenient for making sure you're testing with the most up to date versions of packages at all times to spot issues fast, and can easily pin older versions if you're blocked by a package update.

I would say you should definitely consider separating every major dependency you test against into a separate environment to ensure things are tested in isolation. To go with that, making every environment you want to matrix test into a separate folder in your tests folder lets you quickly and easily swap between testing with pytest/poetry by hand and using tox/nox for matrix testing. It's incredibly handy to have them logically separated.

If you'd prefer to use nox, might I suggest you swap the tox file for a nox file in this PR and keep the separated test folders? If you still have another PR that removes some tests that you'd like this rebased off of I can rebase it fairly easily after that's merged.
> If you'd rather use nox for creating venvs that's fine by me. I'm not personally a fan of nox, I like the more declarative style of tox and like to let it do the entire job of managing environments rather than mixing tools with things like poetry being run by nox. Not using a lock file based installer for testing is convenient for making sure you're testing with the most up to date versions of packages at all times to spot issues fast, and can easily pin older versions if you're blocked by a package update.

> I would say you should definitely consider separating every major dependency you test against into a separate environment to ensure things are tested in isolation. To go with that, making every environment you want to matrix test into a separate folder in your tests folder lets you quickly and easily swap between testing with pytest/poetry by hand and using tox/nox for matrix testing. It's incredibly handy to have them logically separated.

I'm not against this, but I have a PR that's going to refactor how the tests for the integration run, there will be only one test folder for all the integrations. I think it might be easier to isolate tests using markers for this specific case. What do you think?

> If you'd prefer to use nox, might I suggest you swap the tox file for a nox file in this PR and keep the separated test folders? If you still have another PR that removes some tests that you'd like this rebased off of I can rebase it fairly easily after that's merged.

I don't have a strong preference, I'm not familiar with either, I did like being able to use python for the configuration but other than that I don't know 😊 
Does tox allow to parallelise runs? At the moment the tests for django, starlette and other run in parallel since they are different actions, I'd like to keep that if possible


> Does tox allow to parallelise runs? At the moment the tests for django, starlette and other run in parallel since they are different actions, I'd like to keep that if possible

Yes, the `-p auto` flag sets it to run with as many threads in parallel as possible based on your CPU count. (https://tox.wiki/en/latest/example/basic.html#parallel-mode)

Would recommend setting `TOX_PARALLEL_NO_SPINNER=1` for CI as it has a spinner by default that will spam your logs.
I'm curious now if nox does parallelization, I didn't see anything in their docs. Do you happen to know?
> I'm curious now if nox does parallelization, I didn't see anything in their docs. Do you happen to know?

it does not :(



Resolving patch:
diff --git a/noxfile.py b/noxfile.py
new file mode 100644
--- /dev/null
+++ b/noxfile.py
@@ -0,0 +1,124 @@
+import nox
+from nox_poetry import Session, session
+
+PYTHON_VERSIONS = ["3.11", "3.10", "3.9", "3.8", "3.7"]
+
+
+@session(python=PYTHON_VERSIONS, name="Tests", tags=["tests"])
+def tests(session: Session) -> None:
+    session.run_always("poetry", "install", external=True)
+
+    session.run(
+        "pytest",
+        "--cov=strawberry",
+        "--cov-append",
+        "--cov-report=xml",
+        "-n",
+        "auto",
+        "--showlocals",
+        "-vv",
+        "-m",
+        "not starlette",
+        "-m",
+        "not django",
+        "-m",
+        "not starlite",
+        "--ignore=tests/mypy",
+        "--ignore=tests/pyright",
+    )
+
+
+@session(python=["3.11"], name="Django tests", tags=["tests"])
+@nox.parametrize("django", ["4.2.0", "4.1.0", "4.0.0", "3.2.0"])
+def tests_django(session: Session, django: str) -> None:
+    session.run_always("poetry", "install", external=True)
+
+    session._session.install(f"django~={django}")  # type: ignore
+
+    session.run(
+        "pytest",
+        "--cov=strawberry",
+        "--cov-append",
+        "--cov-report=xml",
+        "-n",
+        "auto",
+        "--showlocals",
+        "-vv",
+        "-m",
+        "django",
+    )
+
+
+@session(python=["3.11"], name="Starlette tests", tags=["tests"])
+@nox.parametrize("starlette", ["0.28.0", "0.27.0", "0.26.1"])
+def tests_starlette(session: Session, starlette: str) -> None:
+    session.run_always("poetry", "install", external=True)
+
+    session._session.install(f"starlette=={starlette}")  # type: ignore
+
+    session.run(
+        "pytest",
+        "--cov=strawberry",
+        "--cov-append",
+        "--cov-report=xml",
+        "-n",
+        "auto",
+        "--showlocals",
+        "-vv",
+        "-m",
+        "starlette",
+    )
+
+
+@session(python=["3.11"], name="Litestar tests", tags=["tests"])
+def tests_litestar(session: Session) -> None:
+    session.run_always("poetry", "install", external=True)
+
+    session.run(
+        "pytest",
+        "--cov=strawberry",
+        "--cov-append",
+        "--cov-report=xml",
+        "-n",
+        "auto",
+        "--showlocals",
+        "-vv",
+        "-m",
+        "starlite",
+    )
+
+
+@session(python=PYTHON_VERSIONS, name="Mypy tests")
+def tests_mypy(session: Session) -> None:
+    session.run_always("poetry", "install", external=True)
+
+    session.run(
+        "pytest",
+        "--cov=strawberry",
+        "--cov-append",
+        "--cov-report=xml",
+        "tests/mypy",
+        "-vv",
+    )
+
+
+@session(python=PYTHON_VERSIONS, name="Pyright tests", tags=["tests"])
+def tests_pyright(session: Session) -> None:
+    session.run_always("poetry", "install", external=True)
+    session.install("pyright")
+
+    session.run(
+        "pytest",
+        "--cov=strawberry",
+        "--cov-append",
+        "--cov-report=xml",
+        "tests/pyright",
+        "-vv",
+    )
+
+
+@session(name="Mypy", tags=["lint"])
+def mypy(session: Session) -> None:
+    session.run_always("poetry", "install", external=True)
+
+    session.run("mypy", "--config-file", "mypy.ini")


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -17,181 +17,130 @@ on:
       - ".github/workflows/test.yml"
 
 jobs:
-  unit-tests:
-    strategy:
-      matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
-
-    name: Python ${{ matrix.python-version }}
+  generate-jobs-tests:
+    name: 💻 Generate test matrix
     runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v3
-      - run: pipx install poetry
-      - uses: actions/setup-python@v4
-        id: setup-python
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "poetry"
-
-      - run: poetry env use ${{ matrix.python-version }}
-      - run: poetry install
-        if: steps.setup-python.outputs.cache-hit != 'true'
-
-      - name: pytest
-        run:
-          poetry run pytest --cov=strawberry --cov-append --cov-report=xml -n auto --showlocals -vv -m "not
-          starlette" -m "not django" -m "not starlite" --ignore=tests/mypy --ignore=tests/pyright
-
-        if: ${{ always() }}
-      - uses: codecov/codecov-action@v3
-        if: ${{ always() }}
-        with:
-          token: c0eec83c-17c8-4e8a-adf9-a34e4c22ef6a
-          fail_ci_if_error: true
-          verbose: true
-
-  unit-tests-on-windows:
-    name: Python 3.10.0 on Windows
-    runs-on: windows-latest
-
+    outputs:
+      sessions: ${{ steps.set-matrix.outputs.sessions }}
     steps:
       - uses: actions/checkout@v3
+      - uses: wntrblm/nox@main
       - run: pipx install poetry
-      - uses: actions/setup-python@v4
-        id: setup-python
-        with:
-          python-version: "3.10.0"
-          cache: "poetry"
-
-      - run: poetry install
-        if: steps.setup-python.outputs.cache-hit != 'true'
+      - run: pipx inject nox nox-poetry
+      - id: set-matrix
+        shell: bash
+        run: |
+          echo sessions=$(
+            nox --json -t tests -l |
+            jq 'map(
+              {
+                session,
+                name: "\( .name ) on \( .python )\( if .call_spec != {} then " (\(.call_spec | to_entries | map("\(.key)=\(.value)") | join(", ")))" else "" end )"
+              }
+            )'
+          ) | tee --append $GITHUB_OUTPUT
 
-      - name: pytest
-        run:
-          poetry run pytest --cov=strawberry --cov-append --cov-report=xml -n auto --showlocals -vv -m "not
-          django" --ignore=tests/mypy --ignore=tests/pyright
-
-        if: ${{ always() }}
-      - uses: codecov/codecov-action@v3
-        if: ${{ always() }}
-        with:
-          token: c0eec83c-17c8-4e8a-adf9-a34e4c22ef6a
-          fail_ci_if_error: true
-          verbose: true
-
-  django-unit-tests:
+  unit-tests:
+    name: 🔬 ${{ matrix.session.name }}
+    needs: [generate-jobs-tests]
+    runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        django: ["4.0", "3.2"]
-
-    name: Django ${{ matrix.django }}
-    runs-on: ubuntu-latest
+        session: ${{ fromJson(needs.generate-jobs-tests.outputs.sessions) }}
 
     steps:
       - uses: actions/checkout@v3
-      - run: pipx install poetry
-      - uses: actions/setup-python@v4
-        id: setup-python
+      - uses: wntrblm/nox@main
         with:
-          python-version: "3.10"
-          cache: "poetry"
-
-      - run: poetry env use python3.10
-      - run: poetry install
-        if: steps.setup-python.outputs.cache-hit != 'true'
+          python-versions: "3.7, 3.8, 3.9, 3.10, 3.11"
 
-      - name: Install django ${{ matrix.django }}
-        run: poetry add --python ^3.10 django@^${{ matrix.django }}
-
-      - name: pytest
-        run:
-          poetry run pytest --cov=strawberry --cov-append --cov-report=xml -n auto --showlocals -vv -m django
+      - name: Pip and nox cache
+        id: cache
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/pip
+            ~/.nox
+            .nox
+          key:
+            ${{ runner.os }}-nox-${{ matrix.session.session }}-${{
+            hashFiles('**/poetry.lock') }}-${{ hashFiles('**/noxfile.py') }}
+          restore-keys: |
+            ${{ runner.os }}-nox-${{ matrix.session.session }}-
+            ${{ runner.os }}-nox-
+
+      - run: pipx install coverage
+      - run: pipx install poetry
+      - run: pipx inject nox nox-poetry
+      - run: nox -r -t tests -s "${{ matrix.session.session }}"
 
+      - name: coverage xml
+        run: coverage xml -i
         if: ${{ always() }}
       - uses: codecov/codecov-action@v3
         if: ${{ always() }}
         with:
-          token: c0eec83c-17c8-4e8a-adf9-a34e4c22ef6a
+          token: ${{ secrets.CODECOV_TOKEN }}
           fail_ci_if_error: true
           verbose: true
 
-  starlette-unit-tests:
-    strategy:
-      matrix:
-        starlette: ["0.23.1", "0.24.0", "0.25.0", "0.26.1"]
-
-    name: Starlette ${{ matrix.starlette }}
+  lint:
+    name: ✨ Lint
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
 
     steps:
       - uses: actions/checkout@v3
-      - run: pipx install poetry
-      - uses: actions/setup-python@v4
-        id: setup-python
+      - uses: wntrblm/nox@main
         with:
-          python-version: "3.10"
-          cache: "poetry"
+          python-versions: "3.7, 3.8, 3.9, 3.10, 3.11"
 
-      - run: poetry env use python3.10
-      - run: poetry install
-        if: steps.setup-python.outputs.cache-hit != 'true'
-
-      - name: Install starlette ${{ matrix.starlette }}
-        run: poetry run pip install starlette==${{ matrix.starlette }}
-
-      - name: Install fastapi less than 0.92.0
-        run: poetry run pip install "fastapi<0.92.0"
-        if: ${{ matrix.starlette == '0.23.1' || matrix.starlette == '0.24.0' }}
-
-      - name: Install fastapi 0.92.0
-        run: poetry run pip install "fastapi==0.92.0"
-        if: ${{ matrix.starlette == '0.25.0' }}
-
-      - name: pytest
-        run:
-          poetry run pytest --cov=strawberry --cov-append --cov-report=xml -n auto --showlocals -vv -m
-          starlette
-
-        if: ${{ always() }}
-      - uses: codecov/codecov-action@v3
-        if: ${{ always() }}
+      - name: Pip and nox cache
+        id: cache
+        uses: actions/cache@v3
         with:
-          token: c0eec83c-17c8-4e8a-adf9-a34e4c22ef6a
-          fail_ci_if_error: true
-          verbose: true
+          path: |
+            ~/.cache/pip
+            ~/.nox
+            .nox
+          key:
+            ${{ runner.os }}-nox-lint-${{ matrix.session.session }}-${{
+            hashFiles('**/poetry.lock') }}-${{ hashFiles('**/noxfile.py') }}
+          restore-keys: |
+            ${{ runner.os }}-lint-nox-${{ matrix.session.session }}-
+            ${{ runner.os }}-lint-nox-
 
-  starlite-unit-tests:
-    strategy:
-      matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11"]
+      - run: pipx install poetry
+      - run: pipx inject nox nox-poetry
+      - run: nox -r -t lint
 
-    name: Python ${{ matrix.python-version }}
-    runs-on: ubuntu-latest
+  unit-tests-on-windows:
+    name: 🪟 Tests on Windows
+    runs-on: windows-latest
 
     steps:
       - uses: actions/checkout@v3
-      - run: pipx install poetry
-      - uses: actions/setup-python@v4
-        id: setup-python
+      - uses: wntrblm/nox@main
         with:
-          python-version: ${{ matrix.python-version }}
-          cache: "poetry"
-
-      - run: poetry env use ${{ matrix.python-version }}
-      - run: poetry install
-        if: steps.setup-python.outputs.cache-hit != 'true'
-
-      - name: pytest
-        run:
-          poetry run coverage run -m pytest --showlocals -vv -m starlite
+          python-versions: "3.7, 3.8, 3.9, 3.10, 3.11"
 
-      - name: coverage xml
-        run: poetry run coverage xml -i
-        if: ${{ always() }}
-      - uses: codecov/codecov-action@v3
-        if: ${{ always() }}
+      - name: Pip and nox cache
+        id: cache
+        uses: actions/cache@v3
         with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          fail_ci_if_error: true
-          verbose: true
+          path: |
+            ~/.cache/pip
+            ~/.nox
+            .nox
+          key:
+            ${{ runner.os }}-nox-windows-${{
+            hashFiles('**/poetry.lock') }}-${{ hashFiles('**/noxfile.py') }}
+          restore-keys: |
+            ${{ runner.os }}-nox-windows
+            ${{ runner.os }}-nox-
+
+      - run: pipx install poetry
+      - run: pipx inject nox nox-poetry
+      - run: nox -r -s "Tests" -p 3.11