Repository: meltano/meltano

Issue:
Run Pytest tests in parallel
https://meltano.slack.com/archives/CKHP6G5V4/p1653508630941889

Looks like pytest doesn't take 45 minutes, not sure why I though it was taking that long before? Maybe on gitlab CI updated the title to reflect that it takes ~5-10 minutes to run pytest today. 

1. Be sure https://github.com/meltano/meltano/pull/6155/files#diff-65c008e1065903299dfa0da25e1f18d0f87f387867ccbbf2b0888eb138bdb255L242 is implemented (This uses a seperate folder for each Test in your temp directory instead of the same folder over and over again)
1. `poetry add -D pytest-xdist==1.*`
1. `poetry run pytest -n 20` 
1. Watch tests run extremely quickly

There's still a large number of failures here as some tests depend on the other I believe? Not exactly sure what causes the failures, but it's clear they weren't all designed with parallel testing in mind.  The gain is big enough it may be worth the time to figure out what causes the failures. 




Discussion:
@visch some tests seem to rely on running after (or not after!) other tests. See https://github.com/meltano/meltano/pull/6135#issue-1266509859. It's a bit crazy 😅 
> @visch some tests seem to rely on running after (or not after!) other tests. See [#6135 (comment)](https://github.com/meltano/meltano/pull/6135#issue-1266509859). It's a bit crazy 😅

Try implementing https://github.com/meltano/meltano/pull/6155/files#diff-65c008e1065903299dfa0da25e1f18d0f87f387867ccbbf2b0888eb138bdb255R241-R242 which separates out the directories for you (You don't need all the other stuff in that PR), I guess I could pull the PR down and give it a shot as well. 

Folders overlapping with eachother seems like the most likely case to me as the tests were written with the idea that there'd be one meltano project folder created everytime the `project_init_service` fixture was used (more often referenced by the `project` fixture)

Caught this on Windows as there's files that stay opened between runs in the folder that causes "Permission Denied" issues (You can't delete a file that has a handle still opened on Windows but you can on Linux) 
> > @visch some tests seem to rely on running after (or not after!) other tests. See [#6135 (comment)](https://github.com/meltano/meltano/pull/6135#issue-1266509859). It's a bit crazy 😅
> 
> Try implementing https://github.com/meltano/meltano/pull/6155/files#diff-65c008e1065903299dfa0da25e1f18d0f87f387867ccbbf2b0888eb138bdb255R241-R242 which separates out the directories for you (You don't need all the other stuff in that PR), I guess I could pull the PR down and give it a shot as well.
> 
> Folders overlapping with eachother seems like the most likely case to me as the tests were written with the idea that there'd be one meltano project folder created everytime the `project_init_service` fixture was used (more often referenced by the `project` fixture)
> 
> Caught this on Windows as there's files that stay opened between runs in the folder that causes "Permission Denied" issues (You can't delete a file that has a handle still opened on Windows but you can on Linux)

I tried this here https://github.com/visch/meltano/pull/2/files, waiting on results.

Nope it didn't help :/ will have to dive I guess, it is curious!  When doing this with Xdist I don't remember having that exact failure, but there are plenty of failures this way as well 🤷  
This should be possible once #6300 is complete. Because #6300 introduces test ordering at the module level via `pytest-order`, we'll need to use the `pytest-xdist` option `--dist=loadfile` as explained here: https://pytest-dev.github.io/pytest-order/stable/other_plugins.html#usage-with-pytest-xdist
We may want to upgrade our CI runners. They currently have 2 cores, but with more our CI pipelines would be far faster once we're running the Python tests in parallel. GitHub charges linearly per-core for their CI runners, so as an example, if we want to use runners with 16 cores, it would cost us 8 times as much to execute the Python tests per CI minute, but my guess is that the tests would probably execute ~6 times faster. Seems like a worthwhile tradeoff.

https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners
Good timing on the public beta for large GitHub runners: https://github.blog/changelog/2022-09-01-github-actions-larger-runners-are-now-in-public-beta/

I've added Meltano to the waitlist to be permitted to use them.

Seems like the larger runners also come with more memory, which we'll need for running many tests in parallel. Particularly for the MSSQL tests, which create a very heavy Java process for each Pytest worker. The memory conveniently scales at the same rate as the processor cores.

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -40,23 +40,20 @@ def tests(session: Session) -> None:
         session.install(".")
 
     session.install(
-        "coverage[toml]",
         "freezegun",
         "mock",
         "pytest",
         "pytest-asyncio",
+        "pytest-cov",
         "pytest-docker",
         "pytest-order",
         "pytest-randomly",
+        "pytest-xdist",
         "requests-mock",
     )
 
     try:
         session.run(
-            "coverage",
-            "run",
-            "--parallel",
-            "-m",
             "pytest",
             f"--randomly-seed={randint(0, 2**32-1)}",  # noqa: S311, WPS432
             *session.posargs,


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -132,8 +132,9 @@ jobs:
 
     - name: Run pytest
       env:
+        COLUMNS: 160
         PYTEST_BACKEND: ${{ matrix.backend-db }}
-
+        COVERAGE_FILE: .coverage.${{ matrix.id }}
         # Postgres backend
         POSTGRES_HOST_AUTH_METHOD: trust
         POSTGRES_ADDRESS: localhost
@@ -141,17 +142,14 @@ jobs:
         POSTGRES_USER: postgres
         POSTGRES_PASSWORD: postgres
         POSTGRES_DB: pytest_warehouse
-
         # MSSQL backend
         MSSQL_ADDRESS: localhost
         MSSQL_PORT: 1433
         MSSQL_USER: sa
         MSSQL_PASSWORD: Meltan0admin
         MSSQL_DB: pytest_warehouse
-
       shell: bash
       run: |
-        export COLUMNS=160
         nox -rs tests --python=${{ matrix.python-version }} -- -m "${{ env.PYTEST_MARKERS }}" 2>&1 | \
           tee >( \
             tail | sed -r "s/[[:cntrl:]]\[([0-9]{1,3};)*[0-9]{1,3}m//g" | grep -E '^=+ [0-9].+ =+$' | \
@@ -177,11 +175,11 @@ jobs:
         $( pattern='in ([0-9]+)\.[0-9]+s' && [[ ${{ env.pytest_results }} =~ $pattern ]] && echo "${BASH_REMATCH[1]}" )s |"
 
     - name: Upload coverage data
-      if: always() && (matrix.python-version == '3.9')
+      if: always()
       uses: actions/upload-artifact@v3.1.0
       with:
         name: coverage-data
-        path: ".coverage.*"
+        path: ".coverage.${{ matrix.id }}"
 
   summary:
     runs-on: ubuntu-latest
diff --git a/tests/fixtures/db/__init__.py b/tests/fixtures/db/__init__.py
--- a/tests/fixtures/db/__init__.py
+++ b/tests/fixtures/db/__init__.py
@@ -3,7 +3,6 @@
 import logging
 import warnings
 from contextlib import closing
-from time import sleep
 from typing import Generator
 
 import pytest
diff --git a/tests/fixtures/db/mssql.py b/tests/fixtures/db/mssql.py
--- a/tests/fixtures/db/mssql.py
+++ b/tests/fixtures/db/mssql.py
@@ -64,12 +64,12 @@ def create_connection_url(
 
 
 @pytest.fixture(scope="session")
-def engine_uri():
+def engine_uri(worker_id: str):
     host = os.getenv("MSSQL_ADDRESS")
     port = os.getenv("MSSQL_PORT", 1433)
     user = os.getenv("MSSQL_USER")
     password = os.getenv("MSSQL_PASSWORD")
-    database = os.getenv("MSSQL_DB", "pytest_meltano")
+    database = f"{os.getenv('MSSQL_DB', 'pytest_meltano')}_{worker_id}"
 
     # Recreate the database using the master database
     master_engine_uri = create_connection_url(host, port, user, password, "master")
diff --git a/tests/fixtures/db/postgresql.py b/tests/fixtures/db/postgresql.py
--- a/tests/fixtures/db/postgresql.py
+++ b/tests/fixtures/db/postgresql.py
@@ -19,12 +19,12 @@ def recreate_database(engine, db_name):
 
 
 @pytest.fixture(scope="session")
-def engine_uri():
+def engine_uri(worker_id: str):
     host = os.getenv("POSTGRES_ADDRESS")
     port = os.getenv("POSTGRES_PORT", 5432)
     user = os.getenv("POSTGRES_USER")
     password = os.getenv("POSTGRES_PASSWORD")
-    database = os.getenv("POSTGRES_DB", "pytest_meltano")
+    database = f"{os.getenv('POSTGRES_DB', 'pytest_meltano')}_{worker_id}"
 
     # create the database
     engine_uri = f"postgresql://{user}:{password}@{host}:{port}/postgres"
diff --git a/tests/fixtures/db/sqlite.py b/tests/fixtures/db/sqlite.py
--- a/tests/fixtures/db/sqlite.py
+++ b/tests/fixtures/db/sqlite.py
@@ -4,8 +4,9 @@
 
 
 @pytest.fixture(scope="session")
-def engine_uri(tmp_path_factory) -> str:
+def engine_uri(tmp_path_factory, worker_id: str) -> str:
     database_path = (
-        tmp_path_factory.mktemp("fixture_db_sqlite_engine_uri") / "pytest_meltano.db"
+        tmp_path_factory.mktemp("fixture_db_sqlite_engine_uri")
+        / f"pytest_meltano_{worker_id}.db"
     )
     return f"sqlite:///{database_path.as_posix()}"
diff --git a/tests/meltano/core/tracking/test_tracker.py b/tests/meltano/core/tracking/test_tracker.py
--- a/tests/meltano/core/tracking/test_tracker.py
+++ b/tests/meltano/core/tracking/test_tracker.py
@@ -173,7 +173,7 @@ def test_analytics_json_is_created(self, project: Project):
             f'["{str(uuid.uuid4())}","{str(uuid.uuid4())}", true]',
             f'client_id":"{str(uuid.uuid4())}","project_id":"{str(uuid.uuid4())}","send_anonymous_usage_stats":true}}',  # noqa: E501
         ],
-        ids=lambda param: hash_sha256(param)[:8],
+        ids=(0, 1, 2, 3, 4),
     )
     def test_invalid_analytics_json_is_overwritten(
         self, project: Project, analytics_json_content: str