Repository: pyvista/pytest-pyvista (language: python)

Issue:
Add downstream tests and guard against leakages across tests
closes #43 and closes #50

Teardown verify_image_cache after each test 
See #49 for some effects where it seems that `verify_image_cache` is leaking outside of test scope.

We should teardown, e.g. removing the object from the theme, to reduce cross talk, after each test.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci_cd.yml b/.github/workflows/ci_cd.yml
index 18471d4..d6d4e15 100644
--- a/.github/workflows/ci_cd.yml
+++ b/.github/workflows/ci_cd.yml
@@ -145,12 +145,12 @@ jobs:
 
   downstream:
     name: Downstream tests
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04  # matching pyvista
     steps:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.11'
 
       - run: git clone --depth=1 https://github.com/pyvista/pyvista.git --branch main --single-branch
   
@@ -170,6 +170,12 @@ jobs:
         working-directory: pyvista
   
       - name: Unit Testing
-        run: xvfb-run python -m pytest --fail_extra_image_cache -v tests/plotting/test_plotting.py
+        run: xvfb-run python -m pytest --fail_extra_image_cache -v --generated_image_dir gen_dir tests/plotting/test_plotting.py
         working-directory: pyvista
-    
+        
+      - name: Upload generated image artifact
+        if: ${{ failure() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: generated_images
+          path: pyvista/gen_dir


Test patch:
diff --git a/pytest_pyvista/pytest_pyvista.py b/pytest_pyvista/pytest_pyvista.py
index b495543..e88865b 100644
--- a/pytest_pyvista/pytest_pyvista.py
+++ b/pytest_pyvista/pytest_pyvista.py
@@ -189,6 +189,8 @@ def __call__(self, plotter):
             and self.fail_extra_image_cache
             and not self.reset_image_cache
         ):
+            # Make sure this doesn't get called again if this plotter doesn't close properly
+            plotter._before_close_callback = None
             raise RuntimeError(f"{image_filename} does not exist in image cache")
 
         if (
@@ -206,6 +208,8 @@ def __call__(self, plotter):
         error = pyvista.compare_images(image_filename, plotter)
 
         if error > allowed_error:
+            # Make sure this doesn't get called again if this plotter doesn't close properly
+            plotter._before_close_callback = None
             raise RuntimeError(
                 f"{test_name} Exceeded image regression error of "
                 f"{allowed_error} with an image error equal to: {error}"
@@ -241,4 +245,8 @@ def verify_image_cache(request, pytestconfig):
     )
     pyvista.global_theme.before_close_callback = verify_image_cache
 
+    def reset():
+        pyvista.global_theme.before_close_callback = None
+
+    request.addfinalizer(reset)
     return verify_image_cache