Repository: holoviz/holoviews

Issue:
Test suite are sensitive to run order
When trying to apply parallelization with `pytest-xdist` to the test suite (in `holoviews/pytest`), I noticed that it is very dependent on the run order.  

If the order is randomized with something like `pytest-random-order` and the following command: `pytest holoviews/tests --random-order-bucket=class`, the test suite will fail. 

Looking at the code a culprit for this could be having `Store.renderers['bokeh']` out in the open and not inside the `setUp` of the class.

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz) Report
Attention: Patch coverage is `0%` with `18 lines` in your changes are missing coverage. Please review.
> Project coverage is 16.85%. Comparing base [(`6f042b3`)](https://app.codecov.io/gh/holoviz/holoviews/commit/6f042b33c236c680c7fffe99ac1182dc6fc22020?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz) to head [(`b68db8c`)](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz).

| [Files](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz) | Patch % | Lines |
|---|---|---|
| [holoviews/tests/ipython/test\_magics.py](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?src=pr&el=tree&filepath=holoviews%2Ftests%2Fipython%2Ftest_magics.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz#diff-aG9sb3ZpZXdzL3Rlc3RzL2lweXRob24vdGVzdF9tYWdpY3MucHk=) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz) |
| [holoviews/tests/plotting/plotly/test\_renderer.py](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?src=pr&el=tree&filepath=holoviews%2Ftests%2Fplotting%2Fplotly%2Ftest_renderer.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz#diff-aG9sb3ZpZXdzL3Rlc3RzL3Bsb3R0aW5nL3Bsb3RseS90ZXN0X3JlbmRlcmVyLnB5) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz) |
| [...loviews/tests/plotting/matplotlib/test\_renderer.py](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?src=pr&el=tree&filepath=holoviews%2Ftests%2Fplotting%2Fmatplotlib%2Ftest_renderer.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz#diff-aG9sb3ZpZXdzL3Rlc3RzL3Bsb3R0aW5nL21hdHBsb3RsaWIvdGVzdF9yZW5kZXJlci5weQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##             main    #6234       +/-   ##
===========================================
- Coverage   88.19%   16.85%   -71.34%     
===========================================
  Files         321      321               
  Lines       67344    67333       -11     
===========================================
- Hits        59391    11347    -48044     
- Misses       7953    55986    +48033     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/holoviz/holoviews/pull/6234?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=holoviz).


Patch:
diff --git a/.github/workflows/nightly_lock.yaml b/.github/workflows/nightly_lock.yaml
index c6fda817a..9b93b85a2 100644
--- a/.github/workflows/nightly_lock.yaml
+++ b/.github/workflows/nightly_lock.yaml
@@ -4,6 +4,9 @@ on:
   schedule:
     - cron: "0 0 * * *"
 
+env:
+  PACKAGE: "holoviews"
+
 jobs:
   pixi_lock:
     name: Pixi lock
@@ -12,11 +15,11 @@ jobs:
     steps:
       - uses: holoviz-dev/holoviz_tasks/pixi_lock@pixi
       - name: Upload lock-file to S3
+        if: "!github.event.pull_request.head.repo.fork"
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           AWS_DEFAULT_REGION: "eu-west-1"
-          PACKAGE: "holoviews"
         run: |
           zip $(date +%Y-%m-%d).zip pixi.lock pixi.toml
           aws s3 cp ./$(date +%Y-%m-%d).zip s3://assets.holoviz.org/lock/$PACKAGE/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 31b4e80a1..9bb53aa00 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -20,7 +20,7 @@ repos:
       - id: check-json
       - id: detect-private-key
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.4.4
+    rev: v0.4.5
     hooks:
       - id: ruff
         files: holoviews/|scripts/
@@ -30,7 +30,7 @@ repos:
       - id: clean-notebook
         args: [--strip-trailing-newlines]
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.6
+    rev: v2.3.0
     hooks:
       - id: codespell
         additional_dependencies:
diff --git a/examples/README.md b/examples/README.md
index c8a747e9d..7e958bc6b 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -9,7 +9,6 @@ This directory contains all the notebooks built as part of the
 - `gallery`: Examples shown on the [gallery page](https://holoviews.org/gallery/index.html).
 - `getting_started`: Notebooks used in the [getting started](https://holoviews.org/getting_started/index.html) guide.
 - `reference`: Notebooks shown in the website [reference gallery](https://holoviews.org/reference/index.html)
-- `topics`: Notebooks shown in the [showcase](https://holoviews.org/reference/showcase/index.html)
 - `user_guide`: Notebooks used in the [user guide](https://holoviews.org/user_guide/index.html).
 
 ## Contributing to examples
diff --git a/examples/reference/elements/bokeh/HexTiles.ipynb b/examples/reference/elements/bokeh/HexTiles.ipynb
index 749db1569..de78a1f1a 100644
--- a/examples/reference/elements/bokeh/HexTiles.ipynb
+++ b/examples/reference/elements/bokeh/HexTiles.ipynb
@@ -28,7 +28,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "``HexTiles`` provide a representation of the data as a bivariate histogram useful for visualizing the structure in larger datasets. The ``HexTiles`` are computed by tesselating the x-, y-ranges of the data, storing the number of points falling in each hexagonal bin. By default ``HexTiles`` simply counts the data in each bin but it also supports weighted aggregations. Currently only linearly spaced bins are supported when using the bokeh backend.\n",
+    "``HexTiles`` provide a representation of the data as a bivariate histogram useful for visualizing the structure in larger datasets. The ``HexTiles`` are computed by tessellating the x-, y-ranges of the data, storing the number of points falling in each hexagonal bin. By default ``HexTiles`` simply counts the data in each bin but it also supports weighted aggregations. Currently only linearly spaced bins are supported when using the bokeh backend.\n",
     "\n",
     "As a simple example we will generate 100,000 normally distributed points and plot them using ``HexTiles``:"
    ]
diff --git a/examples/reference/elements/matplotlib/HexTiles.ipynb b/examples/reference/elements/matplotlib/HexTiles.ipynb
index 98a82a92d..49117ee72 100644
--- a/examples/reference/elements/matplotlib/HexTiles.ipynb
+++ b/examples/reference/elements/matplotlib/HexTiles.ipynb
@@ -29,7 +29,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "``HexTiles`` provide a representation of the data as a bivariate histogram useful for visualizing the structure in larger datasets. The ``HexTiles`` are computed by tesselating the x-, y-ranges of the data, storing the number of points falling in each hexagonal bin. By default ``HexTiles`` simply counts the data in each bin but it also supports weighted aggregations. Currently only linearly spaced bins are supported when using the bokeh backend.\n",
+    "``HexTiles`` provide a representation of the data as a bivariate histogram useful for visualizing the structure in larger datasets. The ``HexTiles`` are computed by tessellating the x-, y-ranges of the data, storing the number of points falling in each hexagonal bin. By default ``HexTiles`` simply counts the data in each bin but it also supports weighted aggregations. Currently only linearly spaced bins are supported when using the bokeh backend.\n",
     "\n",
     "As a simple example we will generate 100,000 normally distributed points and plot them using ``HexTiles``:"
    ]
diff --git a/examples/user_guide/Deploying_Bokeh_Apps.ipynb b/examples/user_guide/Deploying_Bokeh_Apps.ipynb
index 139df4592..6b8bdfd6c 100644
--- a/examples/user_guide/Deploying_Bokeh_Apps.ipynb
+++ b/examples/user_guide/Deploying_Bokeh_Apps.ipynb
@@ -789,7 +789,7 @@
    "metadata": {},
    "source": [
     "If instead we want to deploy this we could add `.servable` as discussed before or use `pn.serve`. Note however that when using `pn.serve` all sessions will share the same state therefore it is best to \n",
-    "wrap the creation of the app in a function which we can then provide to `pn.serve`. For more detail on deploying Panel applications also see the [Panel server deployment guide](https://panel.holoviz.org/user_guide/Server_Deployment.html).\n",
+    "wrap the creation of the app in a function which we can then provide to `pn.serve`. For more detail on deploying Panel applications also see the [Panel server deployment guide](https://panel.holoviz.org/how_to/server/index.html).\n",
     "\n",
     "Now we can reimplement the same example using Bokeh allowing us to compare and contrast the approaches:"
    ]
diff --git a/holoviews/core/sheetcoords.py b/holoviews/core/sheetcoords.py
index a6e8d163c..690a55d70 100644
--- a/holoviews/core/sheetcoords.py
+++ b/holoviews/core/sheetcoords.py
@@ -365,7 +365,7 @@ class Slice(np.ndarray):
         else:
             slicespec=Slice._boundsspec2slicespec(bounds.lbrt(),sheet_coordinate_system)
         # Using numpy.int32 for legacy reasons
-        a = np.array(slicespec, dtype=np.int32, copy=False).view(cls)
+        a = np.asarray(slicespec, dtype=np.int32).view(cls)
         return a
 
 
diff --git a/holoviews/core/util.py b/holoviews/core/util.py
index c2ff670d5..2e2732bfb 100644
--- a/holoviews/core/util.py
+++ b/holoviews/core/util.py
@@ -50,6 +50,9 @@ _NP_SAMPLE_SIZE = 1_000_000
 _PANDAS_ROWS_LARGE = 1_000_000
 _PANDAS_SAMPLE_SIZE = 1_000_000
 
+numpy_version = Version(Version(np.__version__).base_version)
+NUMPY_GE_200 = numpy_version >= Version("2")
+
 pandas_version = Version(pd.__version__)
 try:
     if pandas_version >= Version('1.3.0'):
diff --git a/pixi.toml b/pixi.toml
index 7e6a08d23..5a37dd2e0 100644
--- a/pixi.toml
+++ b/pixi.toml
@@ -15,6 +15,7 @@ test-311 = ["py311", "test-core", "test", "example", "test-example", "test-unit-
 test-312 = ["py312", "test-core", "test", "example", "test-example", "test-unit-task"]
 test-ui = ["py312", "test-core", "test", "test-ui"]
 test-core = ["py312", "test-core", "test-unit-task"]
+test-numpy = ["py312", "test-core", "test-unit-task", "numpy2", "test-example"]
 test-gpu = ["py311", "test-core", "test", "test-gpu"]
 docs = ["py311", "example", "doc"]
 build = ["py311", "build"]
@@ -69,13 +70,15 @@ xarray = ">=0.10.4"
 # =================== TESTS ===================
 # =============================================
 [feature.test-core.dependencies]
+psutil = "*"
 pytest = "*"
 pytest-cov = "*"
 pytest-github-actions-annotate-failures = "*"
 pytest-rerunfailures = "*"
+pytest-xdist = "*"
 
 [feature.test-unit-task.tasks] # So it is not showing up in the test-ui environment
-test-unit = 'pytest holoviews/tests'
+test-unit = 'pytest holoviews/tests -n logical --dist loadgroup'
 
 [feature.test.dependencies]
 cftime = "*"
@@ -102,9 +105,7 @@ tsdownsample = "*" # currently not available on Windows
 test-example = 'pytest -n logical --dist loadscope --nbval-lax examples'
 
 [feature.test-example.dependencies]
-psutil = "*"
 nbval = "*"
-pytest-xdist = "*"
 
 [feature.test-ui]
 channels = ["pyviz/label/dev", "microsoft", "conda-forge"]
@@ -134,6 +135,39 @@ rmm = { version = "*", channel = "rapidsai" }
 [feature.test-gpu.tasks]
 test-gpu = { cmd = "pytest holoviews/tests --gpu", env = { NUMBA_CUDA_LOW_OCCUPANCY_WARNINGS = '0' } }
 
+[feature.numpy2]
+channels = ["pyviz/label/dev", "conda-forge/label/numpy_rc", "numba/label/dev", "conda-forge"]
+
+[feature.numpy2.dependencies]
+numpy = "2.*"
+numba = { version = "0.60.*", channel = "numba/label/dev" }
+llvmlite = { version = "*", channel = "numba/label/dev" }
+
+# test dependencies
+cftime = "*"
+contourpy = "*"
+dask-core = "*"
+datashader = ">=0.11.1"
+ffmpeg = "*"
+ibis-sqlite = "*"
+nbconvert = "*"
+networkx = "*"
+pillow = "*"
+scipy = ">=1.10" # Python 3.9 + Windows downloads 1.9
+selenium = "*"
+shapely = "*"
+# spatialpandas = "*"
+xarray = ">=0.10.4"
+xyzservices = "*"
+
+# Examples (removed duplicates)
+# netcdf4 = "*"
+notebook = "*"
+pooch = "*"
+# pyarrow = "*"
+# scikit-image = "*"
+streamz = ">=0.5.0"
+
 # =============================================
 # =================== DOCS ====================
 # =============================================
diff --git a/pyproject.toml b/pyproject.toml
index ed94c5e2d..a1d48ea35 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -90,7 +90,7 @@ filterwarnings = [
     # 2023-01: Sqlalchemy 2.0 warning:
     "ignore: Deprecated API features detected:DeprecationWarning:ibis.backends.base.sql.alchemy", # https://github.com/ibis-project/ibis/issues/5048
     # 2023-03: Already handling the nested sequence
-    "ignore:Creating an ndarray from ragged nested sequences:numpy.VisibleDeprecationWarning:holoviews.core.data.spatialpandas",
+    "ignore:Creating an ndarray from ragged nested sequences::holoviews.core.data.spatialpandas",
     # 2023-09: Dash needs to update their code to use the comm module and pkg_resources
     "ignore:The `.+?` class has been deprecated:DeprecationWarning:dash._jupyter",
     "ignore:pkg_resources is deprecated as an API:DeprecationWarning:dash.dash",
@@ -180,6 +180,6 @@ known-first-party = ["holoviews"]
 combine-as-imports = true
 
 [tool.codespell]
-ignore-words-list = "lod,nd,ndoes,reenabled,spreaded,whn,ser"
+ignore-words-list = "lod,nd,ndoes,reenabled,spreaded,whn,ser,assertIn"
 skip = "doc/generate_modules.py"
 write-changes = true


Test patch:
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
index b16390f64..1a8340ad1 100644
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -85,7 +85,10 @@ jobs:
         run: |
           MATRIX=$(jq -nsc '{
               "os": ["ubuntu-latest", "macos-latest", "windows-latest"],
-              "environment": ["test-39", "test-312"]
+              "environment": ["test-39", "test-312"],
+              "include": [
+                { "os": "ubuntu-latest", "environment": "test-numpy" }
+              ]
           }')
           echo "MATRIX=$MATRIX" >> $GITHUB_ENV
       - name: Set test matrix with 'full' option
@@ -93,7 +96,10 @@ jobs:
         run: |
           MATRIX=$(jq -nsc '{
               "os": ["ubuntu-latest", "macos-latest", "windows-latest"],
-              "environment": ["test-39", "test-310", "test-311", "test-312"]
+              "environment": ["test-39", "test-310", "test-311", "test-312"],
+              "include": [
+                { "os": "ubuntu-latest", "environment": "test-numpy" }
+              ]
           }')
           echo "MATRIX=$MATRIX" >> $GITHUB_ENV
       - name: Set test matrix with 'downstream' option
diff --git a/examples/conftest.py b/examples/conftest.py
index 22dced713..a6ba9f708 100644
--- a/examples/conftest.py
+++ b/examples/conftest.py
@@ -1,6 +1,7 @@
 import os
 import platform
 import sys
+from importlib.util import find_spec
 
 import bokeh
 import pandas as pd
@@ -62,6 +63,22 @@ if system == "Windows" and os.environ.get("GITHUB_RUN_ID"):
         "user_guide/Plotting_with_Matplotlib.ipynb",
     ]
 
+# 2024-05: Numpy 2.0
+if find_spec("datashader") is None:
+    collect_ignore_glob += [
+        "reference/elements/matplotlib/ImageStack.ipynb",
+        "reference/elements/plotly/ImageStack.ipynb",
+        "user_guide/15-Large_Data.ipynb",
+        "user_guide/16-Streaming_Data.ipynb",
+        "user_guide/Linked_Brushing.ipynb",
+        "user_guide/Network_Graphs.ipynb",
+    ]
+
+if find_spec("scikit-image") is None:
+    collect_ignore_glob += [
+        "user_guide/Network_Graphs.ipynb",
+    ]
+
 
 def pytest_runtest_makereport(item, call):
     """
diff --git a/holoviews/tests/conftest.py b/holoviews/tests/conftest.py
index c0b202f58..abc573291 100644
--- a/holoviews/tests/conftest.py
+++ b/holoviews/tests/conftest.py
@@ -79,10 +79,10 @@ def _plotting_backend(backend):
     if not hv.extension._loaded:
         hv.extension(backend)
     hv.renderer(backend)
-    prev_backend = hv.Store.current_backend
-    hv.Store.current_backend = backend
+    curent_backend = hv.Store.current_backend
+    hv.Store.set_current_backend(backend)
     yield
-    hv.Store.current_backend = prev_backend
+    hv.Store.set_current_backend(curent_backend)
 
 
 @pytest.fixture
@@ -125,3 +125,17 @@ def serve_hv(page, port):  # noqa: F811
         return page
 
     return serve_and_return_page
+
+
+@pytest.fixture(autouse=True)
+def reset_store():
+    _custom_options = {k: {} for k in hv.Store._custom_options}
+    _options = hv.Store._options.copy()
+    current_backend = hv.Store.current_backend
+    renderers = hv.Store.renderers.copy()
+    yield
+    hv.Store._custom_options = _custom_options
+    hv.Store._options = _options
+    hv.Store._weakrefs = {}
+    hv.Store.renderers = renderers
+    hv.Store.set_current_backend(current_backend)
diff --git a/holoviews/tests/core/data/test_imageinterface.py b/holoviews/tests/core/data/test_imageinterface.py
index 71630c05f..6d9b10048 100644
--- a/holoviews/tests/core/data/test_imageinterface.py
+++ b/holoviews/tests/core/data/test_imageinterface.py
@@ -2,6 +2,7 @@ import datetime as dt
 from unittest import SkipTest
 
 import numpy as np
+import pytest
 
 from holoviews import HSV, RGB, Curve, Dataset, Dimension, Image, Table
 from holoviews.core.data.interface import DataError
@@ -465,17 +466,9 @@ class RGBElement_ImageInterfaceTests(BaseRGBElementInterfaceTests):
 
     __test__ = True
 
+    @pytest.mark.xfail(reason="Raises DataError in ImageInterface")
     def test_reduce_to_single_values(self):
-        try:
-            super().test_reduce_to_single_values()
-        except DataError:
-            msg = (
-                "RGB element can't run with this command: "
-                "'pytest holoviews/tests -k test_reduce_to_single_values'"
-                "but runs fine with 'pytest holoviews/tests/core/'"
-            )
-            raise SkipTest(msg)
-
+        super().test_reduce_to_single_values()
 
 class BaseHSVElementInterfaceTests(InterfaceTests):
 
diff --git a/holoviews/tests/core/test_dimensioned.py b/holoviews/tests/core/test_dimensioned.py
index fe4179c1c..f65060dea 100644
--- a/holoviews/tests/core/test_dimensioned.py
+++ b/holoviews/tests/core/test_dimensioned.py
@@ -29,17 +29,6 @@ class CustomBackendTestCase(LoggingComparisonTestCase):
         self.register_custom(ExampleElement, 'backend_2', ['plot_custom2'])
         Store.set_current_backend('backend_1')
 
-    def tearDown(self):
-        super().tearDown()
-        Store._weakrefs = {}
-        Store._options.pop('backend_1')
-        Store._options.pop('backend_2')
-        Store._custom_options.pop('backend_1')
-        Store._custom_options.pop('backend_2')
-        Store.set_current_backend(self.current_backend)
-        Store.renderers.pop('backend_1')
-        Store.renderers.pop('backend_2')
-
     @classmethod
     def register_custom(cls, objtype, backend, custom_plot=None, custom_style=None):
         if custom_style is None:
diff --git a/holoviews/tests/core/test_dimensions.py b/holoviews/tests/core/test_dimensions.py
index 06f77066b..7b79e1bce 100644
--- a/holoviews/tests/core/test_dimensions.py
+++ b/holoviews/tests/core/test_dimensions.py
@@ -5,6 +5,7 @@ import numpy as np
 import pandas as pd
 
 from holoviews.core import Dimension, Dimensioned
+from holoviews.core.util import NUMPY_GE_200
 from holoviews.element.comparison import ComparisonTestCase
 
 from ..utils import LoggingComparisonTestCase
@@ -243,7 +244,10 @@ class DimensionCloneTest(ComparisonTestCase):
 class DimensionDefaultTest(ComparisonTestCase):
 
     def test_validate_default_against_values(self):
-        msg = r"Dimension\('A'\) default 1\.1 not found in declared values: \[0, 1\]"
+        if NUMPY_GE_200:
+            msg = r"Dimension\('A'\) default 1\.1 not found in declared values: \[np\.int64\(0\), np\.int64\(1\)\]"
+        else:
+            msg = r"Dimension\('A'\) default 1\.1 not found in declared values: \[0, 1\]"
         with self.assertRaisesRegex(ValueError, msg):
             Dimension('A', values=[0, 1], default=1.1)
 
diff --git a/holoviews/tests/core/test_options.py b/holoviews/tests/core/test_options.py
index 3ba86ad92..e9aa318ff 100644
--- a/holoviews/tests/core/test_options.py
+++ b/holoviews/tests/core/test_options.py
@@ -1,5 +1,4 @@
 import contextlib
-import os
 import pickle
 from unittest import SkipTest
 
@@ -301,7 +300,6 @@ class TestStoreInheritanceDynamic(ComparisonTestCase):
 
     def tearDown(self):
         Store.options(val=self.store_copy)
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         super().tearDown()
 
     def initialize_option_tree(self):
@@ -504,7 +502,6 @@ class TestStoreInheritance(ComparisonTestCase):
 
     def tearDown(self):
         Store.options(val=self.store_copy)
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         super().tearDown()
 
     def lookup_options(self, obj, group):
@@ -574,7 +571,6 @@ class TestOptionsMethod(ComparisonTestCase):
 
     def tearDown(self):
         Store.options(val=self.store_copy)
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         super().tearDown()
 
     def lookup_options(self, obj, group):
@@ -627,7 +623,6 @@ class TestOptsMethod(ComparisonTestCase):
 
     def tearDown(self):
         Store.options(val=self.store_copy)
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         super().tearDown()
 
     def lookup_options(self, obj, group):
@@ -733,7 +728,6 @@ class TestOptionTreeFind(ComparisonTestCase):
     def tearDown(self):
         Options._option_groups = self.original_option_groups
         Store.options(val=self.original_options)
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
 
     def test_optiontree_find1(self):
         self.assertEqual(self.options.find('MyType').options('group').options,
@@ -814,7 +808,6 @@ class TestCrossBackendOptions(ComparisonTestCase):
         Store.options(val=self.store_mpl, backend='matplotlib')
         Store.options(val=self.store_bokeh, backend='bokeh')
         Store.current_backend = 'matplotlib'
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
 
         if self.plotly_options is not None:
             Store._options['plotly'] = self.plotly_options
@@ -965,6 +958,7 @@ class TestLookupOptions(ComparisonTestCase):
             self.assertNotIn("muted_alpha", options_matplotlib.keys())
 
 
+@pytest.mark.usefixtures("bokeh_backend")
 class TestCrossBackendOptionSpecification(ComparisonTestCase):
     """
     Test the style system can style a single object across backends.
@@ -989,7 +983,6 @@ class TestCrossBackendOptionSpecification(ComparisonTestCase):
         Store.options(val=self.store_mpl, backend='matplotlib')
         Store.options(val=self.store_bokeh, backend='bokeh')
         Store.current_backend = 'matplotlib'
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
 
         if self.plotly_options is not None:
             Store._options['plotly'] = self.plotly_options
@@ -1121,8 +1114,6 @@ class TestCrossBackendOptionSpecification(ComparisonTestCase):
 
 class TestCrossBackendOptionPickling(TestCrossBackendOptions):
 
-    cleanup = ['test_raw_pickle.pkl', 'test_pickle_mpl_bokeh.pkl']
-
     def setUp(self):
         super().setUp()
         self.raw = Image(np.random.rand(10,10))
@@ -1131,19 +1122,24 @@ class TestCrossBackendOptionPickling(TestCrossBackendOptions):
         Store.current_backend = 'bokeh'
         StoreOptions.set_options(self.raw, style={'Image':{'cmap':'Purple'}})
 
-    def tearDown(self):
-        super().tearDown()
-        for f in self.cleanup:
-            try:
-                os.remove(f)
-            except Exception:
-                pass
+    def _create_tmp_path(self):
+        # Remove this when we use pytest
+        import tempfile
+        from pathlib import Path
+
+        tmp_dir = tempfile.TemporaryDirectory()
+        tmp_path = Path(tmp_dir.name)
+
+        return tmp_dir, tmp_path
+
 
     def test_raw_pickle(self):
         """
         Test usual pickle saving and loading (no style information preserved)
         """
-        fname= 'test_raw_pickle.pkl'
+        tmp_dir, tmp_path = self._create_tmp_path()  # Pytest remove
+
+        fname= tmp_path / 'test_raw_pickle.pkl'
         with open(fname,'wb') as handle:
             pickle.dump(self.raw, handle)
         self.clear_options()
@@ -1160,11 +1156,15 @@ class TestCrossBackendOptionPickling(TestCrossBackendOptions):
         bokeh_opts = Store.lookup_options('bokeh', img, 'style').options
         self.assertEqual(bokeh_opts, {})
 
+        tmp_dir.cleanup()  # Pytest remove
+
     def test_pickle_mpl_bokeh(self):
         """
         Test pickle saving and loading with Store (style information preserved)
         """
-        fname = 'test_pickle_mpl_bokeh.pkl'
+        tmp_dir, tmp_path = self._create_tmp_path()  # Pytest remove
+
+        fname = tmp_path / 'test_pickle_mpl_bokeh.pkl'
         with open(fname,'wb') as handle:
             Store.dump(self.raw, handle)
         self.clear_options()
@@ -1180,3 +1180,5 @@ class TestCrossBackendOptionPickling(TestCrossBackendOptions):
         Store.current_backend = 'bokeh'
         bokeh_opts = Store.lookup_options('bokeh', img, 'style').options
         self.assertEqual(bokeh_opts, {'cmap':'Purple'})
+
+        tmp_dir.cleanup()  # Pytest remove
diff --git a/holoviews/tests/element/test_comparisondimension.py b/holoviews/tests/element/test_comparisondimension.py
index 1234ce27c..3bdfb04c4 100644
--- a/holoviews/tests/element/test_comparisondimension.py
+++ b/holoviews/tests/element/test_comparisondimension.py
@@ -2,6 +2,7 @@
 Test cases for Dimension and Dimensioned object comparison.
 """
 from holoviews.core import Dimension, Dimensioned
+from holoviews.core.util import NUMPY_GE_200
 from holoviews.element.comparison import ComparisonTestCase
 
 
@@ -74,7 +75,11 @@ class DimensionsComparisonTestCase(ComparisonTestCase):
         try:
             self.assertEqual(self.dimension4, self.dimension8)
         except AssertionError as e:
-            self.assertEqual(str(e), "Dimension parameter 'values' mismatched: [] != ['a', 'b']")
+            if NUMPY_GE_200:
+                msg = "Dimension parameter 'values' mismatched: [] != [np.str_('a'), np.str_('b')]"
+            else:
+                msg = "Dimension parameter 'values' mismatched: [] != ['a', 'b']"
+            self.assertEqual(str(e), msg)
 
     def test_dimension_comparison_types_unequal(self):
         try:
diff --git a/holoviews/tests/ipython/test_displayhooks.py b/holoviews/tests/ipython/test_displayhooks.py
index cf1a7ba3e..d8b20b5c2 100644
--- a/holoviews/tests/ipython/test_displayhooks.py
+++ b/holoviews/tests/ipython/test_displayhooks.py
@@ -16,7 +16,6 @@ class TestDisplayHooks(IPTestCase):
         Store.display_formats = self.format
 
     def tearDown(self):
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         self.ip.run_line_magic("unload_ext", "holoviews.ipython")
         del self.ip
         Store.display_hooks = self.backup
diff --git a/holoviews/tests/ipython/test_magics.py b/holoviews/tests/ipython/test_magics.py
index ec7055616..cdda43f09 100644
--- a/holoviews/tests/ipython/test_magics.py
+++ b/holoviews/tests/ipython/test_magics.py
@@ -1,17 +1,14 @@
-from unittest import SkipTest
+import pytest
+from pyviz_comms import CommManager
 
 import holoviews as hv
 from holoviews.core.options import Store
+from holoviews.operation import Compositor
 
 try:
-    from holoviews import ipython            # noqa (Import test)
     from holoviews.ipython import IPTestCase
 except ImportError:
-    raise SkipTest("Required dependencies not satisfied for testing magics")
-
-from pyviz_comms import CommManager
-
-from holoviews.operation import Compositor
+    pytest.skip("IPython required to test IPython magics", allow_module_level=True)
 
 
 class ExtensionTestCase(IPTestCase):
@@ -20,10 +17,9 @@ class ExtensionTestCase(IPTestCase):
         super().setUp()
         self.ip.run_line_magic("load_ext", "holoviews.ipython")
         for renderer in Store.renderers.values():
-            renderer.comm_manager = CommManager
+            renderer.comm_manager = CommManager  # TODO: Should set it back
 
     def tearDown(self):
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         self.ip.run_line_magic("unload_ext", "holoviews.ipython")
         del self.ip
         super().tearDown()
@@ -53,6 +49,7 @@ class TestOptsMagic(ExtensionTestCase):
             Store.lookup_options('matplotlib',
                                  self.get_object('mat1'), 'style').options.get('cmap',None),'hot')
 
+    @pytest.mark.flaky(reruns=3)
     def test_cell_opts_style_dynamic(self):
 
         self.cell("dmap = DynamicMap(lambda X: Curve(np.random.rand(5,2), name='dmap'), kdims=['x'])"
@@ -95,6 +92,7 @@ class TestOptsMagic(ExtensionTestCase):
                                  self.get_object('mat1'), 'plot').options.get('show_title',True),False)
 
 
+    @pytest.mark.flaky(reruns=3)
     def test_cell_opts_plot_dynamic(self):
 
         self.cell("dmap = DynamicMap(lambda X: Image(np.random.rand(5,5), name='dmap'), kdims=['x'])"
diff --git a/holoviews/tests/plotting/bokeh/test_callbacks.py b/holoviews/tests/plotting/bokeh/test_callbacks.py
index 29d7c63a7..1d6e523c8 100644
--- a/holoviews/tests/plotting/bokeh/test_callbacks.py
+++ b/holoviews/tests/plotting/bokeh/test_callbacks.py
@@ -127,6 +127,7 @@ class TestCallbacks(CallbackTestCase):
         self.assertEqual(data['x'], np.arange(10))
         self.assertEqual(data['y'], np.arange(10, 20))
 
+    @pytest.mark.flaky(reruns=3)
     def test_callback_cleanup(self):
         stream = PointerX(x=0)
         dmap = DynamicMap(lambda x: Curve([x]), streams=[stream])
diff --git a/holoviews/tests/plotting/bokeh/test_curveplot.py b/holoviews/tests/plotting/bokeh/test_curveplot.py
index 5a585e55f..0ef1b5b8e 100644
--- a/holoviews/tests/plotting/bokeh/test_curveplot.py
+++ b/holoviews/tests/plotting/bokeh/test_curveplot.py
@@ -32,6 +32,7 @@ class TestCurvePlot(TestBokehPlot):
     def test_batched_curve_subscribers_correctly_linked(self):
         # Checks if a stream callback is created to link batched plot
         # to the stream
+        Callback._callbacks.clear()  # Reset to not be sensitive to other test
         posx = PointerX()
         opts = {'NdOverlay': dict(legend_limit=0),
                 'Curve': dict(line_color=Cycle(values=['red', 'blue']))}
diff --git a/holoviews/tests/plotting/bokeh/test_server.py b/holoviews/tests/plotting/bokeh/test_server.py
index 5686bfbff..418170a35 100644
--- a/holoviews/tests/plotting/bokeh/test_server.py
+++ b/holoviews/tests/plotting/bokeh/test_server.py
@@ -80,6 +80,7 @@ class TestBokehServerSetup(ComparisonTestCase):
 
 
 
+@pytest.mark.flaky(reruns=3)
 class TestBokehServer(ComparisonTestCase):
 
     def setUp(self):
@@ -128,7 +129,6 @@ class TestBokehServer(ComparisonTestCase):
         self.assertEqual(cb.streams, [stream])
         assert 'rangesupdate' in plot.state._event_callbacks
 
-    @pytest.mark.flaky(reruns=3)
     def test_launch_server_with_complex_plot(self):
         dmap = DynamicMap(lambda x_range, y_range: Curve([]), streams=[RangeXY()])
         overlay = dmap * HLine(0)
diff --git a/holoviews/tests/plotting/matplotlib/test_renderer.py b/holoviews/tests/plotting/matplotlib/test_renderer.py
index 0c8f55aa2..569a3f02d 100644
--- a/holoviews/tests/plotting/matplotlib/test_renderer.py
+++ b/holoviews/tests/plotting/matplotlib/test_renderer.py
@@ -150,6 +150,7 @@ class MPLRendererTest(ComparisonTestCase):
         self.assertEqual(obj.widget_location, 'top')
         self.assertEqual(obj.widget_type, 'individual')
 
+    @pytest.mark.filterwarnings('ignore:Attempted to send message over Jupyter Comm:UserWarning')
     def test_render_dynamicmap_with_dims(self):
         dmap = DynamicMap(lambda y: Curve([1, 2, y]), kdims=['y']).redim.range(y=(0.1, 5))
         obj, _ = self.renderer._validate(dmap, None)
@@ -164,6 +165,7 @@ class MPLRendererTest(ComparisonTestCase):
         (_, y) = artist.get_data()
         self.assertEqual(y[2], 3.1)
 
+    @pytest.mark.filterwarnings('ignore:Attempted to send message over Jupyter Comm:UserWarning')
     def test_render_dynamicmap_with_stream(self):
         stream = Stream.define('Custom', y=2)()
         dmap = DynamicMap(lambda y: Curve([1, 2, y]), kdims=['y'], streams=[stream])
@@ -178,6 +180,7 @@ class MPLRendererTest(ComparisonTestCase):
         (_, y) = artist.get_data()
         self.assertEqual(y[2], 3)
 
+    @pytest.mark.filterwarnings('ignore:Attempted to send message over Jupyter Comm:UserWarning')
     def test_render_dynamicmap_with_stream_dims(self):
         stream = Stream.define('Custom', y=2)()
         dmap = DynamicMap(lambda x, y: Curve([x, 1, y]), kdims=['x', 'y'],
diff --git a/holoviews/tests/plotting/plotly/test_renderer.py b/holoviews/tests/plotting/plotly/test_renderer.py
index 3dfce0870..4e25d32d6 100644
--- a/holoviews/tests/plotting/plotly/test_renderer.py
+++ b/holoviews/tests/plotting/plotly/test_renderer.py
@@ -1,9 +1,9 @@
 """
 Test cases for rendering exporters
 """
-
 import panel as pn
 import param
+import pytest
 from panel.widgets import DiscreteSlider, FloatSlider, Player
 from pyviz_comms import CommManager
 
@@ -92,6 +92,7 @@ class PlotlyRendererTest(ComparisonTestCase):
         self.assertEqual(obj.widget_location, 'top')
         self.assertEqual(obj.widget_type, 'individual')
 
+    @pytest.mark.filterwarnings('ignore:Attempted to send message over Jupyter Comm:UserWarning')
     def test_render_dynamicmap_with_dims(self):
         dmap = DynamicMap(lambda y: Curve([1, 2, y]), kdims=['y']).redim.range(y=(0.1, 5))
         obj, _ = self.renderer._validate(dmap, None)
@@ -105,6 +106,7 @@ class PlotlyRendererTest(ComparisonTestCase):
         y = plot.handles['fig']['data'][0]['y']
         self.assertEqual(y[2], 3.1)
 
+    @pytest.mark.filterwarnings('ignore:Attempted to send message over Jupyter Comm:UserWarning')
     def test_render_dynamicmap_with_stream(self):
         stream = Stream.define('Custom', y=2)()
         dmap = DynamicMap(lambda y: Curve([1, 2, y]), kdims=['y'], streams=[stream])
@@ -118,6 +120,7 @@ class PlotlyRendererTest(ComparisonTestCase):
         y = plot.handles['fig']['data'][0]['y']
         self.assertEqual(y[2], 3)
 
+    @pytest.mark.filterwarnings('ignore:Attempted to send message over Jupyter Comm:UserWarning')
     def test_render_dynamicmap_with_stream_dims(self):
         stream = Stream.define('Custom', y=2)()
         dmap = DynamicMap(lambda x, y: Curve([x, 1, y]), kdims=['x', 'y'],
diff --git a/holoviews/tests/test_streams.py b/holoviews/tests/test_streams.py
index 17cb93bc3..2bc957f67 100644
--- a/holoviews/tests/test_streams.py
+++ b/holoviews/tests/test_streams.py
@@ -11,7 +11,7 @@ from panel.widgets import IntSlider
 
 import holoviews as hv
 from holoviews.core.spaces import DynamicMap
-from holoviews.core.util import Version
+from holoviews.core.util import NUMPY_GE_200, Version
 from holoviews.element import Curve, Histogram, Points, Polygons, Scatter
 from holoviews.element.comparison import ComparisonTestCase
 from holoviews.streams import *  # noqa (Test all available streams)
@@ -1421,6 +1421,7 @@ class TestExprSelectionStream(ComparisonTestCase):
 
 
     def test_selection_expr_stream_polygon_index_cols(self):
+        # TODO: Should test both spatialpandas and shapely
         # Create SelectionExpr on element
         try: import shapely # noqa
         except ImportError:
@@ -1444,10 +1445,12 @@ class TestExprSelectionStream(ComparisonTestCase):
         self.assertIsNone(expr_stream.bbox)
         self.assertIsNone(expr_stream.selection_expr)
 
+        fmt = lambda x: list(map(np.str_, x)) if NUMPY_GE_200 else x
+
         expr_stream.input_streams[2].event(index=[0, 1])
         self.assertEqual(
             repr(expr_stream.selection_expr),
-            repr(dim('cat').isin(['a', 'b']))
+            repr(dim('cat').isin(fmt(['a', 'b'])))
         )
         self.assertEqual(expr_stream.bbox, None)
         self.assertEqual(len(events), 1)
@@ -1456,7 +1459,7 @@ class TestExprSelectionStream(ComparisonTestCase):
         expr_stream.input_streams[0].event(bounds=(0, 0, 4, 1))
         self.assertEqual(
             repr(expr_stream.selection_expr),
-            repr(dim('cat').isin(['a', 'b']))
+            repr(dim('cat').isin(fmt(['a', 'b'])))
         )
         self.assertEqual(len(events), 1)
 
@@ -1464,7 +1467,7 @@ class TestExprSelectionStream(ComparisonTestCase):
         expr_stream.input_streams[1].event(geometry=np.array([(0, 0), (4, 0), (4, 2), (0, 2)]))
         self.assertEqual(
             repr(expr_stream.selection_expr),
-            repr(dim('cat').isin(['a', 'b', 'c']))
+            repr(dim('cat').isin(fmt(['a', 'b', 'c'])))
         )
         self.assertEqual(len(events), 2)
 
@@ -1472,7 +1475,7 @@ class TestExprSelectionStream(ComparisonTestCase):
         expr_stream.input_streams[2].event(index=[1, 2])
         self.assertEqual(
             repr(expr_stream.selection_expr),
-            repr(dim('cat').isin(['b', 'c']))
+            repr(dim('cat').isin(fmt(['b', 'c'])))
         )
         self.assertEqual(expr_stream.bbox, None)
         self.assertEqual(len(events), 3)
diff --git a/holoviews/tests/util/test_utils.py b/holoviews/tests/util/test_utils.py
index 4355ebcb4..0d170965b 100644
--- a/holoviews/tests/util/test_utils.py
+++ b/holoviews/tests/util/test_utils.py
@@ -93,7 +93,6 @@ class TestOptsUtil(LoggingComparisonTestCase):
     def tearDown(self):
         Store.current_backend = self.backend
         Store.options(val=self.store_copy)
-        Store._custom_options = {k:{} for k in Store._custom_options.keys()}
         super().tearDown()
 
     def test_opts_builder_repr(self):