Repository: holoviz/panel

Issue:
Document how to test panel
I would like it to be easy for my self and other Panel users to test panel apps. This includes

- unittests with pytest
- performance tests with pytest-benchmark
- ui tests with PlayWright
- Load tests with Locust + PlayWright

(I believe the mentioned frameworks are the best)

Please document this.


Discussion:

Resolving patch:
diff --git a/panel/io/state.py b/panel/io/state.py
--- a/panel/io/state.py
+++ b/panel/io/state.py
@@ -696,6 +696,22 @@ def publish(
             self._rest_endpoints[endpoint] = ([parameterized], parameters, cb)
         parameterized.param.watch(cb, parameters)
 
+    def reset(self):
+        """
+        Resets the state object killing running servers and clearing
+        any other state held by the server.
+        """
+        self.kill_all_servers()
+        self._indicators.clear()
+        self._locations.clear()
+        self._templates.clear()
+        self._views.clear()
+        self._loaded.clear()
+        self.cache.clear()
+        self._scheduled.clear()
+        if self._thread_pool is not None:
+            self._thread_pool.shutdown(wait=False)
+            self._thread_pool = None
 
     def schedule_task(
         self, name: str, callback: Callable[[], None], at: Tat =None,


Test patch:
diff --git a/doc/how_to/test/loadtests.md b/doc/how_to/test/loadtests.md
new file mode 100644
--- /dev/null
+++ b/doc/how_to/test/loadtests.md
@@ -0,0 +1,246 @@
+# Add Load Tests with Locust and Playwright
+
+*Load testing* means testing the performance of the entire Panel application and the server(s) running it.
+
+This kind of testing is really useful if you want to
+
+- develop fast and snappy applications and/ or
+- develop applications that scale to many users
+
+Before you get started ensure you have installed the required dependencies:
+
+```bash
+pip install panel pytest locust pytest-playwright pytest-asyncio loadwright==0.1.0
+```
+
+and ensure `playwright` sets up the browsers it will use to display the applications:
+
+```bash
+playwright install
+```
+
+## Create the app
+
+Lets create a simple data app for testing. The app sleeps 0.5 seconds (default) when loaded and when the button is clicked.
+
+![app.py](https://assets.holoviz.org/panel/gifs/pytest.gif)
+
+Create the file `app.py` and add the code below:
+
+```python
+# app.py
+import time
+
+import panel as pn
+import param
+
+class App(pn.viewable.Viewer):
+    run = param.Event(doc="Runs for click_delay seconds when clicked")
+    runs = param.Integer(doc="The number of runs")
+    status = param.String("No runs yet")
+
+    load_delay = param.Number(0.5)
+    run_delay = param.Number(0.5)
+
+    def __init__(self, **params):
+        super().__init__(**params)
+
+        result = self._load()
+        self._time = time.time()
+
+
+        self._status_pane = pn.pane.Markdown(self.status, height=40, align="start", margin=(0,5,10,5))
+        self._result_pane = pn.Column(result)
+        self._view = pn.Column(
+            pn.Row(pn.widgets.Button.from_param(self.param.run, sizing_mode="fixed"), self._status_pane),
+            self._result_pane
+        )
+
+    def __panel__(self):
+        return self._view
+
+    def _start_run(self):
+        self.status = f"Running ..."
+        self._time = time.time()
+
+    def _stop_run(self):
+        now = time.time()
+        duration = round(now-self._time,3)
+        self._time = now
+        self.runs+=1
+        self.status=f"Finished run {self.runs} in {duration}sec"
+
+    @pn.depends("run", watch=True)
+    def _run_with_status_update(self):
+        self._start_run()
+        self._result_pane[:] = [self._run()]
+        self._stop_run()
+
+    @pn.depends("status", watch=True)
+    def _update_status_pane(self):
+        self._status_pane.object = self.status
+
+    def _load(self):
+        time.sleep(self.load_delay)
+        return "Loaded"
+
+    def _run(self):
+        time.sleep(self.run_delay)
+        return f"Result {self.runs+1}"
+
+if pn.state.served:
+    pn.extension(sizing_mode="stretch_width")
+    App().servable()
+```
+
+Serve the app via `panel serve app.py` and open [http://localhost:5006/app](http://localhost:5006/app) in your browser.
+
+## Create a conftest.py
+
+The `conftest.py` file should be placed alongside your tests and will be loaded automatically by pytest. It is often used to declare [fixtures](https://docs.pytest.org/en/6.2.x/fixture.html) that allow you declare reusable components. It will:
+
+- provide us with an available `port`.
+- clean up the Panel *state* after each test.
+
+Create the file `conftest.py` and add the code below.
+
+```python
+# conftest.py
+"""Shared configuration and fixtures for testing Panel"""
+import panel as pn
+import pytest
+
+PORT = [6000]
+
+@pytest.fixture
+def port():
+    PORT[0] += 1
+    return PORT[0]
+
+@pytest.fixture(autouse=True)
+def server_cleanup():
+    """
+    Clean up server state after each test.
+    """
+    try:
+        yield
+    finally:
+        pn.state.reset()
+```
+
+For more inspiration see the [Panel `conftest.py` file](https://github.com/holoviz/panel/blob/main/panel/tests/conftest.py)
+
+## Test the initial load with Locust
+
+[Locust](https://locust.io/) can help you test the behaviour of users that load (i.e. requests) your Panel app. Locust provides many useful performance related statistics and charts.
+
+Create the file `locustfile.py` and add the code below.
+
+```python
+#locustfile.py
+from locust import HttpUser, task
+
+class RequestOnlyUser(HttpUser):
+    @task
+    def goto(self):
+        self.client.get("/app")
+```
+
+Start the Panel server:
+
+```bash
+panel serve app.py
+```
+
+Start the Locust server
+
+```bash
+locust --host http://localhost:5006
+```
+
+Open [http://localhost:8089](http://localhost:8089). Keep the default settings and click the *Start swarming* button. This should look like the below:
+
+![panel-locust.gif](https://assets.holoviz.org/panel/gifs/locust.gif)
+
+The median response time is ~530ms when one user requests the page every second. If you try to increase to 10 simultanous users you will see a median response time of ~5300ms. If this is a likely scenario, you will have to look into how to improve the performance of your app.
+
+## Test advanced interactions with Loadwright
+
+[Loadwright](https://github.com/awesome-panel/loadwright) is a young load testing framework built on top of Playwright and Panel.
+
+Lets replicate a user that:
+
+- Opens the app in the browser
+- Clicks the *Run* button `n_clicks` times.
+
+Create the file `test_loadwright.py` and add the code below:
+
+```python
+# test_loadwright.py
+import param
+import pytest
+
+from loadwright import LoadTestRunner, LoadTestViewer, User
+
+import panel as pn
+
+from app import App
+
+HEADLESS = False
+
+class LoadAndClickUser(User):
+    """A custom User that loads a page and clicks a button n_clicks times"""
+
+    n_clicks = param.Integer(
+        default=1, bounds=(0, None), doc="The number of times to click the button"
+    )
+
+    async def run(self):
+        with self.event(name="load", user=self.name):
+            await self.page.goto(self.url)
+            await self.page.get_by_role("button", name="Run").wait_for()
+        await self.sleep()
+
+        for click_index in range(self.n_clicks):
+            with self.event(name="interact", user=self.name):
+                await self.page.get_by_role("button", name="Run").first.click()
+                await self.page.get_by_text(f"Finished run {click_index+1}").wait_for()
+            await self.sleep()
+
+@pytest.mark.asyncio
+async def test_component_2(port=6001):
+    """We can run the LoadTestRunner with 5 users each clicking 5 times"""
+    async with LoadTestRunner.serve(App, port=port) as host:
+        runner = LoadTestRunner(host=host, headless=HEADLESS, user=LoadAndClickUser(n_clicks=5), n_users=5)
+        await runner.run()
+        # You can access the data of the run via runner.logger.data
+        assert runner.logger.data.duration.mean()<=10
+
+if pn.state.served:
+    import panel as pn
+    pn.extension(sizing_mode="stretch_width")
+    viewer = LoadTestViewer(data="test_results/loadwright.csv")
+    pn.template.FastListTemplate(
+        site="LoadWright",
+        title="Load Testing with Playwright and Panel",
+        main=[viewer]
+    ).servable()
+```
+
+Run the tests with pytest
+
+```bash
+pytest test_loadwright.py
+```
+
+View the results with Panel
+
+```bash
+$ panel serve test_loadwright.py
+...
+2023-01-01 15:53:03,396 Bokeh app running at: http://localhost:5006/test_loadwright
+```
+
+![Loadwright Demo](https://assets.holoviz.org/panel/gifs/loadwright.gif).
+
+You will find an *archive* of *test results* in the `tests_results/archive` folder.
diff --git a/doc/how_to/test/pytest.md b/doc/how_to/test/pytest.md
new file mode 100644
--- /dev/null
+++ b/doc/how_to/test/pytest.md
@@ -0,0 +1,205 @@
+# Add Unit and Performance tests with Pytest
+
+Testing is key to developing robust and performant applications. You can test Panel data apps using
+Python and the test tools you know and love.
+
+[![pytest](https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Pytest_logo.svg/600px-Pytest_logo.svg.png)](https://docs.pytest.org/en/latest/)
+
+[Pytest](https://docs.pytest.org/en/latest/) is the most used Python testing framework. We will use it below to write *unit* and *performance* tests.
+
+Before we get started, you should
+
+```bash
+pip install panel pytest pytest-benchmark
+```
+
+## Create the app
+
+Let's create a simple data app for testing. The app sleeps 0.5 seconds (default) when loaded and when the button is clicked.
+
+![app.py](https://assets.holoviz.org/panel/gifs/pytest.gif)
+
+Create the file `app.py` and add the code below (don't worry about the contents of the app for now):
+
+```python
+# app.py
+import time
+
+import panel as pn
+import param
+
+class App(pn.viewable.Viewer):
+    run = param.Event(doc="Runs for click_delay seconds when clicked")
+    runs = param.Integer(doc="The number of runs")
+    status = param.String("No runs yet")
+
+    load_delay = param.Number(0.5)
+    run_delay = param.Number(0.5)
+
+    def __init__(self, **params):
+        super().__init__(**params)
+
+        result = self._load()
+        self._time = time.time()
+
+        self._status_pane = pn.pane.Markdown(self.status, height=40, align="start", margin=(0,5,10,5))
+        self._result_pane = pn.Column(result)
+
+        button = pn.widgets.Button.from_param(self.param.run, sizing_mode="fixed")
+        self._view = pn.Column(
+            pn.Row(button, self._status_pane),
+            self._result_pane
+        )
+
+    def __panel__(self):
+        return self._view
+
+    def _start_run(self):
+        self.status = f"Running ..."
+        self._time = time.time()
+
+    def _stop_run(self):
+        now = time.time()
+        duration = round(now-self._time,3)
+        self._time = now
+        self.runs += 1
+        self.status = f"Finished run {self.runs} in {duration}sec"
+
+    @pn.depends("run", watch=True)
+    def _run_with_status_update(self):
+        self._start_run()
+        self._result_pane[:] = [self._run()]
+        self._stop_run()
+
+    @pn.depends("status", watch=True)
+    def _update_status_pane(self):
+        self._status_pane.object = self.status
+
+    def _load(self):
+        time.sleep(self.load_delay)
+        return "Loaded"
+
+    def _run(self):
+        time.sleep(self.run_delay)
+        return f"Result {self.runs+1}"
+
+
+if pn.state.served:
+    pn.extension(sizing_mode="stretch_width")
+
+    App().servable()
+```
+
+Now serve the app via `panel serve app.py` and open [http://localhost:5006/app](http://localhost:5006/app) in your browser to see what it does.
+
+## Create the unit tests
+
+Let's test:
+
+- The initial *state* of the App
+- That the app *state* changes appropriately when the *Run* button is clicked.
+
+Create the file `test_app.py` and add the code below.
+
+```python
+# test_app.py
+import pytest
+
+from app import App
+
+@pytest.fixture
+def app():
+    return App(sleep_delay=0.001, load_delay=0.001)
+
+def test_constructor(app):
+    """Tests default values of App"""
+    # Then
+    assert app.run == False
+    assert app.status == "No runs yet"
+    assert app.runs == 0
+
+def test_run(app):
+    """Tests behaviour when Run button is clicked once"""
+    # When
+    app.param.trigger('run')
+    # Then
+    assert app.runs == 1
+    assert app.status.startswith("Finished run 1 in")
+
+def test_run_twice(app):
+    """Tests behaviour when Run button is clicked twice"""
+    # When
+    app.param.trigger('run')
+    app.param.trigger('run')
+    # Then
+    assert app.runs == 2
+    assert app.status.startswith("Finished run 2 in")
+```
+
+Let's run `pytest test_app.py`:
+
+```bash
+$ pytest test_app.py
+=================================== test session starts
+...
+collected 3 items
+
+test_app.py ...                                                                       [100%]
+
+=============================== 3 passed
+```
+
+## Create a performance test
+
+The performance of your data app is key to providing a good user experience. You can test the performance of functions and methods using [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark).
+
+Let's test that:
+
+- the *duration* of the run is as expected.
+
+Create the file `test_app_performance.py`:
+
+```python
+# test_app_performance.py
+import pytest
+from app import App
+
+@pytest.fixture
+def app():
+    return App(run_delay=0.001, load_delay=0.001)
+
+def test_run_performance(app: App, benchmark):
+    """Test the duration when the Run button is clicked"""
+    app.run_delay=0.3
+
+    def run():
+        app.param.trigger('run')
+
+    benchmark(run)
+    assert benchmark.stats['min'] >= 0.3
+    assert benchmark.stats['max'] < 0.4
+```
+
+Run `pytest test_app_performance.py`.
+
+```bash
+$ pytest test_app_performance.py
+============================================================================================================================= test session starts
+...
+collected 1 item
+
+test_app_performance.py .                                                                                                                                                                                                                                                 [100%]
+
+------------------------------------------------- benchmark: 1 tests ------------------------------------------------
+Name (time in ms)             Min       Max      Mean  StdDev    Median     IQR  Outliers     OPS  Rounds  Iterations
+---------------------------------------------------------------------------------------------------------------------
+test_run_performance     307.6315  316.8270  312.2731  4.2335  314.1614  7.5190       3;0  3.2023       5           1
+---------------------------------------------------------------------------------------------------------------------
+
+Legend:
+  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
+  OPS: Operations Per Second, computed as 1 / Mean
+========================================================================================================================= 1 passed in 3.23s
+```
+
+Notice how we used the `benchmark` *fixture* of [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/latest/) to test the performance of the `run` event.
diff --git a/doc/how_to/test/uitests.md b/doc/how_to/test/uitests.md
new file mode 100644
--- /dev/null
+++ b/doc/how_to/test/uitests.md
@@ -0,0 +1,193 @@
+# Add UI tests with Pytest and Playwright
+
+Testing is key to developing robust and performant applications. Particularly when you build complex UIs you will want to ensure that it behaves as expected. Unit tests will allow you test that the logic on the backend behaves correctly, but to test that the UI is rendered correctly and responds appropriately, it can be useful to also test the UI.
+
+For testing the UI we recommend the framework [Playwright](https://playwright.dev/). Panel itself is tested with this framework.
+
+Before you get started ensure you have installed the required dependencies:
+
+```bash
+pip install panel pytest pytest-playwright
+```
+
+and ensure `playwright` sets up the browsers it will use to display the applications:
+
+```bash
+playwright install
+```
+
+## Create the app
+
+Let's create a simple data app for testing. The app sleeps 0.5 seconds (default) when loaded and when the button is clicked.
+
+![app.py](https://assets.holoviz.org/panel/gifs/pytest.gif)
+
+Create the file `app.py` and add the code below (don't worry about the contents of the app for now):
+
+```python
+# app.py
+import time
+
+import panel as pn
+import param
+
+class App(pn.viewable.Viewer):
+    run = param.Event(doc="Runs for click_delay seconds when clicked")
+    runs = param.Integer(doc="The number of runs")
+    status = param.String("No runs yet")
+
+    load_delay = param.Number(0.5)
+    run_delay = param.Number(0.5)
+
+    def __init__(self, **params):
+        super().__init__(**params)
+
+        result = self._load()
+        self._time = time.time()
+
+
+        self._status_pane = pn.pane.Markdown(self.status, height=40, align="start", margin=(0,5,10,5))
+        self._result_pane = pn.Column(result)
+        self._view = pn.Column(
+            pn.Row(pn.widgets.Button.from_param(self.param.run, sizing_mode="fixed"), self._status_pane),
+            self._result_pane
+        )
+
+    def __panel__(self):
+        return self._view
+
+    def _start_run(self):
+        self.status = f"Running ..."
+        self._time = time.time()
+
+    def _stop_run(self):
+        now = time.time()
+        duration = round(now-self._time,3)
+        self._time = now
+        self.runs+=1
+        self.status=f"Finished run {self.runs} in {duration}sec"
+
+    @pn.depends("run", watch=True)
+    def _run_with_status_update(self):
+        self._start_run()
+        self._result_pane[:] = [self._run()]
+        self._stop_run()
+
+    @pn.depends("status", watch=True)
+    def _update_status_pane(self):
+        self._status_pane.object = self.status
+
+    def _load(self):
+        time.sleep(self.load_delay)
+        return "Loaded"
+
+    def _run(self):
+        time.sleep(self.run_delay)
+        return f"Result {self.runs+1}"
+
+
+if pn.state.served:
+    pn.extension(sizing_mode="stretch_width")
+
+    App().servable()
+```
+
+Serve the app via `panel serve app.py` and open [http://localhost:5006/app](http://localhost:5006/app) in your browser to see what it does.
+
+## Create a conftest.py
+
+The `conftest.py` file should be placed alongside your tests and will be loaded automatically by pytest. It is often used to declare [fixtures](https://docs.pytest.org/en/latest/explanation/fixtures.html) that allow you declare reusable components. It will:
+
+- provide us with an available `port`.
+- clean up the Panel *state* after each test.
+
+Create the file `conftest.py` and add the code below.
+
+```python
+# conftest.py
+"""Shared configuration and fixtures for testing Panel"""
+import panel as pn
+import pytest
+
+PORT = [6000]
+
+@pytest.fixture
+def port():
+    PORT[0] += 1
+    return PORT[0]
+
+@pytest.fixture(autouse=True)
+def server_cleanup():
+    """
+    Clean up server state after each test.
+    """
+    try:
+        yield
+    finally:
+        pn.state.reset()
+```
+
+For more inspiration see the [Panel `conftest.py` file](https://github.com/holoviz/panel/blob/main/panel/tests/conftest.py)
+
+### Test the app UI
+
+Now let us actually set up some UI tests, we will want to assert that the app:
+
+- Responds when we make an initial request
+- Renders a *Run* button
+- Updates as expected when the *Run* button is clicked
+
+Create the file `test_app_frontend.py` and add the code below.
+
+```python
+# test_app_frontend.py
+import time
+
+import panel as pn
+from app import App
+
+CLICKS = 2
+
+def test_component(page, port):
+    # Given
+    component = App()
+    url = f"http://localhost:{port}"
+    # When
+    server = pn.serve(component, port=port, threaded=True, show=False)
+    time.sleep(0.2)
+    # Then
+    page.goto(url)
+    page.get_by_role("button", name="Run").wait_for()
+
+    for index in range(CLICKS):
+        page.get_by_role("button", name="Run").first.click()
+        page.get_by_text(f"Finished run {index+1}").wait_for()
+    # Clean up
+    server.stop()
+```
+
+Let's run `pytest`. We will add the `--headed` and `--slowmo` arguments to see what is going on in the browser. This is very illustrative and also helpful for debugging purposes.
+
+```bash
+pytest test_app_frontend.py --headed --slowmo 1000
+```
+
+![Playwright UI test with --headed enabled](https://assets.holoviz.org/panel/gifs/uitest.gif)
+
+### Record the test code
+
+Writing code to test complex UIs can be quite cumbersome, thankfully there is an easier way. Playwright allows you to record UI interactions as you navigate your live app and translates these interactions as code (see the [Playwright codegen](https://playwright.dev/python/docs/codegen) documentation for more detail).
+
+You can try it yourself by launching the app again:
+
+```bash
+panel serve app.py
+```
+
+and starting the Playwright recorder:
+
+```bash
+playwright codegen http://localhost:5006/app
+```
+
+![Playwright Code generation demo](https://assets.holoviz.org/panel/gifs/codegen.gif)
diff --git a/panel/tests/conftest.py b/panel/tests/conftest.py
--- a/panel/tests/conftest.py
+++ b/panel/tests/conftest.py
@@ -311,7 +311,6 @@ def module_cleanup():
         if not any(model.__module__.startswith(tr) for tr in to_reset)
     }
 
-
 @pytest.fixture(autouse=True)
 def server_cleanup():
     """
@@ -320,17 +319,7 @@ def server_cleanup():
     try:
         yield
     finally:
-        state.kill_all_servers()
-        state._indicators.clear()
-        state._locations.clear()
-        state._templates.clear()
-        state._views.clear()
-        state._loaded.clear()
-        state.cache.clear()
-        state._scheduled.clear()
-        if state._thread_pool is not None:
-            state._thread_pool.shutdown(wait=False)
-            state._thread_pool = None
+        state.reset()
 
 @pytest.fixture(autouse=True)
 def cache_cleanup():