Repository: zulip/zulip

Issue:
Some Puppeteer tests fail when repeated during the same run
The `tools/test-js-with-puppeteer --interactive` mechanism for debugging failing tests depends on the ability to repeatedly run the same test without restarting the test server. A shell command for trying every test repeated twice is:

```bash
tools/test-js-with-puppeteer {00..16}{,}
```

This uncovers several failures, such as:

```console
$ tools/test-js-with-puppeteer 02 02
…
AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected ... Lines skipped

  [
    [
      'Verona > test',
      [
+       'verona test d',
        'verona test a',
...
      ]
    ]
  ]
    at CommonUtils.check_messages_sent (/srv/zulip/frontend_tests/puppeteer_lib/common.ts:453:16)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at expect_verona_stream (/srv/zulip/frontend_tests/puppeteer_tests/02-message-basics.ts:27:5)
    at test_navigations_from_home (/srv/zulip/frontend_tests/puppeteer_tests/02-message-basics.ts:96:5)
    at message_basic_tests (/srv/zulip/frontend_tests/puppeteer_tests/02-message-basics.ts:450:5)
    at CommonUtils.run_test (/srv/zulip/frontend_tests/puppeteer_lib/common.ts:510:13) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: [Array],
  expected: [Array],
  operator: 'deepStrictEqual'
}
Waiting for children to stop...

The Puppeteer frontend tests failed!
For help debugging, read:
  https://zulip.readthedocs.io/en/latest/testing/testing-with-puppeteer.html
or report and ask for help in chat.zulip.org
It's also worthy to see screenshots generated on failure stored under var/puppeteer/*.png
```

```console
$ tools/test-js-with-puppeteer 04 04
…
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ 'Verona'
- 'Puppeteer'
    at create_stream (/srv/zulip/frontend_tests/puppeteer_tests/04-subscriptions.ts:174:12)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at test_stream_creation (/srv/zulip/frontend_tests/puppeteer_tests/04-subscriptions.ts:217:5)
    at subscriptions_tests (/srv/zulip/frontend_tests/puppeteer_tests/04-subscriptions.ts:256:5)
    at CommonUtils.run_test (/srv/zulip/frontend_tests/puppeteer_lib/common.ts:510:13) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 'Verona',
  expected: 'Puppeteer',
  operator: 'strictEqual'
}
Waiting for children to stop...

The Puppeteer frontend tests failed!
For help debugging, read:
  https://zulip.readthedocs.io/en/latest/testing/testing-with-puppeteer.html
or report and ask for help in chat.zulip.org
It's also worthy to see screenshots generated on failure stored under var/puppeteer/*.png
```



Discussion:
Hello @zulip/server-testing members, this issue was labeled with the "area: testing-infrastructure" label, so you may want to check it out!

<!-- areaLabelAddition -->
@zulipbot claim
I think the best path forward for this might be to reset the frontend testing database after each individual test file is completed, rather than only during startup.

Resolving patch:
diff --git a/zerver/models.py b/zerver/models.py
--- a/zerver/models.py
+++ b/zerver/models.py
@@ -1821,6 +1821,11 @@ def __str__(self) -> str:
 get_client_cache: Dict[str, Client] = {}
 
 
+def clear_client_cache() -> None:  # nocoverage
+    global get_client_cache
+    get_client_cache = {}
+
+
 def get_client(name: str) -> Client:
     # Accessing KEY_PREFIX through the module is necessary
     # because we need the updated value of the variable.
diff --git a/zerver/views/development/cache.py b/zerver/views/development/cache.py
new file mode 100644
--- /dev/null
+++ b/zerver/views/development/cache.py
@@ -0,0 +1,21 @@
+import os
+
+from django.http import HttpRequest, HttpResponse
+from django.views.decorators.csrf import csrf_exempt
+
+from zerver.decorator import require_post
+from zerver.lib.cache import get_cache_backend
+from zerver.lib.response import json_success
+from zerver.models import clear_client_cache, flush_per_request_caches
+
+ZULIP_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../")
+
+# This is used only by the Puppeteer Tests to clear all the cache after each run.
+@csrf_exempt
+@require_post
+def remove_caches(request: HttpRequest) -> HttpResponse:
+    cache = get_cache_backend(None)
+    cache.clear()
+    clear_client_cache()
+    flush_per_request_caches()
+    return json_success()
diff --git a/zproject/dev_urls.py b/zproject/dev_urls.py
--- a/zproject/dev_urls.py
+++ b/zproject/dev_urls.py
@@ -10,6 +10,7 @@
 from django.views.static import serve
 
 from zerver.views.auth import config_error, login_page
+from zerver.views.development.cache import remove_caches
 from zerver.views.development.email_log import clear_emails, email_page, generate_all_emails
 from zerver.views.development.integrations import (
     check_send_webhook_fixture_message,
@@ -73,6 +74,8 @@
     path("devtools/integrations/<integration_name>/fixtures", get_fixtures),
     path("config-error/<error_category_name>", config_error, name="config_error"),
     path("config-error/remoteuser/<error_category_name>", config_error),
+    # Special endpoint to remove all the server-side caches.
+    path("flush_caches", remove_caches),
 ]
 
 # Serve static assets via the Django server


Test patch:
diff --git a/zerver/lib/test_fixtures.py b/zerver/lib/test_fixtures.py
--- a/zerver/lib/test_fixtures.py
+++ b/zerver/lib/test_fixtures.py
@@ -386,3 +386,49 @@ def remove_test_run_directories(expiry_time: int = 60 * 60) -> int:
             except FileNotFoundError:
                 pass
     return removed
+
+
+def reset_zulip_test_database() -> None:
+    """
+    This function is used to reset the zulip_test database fastest way possible,
+    i.e. First, it deletes the database and then clones it from zulip_test_template.
+    This function is used with puppeteer tests, so it can quickly reset the test
+    database after each run.
+    """
+    from zerver.lib.test_runner import destroy_test_databases
+
+    # Make sure default database is 'zulip_test'.
+    assert connections["default"].settings_dict["NAME"] == "zulip_test"
+
+    # Clearing all the active PSQL sessions with 'zulip_test'.
+    run(
+        [
+            "env",
+            "PGHOST=localhost",
+            "PGUSER=zulip_test",
+            "scripts/setup/terminate-psql-sessions",
+            "zulip_test",
+        ]
+    )
+
+    destroy_test_databases()
+    # Pointing default database to test database template, so we can instantly clone it.
+    settings.DATABASES["default"]["NAME"] = settings.BACKEND_DATABASE_TEMPLATE
+    connection = connections["default"]
+    clone_database_suffix = "clone"
+    connection.creation.clone_test_db(
+        suffix=clone_database_suffix,
+    )
+    settings_dict = connection.creation.get_test_db_clone_settings(clone_database_suffix)
+    # We manually rename the clone database to 'zulip_test' because when cloning it,
+    # its name is set to original database name + some suffix.
+    # Also, we need it to be 'zulip_test' so that our running server can recognize it.
+    with connection.cursor() as cursor:
+        cursor.execute("ALTER DATABASE zulip_test_template_clone RENAME TO zulip_test;")
+    settings_dict["NAME"] = "zulip_test"
+    # connection.settings_dict must be updated in place for changes to be
+    # reflected in django.db.connections. If the following line assigned
+    # connection.settings_dict = settings_dict, new threads would connect
+    # to the default database instead of the appropriate clone.
+    connection.settings_dict.update(settings_dict)
+    connection.close()
diff --git a/zerver/lib/test_helpers.py b/zerver/lib/test_helpers.py
--- a/zerver/lib/test_helpers.py
+++ b/zerver/lib/test_helpers.py
@@ -495,6 +495,7 @@ def find_pattern(pattern: Any, prefixes: List[str]) -> None:
             "docs/(?P<path>.+)",
             "casper/(?P<path>.+)",
             "static/(?P<path>.+)",
+            "flush_caches",
             *(webhook.url for webhook in WEBHOOK_INTEGRATIONS if not include_webhooks),
         }