Repository: carpentries/amy

Issue:
Remove logging output from test runs
When running `./manage.py test` all `logging.debug()` (etc.) calls are shown in the console.

![obraz](https://user-images.githubusercontent.com/72821/222895516-88d4b82f-e13b-41f2-a601-14f582e277c4.png)


They should be conditionally disabled in the logs, or captured and shown after the suite is run (like `pytest` does).

Unfortunately switching to `pytest` is not straighforward in our case (I tried it and some tests failed when run in `pytest`, fortunately most of them run fine).

The solution is to either:
1) Create a filter in logging settings, so that when they run in test environment they don't output to console (not ideal solution)
2) Create a custom test runner that suppresses log output (preferred way) by either selecting their threshold level very high (not ideal) or by intercepting the log output internally (preferred way although probably more difficult).


Discussion:

Resolving patch:
diff --git a/config/settings.py b/config/settings.py
--- a/config/settings.py
+++ b/config/settings.py
@@ -575,6 +575,11 @@
     "DEFAULT_RESULT_TTL": 31536000,  # 1 year in seconds for keeping job results
 }
 
+# Test runner
+# -----------------------------------------------------------------------------
+# A custom test runner tailored for our needs.
+# https://docs.djangoproject.com/en/4.1/topics/testing/advanced/#defining-a-test-runner
+TEST_RUNNER = "workshops.tests.runner.SilenceLogsRunner"
 
 # Autoemails application settings
 # -----------------------------------------------------------------------------


Test patch:
diff --git a/amy/workshops/tests/runner.py b/amy/workshops/tests/runner.py
new file mode 100644
--- /dev/null
+++ b/amy/workshops/tests/runner.py
@@ -0,0 +1,60 @@
+import io
+import logging
+import sys
+
+from django.test.runner import DiscoverRunner
+
+
+class SilenceLogsRunner(DiscoverRunner):
+    log_output: bool
+
+    def __init__(self, **kwargs):
+        self.log_output = kwargs.pop("log_output", False)
+        super().__init__(**kwargs)
+
+    @classmethod
+    def add_arguments(cls, parser):
+        """New argument to CLI to output loggers at the end of the suite."""
+        super().add_arguments(parser)
+        parser.add_argument(
+            "--log-output",
+            action="store_true",
+            help="Prints logger entries after the test suite finishes.",
+        )
+
+    def get_all_loggers(self) -> list[logging.Logger]:
+        root_logger = logging.getLogger()
+        return [root_logger] + [
+            logging.getLogger(name) for name in logging.root.manager.loggerDict
+        ]
+
+    def print_handler_stream_to_stderr(self, handler: logging.StreamHandler) -> None:
+        stream_value = handler.stream.getvalue()
+        if stream_value:
+            print(stream_value, file=sys.stderr)
+
+    def setup_test_environment(self, **kwargs) -> None:
+        loggers = self.get_all_loggers()
+        for logger in loggers:
+            for handler in logger.handlers:
+                if not isinstance(handler, logging.StreamHandler):
+                    continue
+
+                handler.stream = io.StringIO()
+
+        return super().setup_test_environment(**kwargs)
+
+    def teardown_test_environment(self, **kwargs) -> None:
+        if self.log_output:
+            print(
+                "--------------------------- Logger output ----------------------------"
+            )
+            loggers = self.get_all_loggers()
+            for logger in loggers:
+                for handler in logger.handlers:
+                    if not isinstance(handler, logging.StreamHandler):
+                        continue
+
+                    self.print_handler_stream_to_stderr(handler)
+
+        return super().teardown_test_environment(**kwargs)