Repository: learningequality/kolibri

Issue:
Kolibri Logs Are Always Classified as ERROR in Google Cloud Logging
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

## Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->

[Discussed on Slack.
](https://learningequality.slack.com/archives/C02AW6J5ZNC/p1641952761041900?thread_ts=1641657482.030500&cid=C02AW6J5ZNC)
The [GCloud Logging docs indicate](https://cloud.google.com/stackdriver/docs/solutions/gke/managing-logs#best_practices
) that this is happening because the application write to STDERR instead of STDOUT:

> Severities: By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level.

Some maybe-helpful notes from that chat:
@rtibbles:
Hrm not seeing a prima facie reason why it would be going to stderr not stdout, our console handler is following the example here https://docs.python.org/3/library/logging.config.html#logging-config-dictschema
Oh, except we don't specify the stream. Can try tweaking it for 0.15.1
I think, looking at the way that the Python StreamHandler class works, it sends everything to the same stream, but I'll double check.
We might need to define separate handlers in order to split the streams

## Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

That we have INFO and ERROR distinctions, at the very least.

## User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->

Makes debugging cloud instances more of a sift than a hunt. 

## Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

Run in Ubuntu LTS 18 containers on the containerd runtime inside Kubernetes.



Discussion:
Yes, the stream defaults to stderr. We would need to have two different console handlers, one that outputs to stdout, one to stderr.
This is incredibly annoying because it produces logs in GCP Log Explorer that look like this (minus the pixelation):

![image](https://user-images.githubusercontent.com/1593835/157296071-51fc760e-c546-4ed3-b845-e59605ae247e.png)


These are responses to simple `/status/` calls that result in an HTTP 200, yet they're flagged as errors in the UI because they come over `stderr` and not `stdout`.

Resolving patch:
diff --git a/kolibri/utils/env.py b/kolibri/utils/env.py
--- a/kolibri/utils/env.py
+++ b/kolibri/utils/env.py
@@ -24,14 +24,16 @@
 logging.StreamHandler(sys.stdout)
 
 if StreamHandler and getLogger and ColoredFormatter:
-    handler = StreamHandler()
+    handler = StreamHandler(stream=sys.stdout)
     handler.setFormatter(
         ColoredFormatter(
             fmt="%(log_color)s%(levelname)-8s %(message)s", log_colors=LOG_COLORS
         )
     )
+    handler.setLevel(logging.INFO)
     logger = getLogger("env")
     logger.addHandler(handler)
+    logger.propagate = False
 else:
     logger = logging.getLogger("env")
 
diff --git a/kolibri/utils/logger.py b/kolibri/utils/logger.py
--- a/kolibri/utils/logger.py
+++ b/kolibri/utils/logger.py
@@ -121,6 +121,16 @@ def filter(self, record):
     return RequireDebugTrue
 
 
+class NoExceptionsFilter(logging.Filter):
+    """
+    A filter that ignores errors and critical messages, used to suppress
+    error messages to stdout that are also being piped to stderr.
+    """
+
+    def filter(self, record):
+        return record.levelno < logging.ERROR
+
+
 def get_default_logging_config(LOG_ROOT, debug=False, debug_database=False):
     """
     A minimal logging config for just kolibri without any Django
@@ -131,7 +141,9 @@ def get_default_logging_config(LOG_ROOT, debug=False, debug_database=False):
     """
 
     DEFAULT_HANDLERS = (
-        ["console"] if NO_FILE_BASED_LOGGING else ["file", "console", "file_debug"]
+        ["console", "console-error"]
+        if NO_FILE_BASED_LOGGING
+        else ["file", "console", "console-error", "file_debug"]
     )
 
     # This is the general level
@@ -141,7 +153,10 @@ def get_default_logging_config(LOG_ROOT, debug=False, debug_database=False):
     return {
         "version": 1,
         "disable_existing_loggers": False,
-        "filters": {"require_debug_true": {"()": FalseFilter}},  # Replaced later
+        "filters": {
+            "require_debug_true": {"()": FalseFilter},  # Replaced later
+            "no_exceptions": {"()": NoExceptionsFilter},
+        },
         "formatters": {
             "verbose": {
                 "format": "%(levelname)s %(asctime)s %(name)s %(process)d %(thread)d %(message)s"
@@ -155,10 +170,18 @@ def get_default_logging_config(LOG_ROOT, debug=False, debug_database=False):
             },
         },
         "handlers": {
+            "console-error": {
+                "level": "ERROR",
+                "class": "logging.StreamHandler",
+                "formatter": "color",
+                "stream": "ext://sys.stderr",
+            },
             "console": {
                 "level": DEFAULT_LEVEL,
+                "filters": ["no_exceptions"],
                 "class": "logging.StreamHandler",
                 "formatter": "color",
+                "stream": "ext://sys.stdout",
             },
             "file": {
                 "level": "INFO",
@@ -178,6 +201,10 @@ def get_default_logging_config(LOG_ROOT, debug=False, debug_database=False):
             },
         },
         "loggers": {
+            "": {
+                "handlers": DEFAULT_HANDLERS,
+                "level": DEFAULT_LEVEL,
+            },
             "kolibri": {
                 "handlers": DEFAULT_HANDLERS,
                 "level": DEFAULT_LEVEL,


Test patch: