Repository: zulip/zulip

Issue:
obscure error messages when zulip push notification gateway is unreachable
our zulip server is hosted in a dmz .

accidentally,  the firewall rule for allowing outgoing connection to push.zulipchat.com was not active.  we block all outgoing connections by default, as we do not want to be some "hop" or source of spam and illegal action, if some machine is being hacked inside the dmz.

we did not take any notice on this,and users didn't complain because push notifications are not that important to them.

but we started to wonder, why we got weird and very intermittend notification mails from django module like this

```
Subject: [Django] chat:Bad Gateway:/api/v1/users/me/apns_device_token

Logger django.request, from module django.utils.log line 230:
Error generated by <**username**> <**user email**> (Member) on <**servername**> deployment

No stack trace available

Deployed code:
- git: None
- ZULIP_VERSION: 4.3


Request info:
- path: /api/v1/users/me/apns_device_token
- POST: {'token': ['<**token string**>'], 'appid': ['org.zulip.Zulip']}
- REMOTE_ADDR: "<**IP**>"
- QUERY_STRING: ""
- SERVER_NAME: ""
```

in server log, there is proper logging of gateway unreachablility :
```
2021-08-17 00:14:59.285 WARN [] ConnectionError while trying to connect to push notification bouncer
2021-08-17 01:06:19.380 WARN [] ConnectionError while trying to connect to push notification bouncer
2021-08-17 02:05:03.906 WARN [] ConnectionError while trying to connect to push notification bouncer
```

it looks there is some room for improvement:
1. don't send weird django error message
2. send proper notification to the admin or/and to the end-user, if push notification message could not be sent

also see https://chat.zulip.org/#narrow/stream/31-production-help/topic/.5BDjango.5D.20chat.3ABad.20Gateway.3A.2Fapi.2Fv1.2Fusers.2Fme.2Fapns_device_token/near/1246328


Discussion:
Hello @zulip/server-api, @zulip/server-production members, this issue was labeled with the "area: production", "area: api" labels, so you may want to check it out!

<!-- areaLabelAddition -->
That error email does indeed seem pretty messy for what is fundamentally a networking issue connecting to the [mobile push notifications service](https://zulip.readthedocs.io/en/latest/production/mobile-push-notifications.html). The code path is here:

```
$ git grep 'while trying to connect'
zerver/lib/remote_server.py:            f"{e.__class__.__name__} while trying to connect to push notification bouncer"
zerver/tests/test_push_notifications.py:                    "ConnectionError while trying to connect to push notification bouncer",
zerver/tests/test_push_notifications.py:                "ConnectionError while trying to connect to push notification bouncer"
```

And my guess is that we're somewhere raising an exception or doing a `logging.error` type call without passing a proper stack trace. We do have a test that suggests that's the case:

```
$ git grep 'Bad Gateway'
zerver/tests/test_push_notifications.py:                        f"ERROR:django.request:Bad Gateway: {endpoint}",
zerver/tests/test_push_notifications.py:                        f"ERROR:django.request:Bad Gateway: {endpoint}",
```

It should be possible to reproduce and debug in a Zulip development environment after registering for the push notifications service, though I don't think Django will try to actually send those error reporting emails in that setting.

Resolving patch:
diff --git a/zerver/lib/remote_server.py b/zerver/lib/remote_server.py
--- a/zerver/lib/remote_server.py
+++ b/zerver/lib/remote_server.py
@@ -161,7 +161,7 @@ def send_analytics_to_remote_server() -> None:
     try:
         result = send_to_push_bouncer("GET", "server/analytics/status", {})
     except PushNotificationBouncerRetryLaterError as e:
-        logging.warning(e.msg)
+        logging.warning(e.msg, exc_info=True)
         return
 
     last_acked_realm_count_id = result["last_realm_count_id"]
diff --git a/zerver/middleware.py b/zerver/middleware.py
--- a/zerver/middleware.py
+++ b/zerver/middleware.py
@@ -26,6 +26,7 @@
 from django.utils import translation
 from django.utils.cache import patch_vary_headers
 from django.utils.deprecation import MiddlewareMixin
+from django.utils.log import log_response
 from django.utils.translation import gettext as _
 from django.views.csrf import csrf_failure as html_csrf_failure
 from django_scim.middleware import SCIMAuthCheckMiddleware
@@ -456,8 +457,23 @@ def process_exception(
                 return json_unauthorized(www_authenticate="session")
 
         if isinstance(exception, JsonableError):
-            return json_response_from_error(exception)
-        if RequestNotes.get_notes(request).error_format == "JSON":
+            response = json_response_from_error(exception)
+            if response.status_code >= 500:
+                # Here we use Django's log_response the way Django uses
+                # it normally to log error responses. However, we make the small
+                # modification of including the traceback to make the log message
+                # more helpful. log_response takes care of knowing not to duplicate
+                # the logging, so Django won't generate a second log message.
+                log_response(
+                    "%s: %s",
+                    response.reason_phrase,
+                    request.path,
+                    response=response,
+                    request=request,
+                    exc_info=True,
+                )
+            return response
+        if RequestNotes.get_notes(request).error_format == "JSON" and not settings.TEST_SUITE:
             capture_exception(exception)
             json_error_logger = logging.getLogger("zerver.middleware.json_error_handler")
             json_error_logger.error(traceback.format_exc(), extra=dict(request=request))


Test patch:
diff --git a/zerver/lib/test_classes.py b/zerver/lib/test_classes.py
--- a/zerver/lib/test_classes.py
+++ b/zerver/lib/test_classes.py
@@ -1590,7 +1590,7 @@ def side_effect(*args: Any, **kwargs: Any) -> None:
                     complete_event_type is not None
                     and all_event_types is not None
                     and complete_event_type not in all_event_types
-                ):
+                ):  # nocoverage
                     raise Exception(
                         f"""
 Error: This test triggered a message using the event "{complete_event_type}", which was not properly
diff --git a/zerver/tests/test_integrations_dev_panel.py b/zerver/tests/test_integrations_dev_panel.py
--- a/zerver/tests/test_integrations_dev_panel.py
+++ b/zerver/tests/test_integrations_dev_panel.py
@@ -22,7 +22,7 @@ def test_check_send_webhook_fixture_message_for_error(self) -> None:
             "custom_headers": "{}",
             "is_json": "true",
         }
-        with self.assertLogs(level="ERROR") as logs:
+        with self.assertLogs(level="ERROR") as logs, self.settings(TEST_SUITE=False):
             response = self.client_post(target_url, data)
 
             self.assertEqual(response.status_code, 500)  # Since the response would be forwarded.
diff --git a/zerver/tests/test_logging_handlers.py b/zerver/tests/test_logging_handlers.py
--- a/zerver/tests/test_logging_handlers.py
+++ b/zerver/tests/test_logging_handlers.py
@@ -81,7 +81,9 @@ def simulate_error(self) -> logging.LogRecord:
             "django.request", level="ERROR"
         ) as request_error_log, self.assertLogs(
             "zerver.middleware.json_error_handler", level="ERROR"
-        ) as json_error_handler_log:
+        ) as json_error_handler_log, self.settings(
+            TEST_SUITE=False
+        ):
             rate_limit_patch.side_effect = capture_and_throw
             result = self.client_get("/json/users")
             self.assert_json_error(result, "Internal server error", status_code=500)
diff --git a/zerver/tests/test_push_notifications.py b/zerver/tests/test_push_notifications.py
--- a/zerver/tests/test_push_notifications.py
+++ b/zerver/tests/test_push_notifications.py
@@ -477,11 +477,9 @@ def test_push_bouncer_api(self) -> None:
                     "ConnectionError while trying to connect to push notification bouncer",
                     502,
                 )
-                self.assertEqual(
-                    error_log.output,
-                    [
-                        f"ERROR:django.request:Bad Gateway: {endpoint}",
-                    ],
+                self.assertIn(
+                    f"ERROR:django.request:Bad Gateway: {endpoint}\nTraceback",
+                    error_log.output[0],
                 )
 
             with responses.RequestsMock() as resp, self.assertLogs(level="WARNING") as warn_log:
@@ -489,11 +487,11 @@ def test_push_bouncer_api(self) -> None:
                 result = self.client_post(endpoint, {"token": token}, subdomain="zulip")
                 self.assert_json_error(result, "Received 500 from push notification bouncer", 502)
                 self.assertEqual(
-                    warn_log.output,
-                    [
-                        "WARNING:root:Received 500 from push notification bouncer",
-                        f"ERROR:django.request:Bad Gateway: {endpoint}",
-                    ],
+                    warn_log.output[0],
+                    "WARNING:root:Received 500 from push notification bouncer",
+                )
+                self.assertIn(
+                    f"ERROR:django.request:Bad Gateway: {endpoint}\nTraceback", warn_log.output[1]
                 )
 
         # Add tokens
@@ -571,13 +569,12 @@ def test_analytics_api(self) -> None:
         user = self.example_user("hamlet")
         end_time = self.TIME_ZERO
 
-        with responses.RequestsMock() as resp, mock.patch(
-            "zerver.lib.remote_server.logging.warning"
-        ) as mock_warning:
+        with responses.RequestsMock() as resp, self.assertLogs(level="WARNING") as mock_warning:
             resp.add(responses.GET, ANALYTICS_STATUS_URL, body=ConnectionError())
             send_analytics_to_remote_server()
-            mock_warning.assert_called_once_with(
-                "ConnectionError while trying to connect to push notification bouncer"
+            self.assertIn(
+                "WARNING:root:ConnectionError while trying to connect to push notification bouncer\nTraceback ",
+                mock_warning.output[0],
             )
             self.assertTrue(resp.assert_call_count(ANALYTICS_STATUS_URL, 1))
 
diff --git a/zerver/webhooks/travis/tests.py b/zerver/webhooks/travis/tests.py
--- a/zerver/webhooks/travis/tests.py
+++ b/zerver/webhooks/travis/tests.py
@@ -114,25 +114,6 @@ def test_travis_exclude_glob_events(self) -> None:
             expect_noop=True,
         )
 
-    def test_travis_invalid_event(self) -> None:
-        payload = self.get_body("build")
-        payload = payload.replace("push", "invalid_event")
-        expected_error_messsage = """
-Error: This test triggered a message using the event "invalid_event", which was not properly
-registered via the @webhook_view(..., event_types=[...]). These registrations are important for Zulip
-self-documenting the supported event types for this integration.
-
-You can fix this by adding "invalid_event" to ALL_EVENT_TYPES for this webhook.
-""".strip()
-        with self.assertLogs("django.request"):
-            with self.assertLogs("zerver.middleware.json_error_handler", level="ERROR") as m:
-                self.client_post(
-                    self.url,
-                    payload,
-                    content_type="application/x-www-form-urlencoded",
-                )
-            self.assertIn(expected_error_messsage, m.output[0])
-
     def test_travis_noop(self) -> None:
         expected_error_message = """
 While no message is expected given expect_noop=True,