Repository: zulip/zulip

Issue:
obscure error messages when zulip push notification gateway is unreachable
our zulip server is hosted in a dmz .

accidentally,  the firewall rule for allowing outgoing connection to push.zulipchat.com was not active.  we block all outgoing connections by default, as we do not want to be some "hop" or source of spam and illegal action, if some machine is being hacked inside the dmz.

we did not take any notice on this,and users didn't complain because push notifications are not that important to them.

but we started to wonder, why we got weird and very intermittend notification mails from django module like this

```
Subject: [Django] chat:Bad Gateway:/api/v1/users/me/apns_device_token

Logger django.request, from module django.utils.log line 230:
Error generated by <**username**> <**user email**> (Member) on <**servername**> deployment

No stack trace available

Deployed code:
- git: None
- ZULIP_VERSION: 4.3


Request info:
- path: /api/v1/users/me/apns_device_token
- POST: {'token': ['<**token string**>'], 'appid': ['org.zulip.Zulip']}
- REMOTE_ADDR: "<**IP**>"
- QUERY_STRING: ""
- SERVER_NAME: ""
```

in server log, there is proper logging of gateway unreachablility :
```
2021-08-17 00:14:59.285 WARN [] ConnectionError while trying to connect to push notification bouncer
2021-08-17 01:06:19.380 WARN [] ConnectionError while trying to connect to push notification bouncer
2021-08-17 02:05:03.906 WARN [] ConnectionError while trying to connect to push notification bouncer
```

it looks there is some room for improvement:
1. don't send weird django error message
2. send proper notification to the admin or/and to the end-user, if push notification message could not be sent

also see https://chat.zulip.org/#narrow/stream/31-production-help/topic/.5BDjango.5D.20chat.3ABad.20Gateway.3A.2Fapi.2Fv1.2Fusers.2Fme.2Fapns_device_token/near/1246328


Discussion:
Hello @zulip/server-api, @zulip/server-production members, this issue was labeled with the "area: production", "area: api" labels, so you may want to check it out!

<!-- areaLabelAddition -->
That error email does indeed seem pretty messy for what is fundamentally a networking issue connecting to the [mobile push notifications service](https://zulip.readthedocs.io/en/latest/production/mobile-push-notifications.html). The code path is here:

```
$ git grep 'while trying to connect'
zerver/lib/remote_server.py:            f"{e.__class__.__name__} while trying to connect to push notification bouncer"
zerver/tests/test_push_notifications.py:                    "ConnectionError while trying to connect to push notification bouncer",
zerver/tests/test_push_notifications.py:                "ConnectionError while trying to connect to push notification bouncer"
```

And my guess is that we're somewhere raising an exception or doing a `logging.error` type call without passing a proper stack trace. We do have a test that suggests that's the case:

```
$ git grep 'Bad Gateway'
zerver/tests/test_push_notifications.py:                        f"ERROR:django.request:Bad Gateway: {endpoint}",
zerver/tests/test_push_notifications.py:                        f"ERROR:django.request:Bad Gateway: {endpoint}",
```

It should be possible to reproduce and debug in a Zulip development environment after registering for the push notifications service, though I don't think Django will try to actually send those error reporting emails in that setting.
Any update on this error? We are having the same problem, only difference is we are running on Zulip 6.0 instead. 


As noted above, the error itself is a networking problem -- you likely have a firewall preventing your server from connecting to the push notifications service.

I'll try to make sure improving the traceback happens to provide more insight into the networking issue happens; it should be easy to do and backport.

Resolving patch:
diff --git a/zerver/lib/remote_server.py b/zerver/lib/remote_server.py
--- a/zerver/lib/remote_server.py
+++ b/zerver/lib/remote_server.py
@@ -161,7 +161,7 @@ def send_analytics_to_remote_server() -> None:
     try:
         result = send_to_push_bouncer("GET", "server/analytics/status", {})
     except PushNotificationBouncerRetryLaterError as e:
-        logging.warning(e.msg)
+        logging.warning(e.msg, exc_info=True)
         return
 
     last_acked_realm_count_id = result["last_realm_count_id"]
diff --git a/zerver/middleware.py b/zerver/middleware.py
--- a/zerver/middleware.py
+++ b/zerver/middleware.py
@@ -16,6 +16,7 @@
 from django.utils import translation
 from django.utils.cache import patch_vary_headers
 from django.utils.deprecation import MiddlewareMixin
+from django.utils.log import log_response
 from django.utils.translation import gettext as _
 from django.views.csrf import csrf_failure as html_csrf_failure
 from django_scim.middleware import SCIMAuthCheckMiddleware
@@ -465,7 +466,22 @@ def process_exception(
                 return json_unauthorized(www_authenticate="session")
 
         if isinstance(exception, JsonableError):
-            return json_response_from_error(exception)
+            response = json_response_from_error(exception)
+            if response.status_code >= 500:
+                # Here we use Django's log_response the way Django uses
+                # it normally to log error responses. However, we make the small
+                # modification of including the traceback to make the log message
+                # more helpful. log_response takes care of knowing not to duplicate
+                # the logging, so Django won't generate a second log message.
+                log_response(
+                    "%s: %s",
+                    response.reason_phrase,
+                    request.path,
+                    response=response,
+                    request=request,
+                    exc_info=True,
+                )
+            return response
         if RequestNotes.get_notes(request).error_format == "JSON" and not settings.TEST_SUITE:
             capture_exception(exception)
             json_error_logger = logging.getLogger("zerver.middleware.json_error_handler")


Test patch:
diff --git a/zerver/tests/test_push_notifications.py b/zerver/tests/test_push_notifications.py
--- a/zerver/tests/test_push_notifications.py
+++ b/zerver/tests/test_push_notifications.py
@@ -460,11 +460,9 @@ def test_push_bouncer_api(self) -> None:
                     "ConnectionError while trying to connect to push notification bouncer",
                     502,
                 )
-                self.assertEqual(
-                    error_log.output,
-                    [
-                        f"ERROR:django.request:Bad Gateway: {endpoint}",
-                    ],
+                self.assertIn(
+                    f"ERROR:django.request:Bad Gateway: {endpoint}\nTraceback",
+                    error_log.output[0],
                 )
 
             with responses.RequestsMock() as resp, self.assertLogs(level="WARNING") as warn_log:
@@ -472,11 +470,11 @@ def test_push_bouncer_api(self) -> None:
                 result = self.client_post(endpoint, {"token": token}, subdomain="zulip")
                 self.assert_json_error(result, "Received 500 from push notification bouncer", 502)
                 self.assertEqual(
-                    warn_log.output,
-                    [
-                        "WARNING:root:Received 500 from push notification bouncer",
-                        f"ERROR:django.request:Bad Gateway: {endpoint}",
-                    ],
+                    warn_log.output[0],
+                    "WARNING:root:Received 500 from push notification bouncer",
+                )
+                self.assertIn(
+                    f"ERROR:django.request:Bad Gateway: {endpoint}\nTraceback", warn_log.output[1]
                 )
 
         # Add tokens
@@ -552,13 +550,12 @@ def test_analytics_api(self) -> None:
         user = self.example_user("hamlet")
         end_time = self.TIME_ZERO
 
-        with responses.RequestsMock() as resp, mock.patch(
-            "zerver.lib.remote_server.logging.warning"
-        ) as mock_warning:
+        with responses.RequestsMock() as resp, self.assertLogs(level="WARNING") as mock_warning:
             resp.add(responses.GET, ANALYTICS_STATUS_URL, body=ConnectionError())
             send_analytics_to_remote_server()
-            mock_warning.assert_called_once_with(
-                "ConnectionError while trying to connect to push notification bouncer"
+            self.assertIn(
+                "WARNING:root:ConnectionError while trying to connect to push notification bouncer\nTraceback ",
+                mock_warning.output[0],
             )
             self.assertTrue(resp.assert_call_count(ANALYTICS_STATUS_URL, 1))