Repository: elastic/apm-agent-python

Issue:
Django 2.2 uncaught exceptions not being sent
**Describe the bug**: ...
I am not sure if it is a bug, because reading the documentation I was not able to find any explicit information talking about uncaught exceptions with Django (only with Flask), but all non handled exceptions are not being sent to APM server. However, running the test command (`python manage.py elasticapm test`) the event is sent successfully 

**Environment (please complete the following information)**
- OS: Linux (Docker image python:3.7.4)
- Python version: 3.7.4
- Framework and version [e.g. Django 2.1]: Django 2.2
- APM Server version: v7.5.0
- Agent version: 5.3.2

**How to reproduce**
In order to tests the uncaught exceptions I did the following tests:
* Manually inserted an `raise ZeroDivisionError('APM TEST')` in one of my functions outside any `try/catch/`
* Executed `raise ZeroDivisionError('APM TEST')` using python shell (`python manage.py shell`)


**Additional context**

`settings.py`

```
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.contenttypes',
    'django.contrib.staticfiles',
    'elasticapm.contrib.django',
    ...
]


MIDDLEWARE = [
    'elasticapm.contrib.django.middleware.TracingMiddleware',
    'elasticapm.contrib.django.middleware.Catch404Middleware',
    ...
]


ELASTIC_APM = {
    'SERVICE_NAME': 'MYSERVICE',
    'SECRET_TOKEN': 'SECRETTOKEN,
    'SERVER_URL': 'SERVERURL',  # noqa
    'DEBUG': True
}

```

*Running check command*
```
root@26cb7a8c0615:/usr/src/app# python manage.py elasticapm check
Service name is set, good job!

SERVER_URL https://APM_ADDR:443 looks fine

DEBUG mode is disabled! Looking good!

Tracing middleware is configured! Awesome!

Looks like everything should be ready!
```


Discussion:
Thanks for the report, @romulo-dbts! I definitely think we should be reporting those exceptions. We probably won't get a chance to look into this until next year, as we're taking some time off for the holidays.
Thanks @basepi. Today I performed more tests and so far my impressions is that only exceptions that occurs inside a scope of a HTTP request are being tracked, but if you have any other exception being thrown inside regular functions the middleware is not able to capture and send them.
That may well be the case. The middleware gives us hooks into requests, I'm not sure that it goes further than that. @beniwohli may know better than I.
Alright, I chatted with @beniwohli and have a better idea of what the situation is here:

Currently, we only [listen to the `got_request_exception` signal](https://github.com/elastic/apm-agent-python/blob/master/elasticapm/contrib/django/apps.py#L103-L105), which means we only capture errors that happen during requests.

We have [an open PR](https://github.com/elastic/apm-agent-python/pull/644) which adds support for Django management commands, but we explicitly excluded `shell` because you can have long-running jobs there.

It's possible we could make ourselves a generic exception handler by overriding [`sys.excepthook`](https://docs.python.org/3/library/sys.html#sys.excepthook), but this is much less clean than just listening for signals, as we could potentially interfere with a user's own use of this function. I also haven't tested to be sure that Django doesn't catch "uncaught" exceptions before they hit that function, which could add a wrinkle.

tl;dr: This is currently expected behavior, but we're open to solutions in this space.
Hi @basepi,

You have raise some good points and we can look into it further, but to be honest at first glance I think that overriding `sys.excepthook` is a good way to go. Regarding the concern of an user has already overwritten `sys.excepthook` we can write a code which implement some policies in order to give the user the possibility of set some precedence and respect its own implementation. 
I wrote a simple snippet to illustrate what I am talking about. In this example the default policy is to execute first the user's `exceptionhook` and after that execute what would be Elastic version of this function.

```
class ElasticExceptionHandler:

    def __init__(self):
        self.original_handler = sys.excepthook
        self.overridden_by_user = not self.original_handler is sys.__excepthook__

    def __call__(self):
        sys.last_type, sys.last_value, last_traceback = exec_info = sys.exc_info()
        sys.last_traceback = last_traceback

        if self.overridden_by_user:
            # if user has overridden sys.excepthook it should take precedence over Elastic
            print('Default excepthook was overridden')
            self.original_handler(exec_info[0], exec_info[1], last_traceback)

        self.elastic_handler()


    def elastic_handler(self):
        print('Calling Elastic Handler')
``` 

Execution using iPython:
iPython is a good example for us because by default it contains a custom version of `sys.excepthook`

```
In [1]: import sys                                                                                   

In [2]: class ElasticExceptionHandler: 
   ...:  
   ...:     def __init__(self): 
   ...:         self.original_handler = sys.excepthook 
   ...:         self.overridden_by_user = not self.original_handler is sys.__excepthook__ 
   ...:  
   ...:     def __call__(self): 
   ...:         sys.last_type, sys.last_value, last_traceback = exec_info = sys.exc_info() 
   ...:         sys.last_traceback = last_traceback 
   ...:  
   ...:         if self.overridden_by_user: 
   ...:             # if user has overridden sys.excepthook it should take precedence over Elastic 
   ...:             print('Default excepthook was overridden') 
   ...:             self.original_handler(exec_info[0], exec_info[1], last_traceback) 
   ...:  
   ...:         self.elastic_handler() 
   ...:  
   ...:  
   ...:     def elastic_handler(self): 
   ...:         print('Calling Elastic Handler') 
   ...:                                                                                              

In [3]: elastic_handler = ElasticExceptionHandler()                                                  

In [4]: elastic_handler()                                                                            
Default excepthook was overridden
No traceback available to show.
Calling Elastic Handler

In [5]: sys.excepthook                                                                               
Out[5]: <bound method InteractiveShell.excepthook of <IPython.terminal.interactiveshell.TerminalInteractiveShell object at 0x7f51017f35f8>>
```
@romulo-dbts I like this solution a lot! I'd be happy to help you through submitting a PR, otherwise we'll add this to our list! Thank you for your work here.
@basepi Is there any documentation about how to setup the project and contribute?
We have a [contributing document](https://github.com/elastic/apm-agent-python/blob/master/CONTRIBUTING.md) but it hasn't had much attention recently and can definitely be improved.

The django middleware and all of our integration code is here: https://github.com/elastic/apm-agent-python/tree/master/elasticapm/contrib/django

I envision adding this `sys.excepthook` override as an off-by-default [config option](https://github.com/elastic/apm-agent-python/blob/9360c58b43021dad7eceeacb7c682794955444b5/elasticapm/conf/__init__.py#L253) which, if set to True, would decorate the `sys.excepthook` much in the way that you described.

If you install the [django test requirements](https://github.com/elastic/apm-agent-python/blob/master/tests/requirements/requirements-django-3.0.txt) then you should be able to run all the django tests with pytest.

I also created a new issue for the lack of good contributing documentation: https://github.com/elastic/apm-agent-python/issues/769

Resolving patch:
diff --git a/elasticapm/base.py b/elasticapm/base.py
--- a/elasticapm/base.py
+++ b/elasticapm/base.py
@@ -197,6 +197,9 @@ def __init__(self, config=None, **inline):
             self._thread_managers["config"] = self.config
         else:
             self._config_updater = None
+        if self.config.use_elastic_excepthook:
+            self.original_excepthook = sys.excepthook
+            sys.excepthook = self._excepthook
         if config.enabled:
             self.start_threads()
 
@@ -559,6 +562,15 @@ def _get_stack_info_for_trace(
             locals_processor_func=locals_processor_func,
         )
 
+    def _excepthook(self):
+        exec_info = sys.exc_info()
+        try:
+            self.original_excepthook(*exec_info)
+        except Exception:
+            self.capture_exception(handled=False)
+        finally:
+            self.capture_exception(exec_info, handled=False)
+
     def load_processors(self):
         """
         Loads processors from self.config.processors, as well as constants.HARDCODED_PROCESSORS.
diff --git a/elasticapm/conf/__init__.py b/elasticapm/conf/__init__.py
--- a/elasticapm/conf/__init__.py
+++ b/elasticapm/conf/__init__.py
@@ -468,6 +468,7 @@ class Config(_ConfigBase):
     django_transaction_name_from_route = _BoolConfigValue("DJANGO_TRANSACTION_NAME_FROM_ROUTE", default=False)
     disable_log_record_factory = _BoolConfigValue("DISABLE_LOG_RECORD_FACTORY", default=False)
     use_elastic_traceparent_header = _BoolConfigValue("USE_ELASTIC_TRACEPARENT_HEADER", default=True)
+    use_elastic_excepthook = _BoolConfigValue("USE_ELASTIC_EXCEPTHOOK", default=False)
     cloud_provider = _ConfigValue("CLOUD_PROVIDER", default=True)
 
     @property


Test patch:
diff --git a/tests/fixtures.py b/tests/fixtures.py
--- a/tests/fixtures.py
+++ b/tests/fixtures.py
@@ -34,11 +34,13 @@
 import os
 import random
 import socket
+import sys
 import time
 import zlib
 from collections import defaultdict
 
 import jsonschema
+import mock
 import pytest
 from pytest_localserver.http import ContentServer
 from werkzeug.wrappers import Request, Response
@@ -146,8 +148,31 @@ def __call__(self, environ, start_response):
         return response(environ, start_response)
 
 
+@pytest.fixture
+def mock_client_excepthook():
+    with mock.patch("tests.fixtures.TempStoreClient._excepthook") as m:
+        yield m
+
+
+@pytest.fixture
+def mock_client_capture_exception():
+    with mock.patch("tests.fixtures.TempStoreClient.capture_exception") as m:
+        yield m
+
+
+@pytest.fixture
+def original_exception_hook(request):
+    mock_params = getattr(request, "param", {})
+    original_excepthook = sys.excepthook
+    mck = mock.Mock(side_effect=mock_params.get("side_effect"))
+    sys.excepthook = mck
+    yield mck
+    sys.excepthook = original_excepthook
+
+
 @pytest.fixture()
 def elasticapm_client(request):
+    original_exceptionhook = sys.excepthook
     client_config = getattr(request, "param", {})
     client_config.setdefault("service_name", "myapp")
     client_config.setdefault("secret_token", "test_key")
@@ -160,6 +185,7 @@ def elasticapm_client(request):
     yield client
     client.close()
     # clear any execution context that might linger around
+    sys.excepthook = original_exceptionhook
     execution_context.set_transaction(None)
     execution_context.set_span(None)