Repository: getsentry/sentry-python

Issue:
Redis integration considered enabled even when Redis client not installed
### How do you use Sentry?

Sentry Saas (sentry.io)

### Version

1.10.1

### Steps to Reproduce

```pycon
D:\>py -3.11 -m venv .venv

D:\>cd .venv

D:\.venv>Scripts\activate.bat

(.venv) D:\.venv>pip install sentry-sdk
Collecting sentry-sdk
  Using cached sentry_sdk-1.10.1-py2.py3-none-any.whl (166 kB)
Collecting certifi
  Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting urllib3>=1.26.11
  Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Installing collected packages: urllib3, certifi, sentry-sdk
Successfully installed certifi-2022.9.24 sentry-sdk-1.10.1 urllib3-1.26.12

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip

(.venv) D:\.venv>python
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sentry_sdk
>>> sentry_sdk.init(debug=True)
 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration redis
 [sentry] DEBUG: Setting SDK name to 'sentry.python'
<sentry_sdk.hub._InitGuard object at 0x0000024182399E90>
>>> sentry_sdk.Hub.current.client.integrations
{'logging': <sentry_sdk.integrations.logging.LoggingIntegration object at 0x0000024181C3F1D0>, 'stdlib': <sentry_sdk.integrations.stdlib.StdlibIntegration object at 0x0000024181FA8E90>, 'excepthook': <sentry_sdk.integrations.excepthook.ExcepthookIntegration object at 0x00000241842D1D10>, 'dedupe': <sentry_sdk.integrations.dedupe.DedupeIntegration object at 0x00000241842D3990>, 'atexit': <sentry_sdk.integrations.atexit.AtexitIntegration object at 0x00000241842D3C10>, 'modules': <sentry_sdk.integrations.modules.ModulesIntegration object at 0x00000241842D3D90>, 'argv': <sentry_sdk.integrations.argv.ArgvIntegration object at 0x00000241842D3F50>, 'threading': <sentry_sdk.integrations.threading.ThreadingIntegration object at 0x00000241842EC710>, 'redis': <sentry_sdk.integrations.redis.RedisIntegration object at 0x0000024184376050>}
```

### Expected Result

Redis integration should not be considered enabled.

### Actual Result

Redis integration is considered enabled according to the logs:
```pycon
> [sentry] DEBUG: Setting up previously not enabled integration redis
> [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
…
> [sentry] DEBUG: Enabling integration redis
```
and the SDK integrations dictionary:
```pycon
>>> sentry_sdk.Hub.current.client.integrations
{'logging': <sentry_sdk.integrations.logging.LoggingIntegration object at 0x0000024181C3F1D0>, …, 'redis': <sentry_sdk.integrations.redis.RedisIntegration object at 0x0000024184376050>}
```

This seems to be because `setup_integrations` does not remove default integrations that raise `DidNotEnable` from the `integrations` dictionary:
https://github.com/getsentry/sentry-python/blob/a5ee1bd8c5b456704b9629fc430fb5203602f3c7/sentry_sdk/integrations/__init__.py#L91-L148

Other automatically enabled integrations don't encounter this issue because they are never added to the `integrations` dictionary, as they raise `DidNotEnable` on import, when iterating over `iter_default_integrations`, through `_generate_default_integrations_iterator.iter_default_integrations`:
https://github.com/getsentry/sentry-python/blob/a5ee1bd8c5b456704b9629fc430fb5203602f3c7/sentry_sdk/integrations/__init__.py#L73-L86 https://github.com/getsentry/sentry-python/blob/a5ee1bd8c5b456704b9629fc430fb5203602f3c7/sentry_sdk/integrations/__init__.py#L25-L51

Redis seems to be the only automatically enabled integration that imports and raises `DidNotEnable` on `ImportError` within `setup_once` rather than beforehand.

However, other automatically enabled integrations that raise `DidNotEnable` in `setup_once` for other reasons also encounter this issue, e.g. aiohttp with v3.3:
```pycon
D:\>py -3.7 -m venv .venv

D:\>cd .venv

D:\.venv>Scripts\activate.bat

(.venv) D:\.venv>pip install aiohttp==3.3.0 sentry-sdk
Collecting aiohttp==3.3.0
  Using cached aiohttp-3.3.0.tar.gz (722 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting sentry-sdk
  Using cached sentry_sdk-1.10.1-py2.py3-none-any.whl (166 kB)
Collecting async-timeout<4.0,>=3.0
  Using cached async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting chardet<4.0,>=2.0
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting attrs>=17.3.0
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.8.1-cp37-cp37m-win_amd64.whl (56 kB)
Collecting multidict<5.0,>=4.0
  Using cached multidict-4.7.6-cp37-cp37m-win_amd64.whl (48 kB)
Collecting certifi
  Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting urllib3>=1.26.11; python_version >= "3.6"
  Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting typing-extensions>=3.7.4; python_version < "3.8"
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting idna>=2.0
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Building wheels for collected packages: aiohttp
  Building wheel for aiohttp (PEP 517) ... done
  Created wheel for aiohttp: filename=aiohttp-3.3.0-py3-none-any.whl size=347624 sha256=caa4a9597eb6975a3441bb3f93b79eb875ad4461fcef8d6e1e71a8cd2255383a
  Stored in directory: c:\users\user\appdata\local\pip\cache\wheels\87\04\63\86f05b8df3510ffa9a983ae853c8756c2d1c5f76c63ceac9a4
Successfully built aiohttp
Installing collected packages: async-timeout, chardet, attrs, multidict, typing-extensions, idna, yarl, aiohttp, certifi, urllib3, sentry-sdk
Successfully installed aiohttp-3.3.0 async-timeout-3.0.1 attrs-22.1.0 certifi-2022.9.24 chardet-3.0.4 idna-3.4 multidict-4.7.6 sentry-sdk-1.10.1 typing-extensions-4.4.0 urllib3-1.26.12 yarl-1.8.1
WARNING: You are using pip version 20.1.1; however, version 22.3.1 is available.
You should consider upgrading via the 'd:\.venv\scripts\python.exe -m pip install --upgrade pip' command.

(.venv) D:\.venv>python
Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:58:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sentry_sdk
>>> sentry_sdk.init(debug=True)
 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration aiohttp
 [sentry] DEBUG: Did not enable default integration aiohttp: AIOHTTP 3.4 or newer required.
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration aiohttp
 [sentry] DEBUG: Enabling integration redis
 [sentry] DEBUG: Setting SDK name to 'sentry.python.aiohttp'
<sentry_sdk.hub._InitGuard object at 0x0000028F71A5DC08>
>>> sentry_sdk.Hub.current.client.integrations
{'logging': <sentry_sdk.integrations.logging.LoggingIntegration object at 0x0000028F71A5DC48>, 'stdlib': <sentry_sdk.integrations.stdlib.StdlibIntegration object at 0x0000028F71A70988>, 'excepthook': <sentry_sdk.integrations.excepthook.ExcepthookIntegration object at 0x0000028F71A709C8>, 'dedupe': <sentry_sdk.integrations.dedupe.DedupeIntegration object at 0x0000028F71A70A88>, 'atexit': <sentry_sdk.integrations.atexit.AtexitIntegration object at 0x0000028F71A70A48>, 'modules': <sentry_sdk.integrations.modules.ModulesIntegration object at 0x0000028F6F987A08>, 'argv': <sentry_sdk.integrations.argv.ArgvIntegration object at 0x0000028F71A83388>, 'threading': <sentry_sdk.integrations.threading.ThreadingIntegration object at 0x0000028F71A83648>, 'aiohttp': <sentry_sdk.integrations.aiohttp.AioHttpIntegration object at 0x0000028F71B2C7C8>, 'redis': <sentry_sdk.integrations.redis.RedisIntegration object at 0x0000028F71A9A788>}
```

Also of note, the only relevant test for this, `test_auto_enabling_integrations_catches_import_error`, ignores the Redis integration, seemingly with the reasoning that the redis package is always installed as a dependency for running tests. As far as I can tell, this isn't the case now and wasn't even the case when the test was added. More likely, the test fails (and failed) because the test checks for a log message starting with "Did not import default integration ", whereas the Redis integration will successfully import, but log "Did not enable default integration " later (even though it's still considered enabled).
https://github.com/getsentry/sentry-python/blob/a5ee1bd8c5b456704b9629fc430fb5203602f3c7/tests/test_basics.py#L52-L69


Discussion:
Hey @Harmon758 

Thanks for reporting this! Good catch! I have put this on our internal backlog so we will fix this. (But please do not hold your breath on it, because we have a lot on our plates right now)

Resolving patch:
diff --git a/sentry_sdk/integrations/__init__.py b/sentry_sdk/integrations/__init__.py
--- a/sentry_sdk/integrations/__init__.py
+++ b/sentry_sdk/integrations/__init__.py
@@ -16,6 +16,11 @@
 
 
 _installer_lock = Lock()
+
+# Set of all integration identifiers we have attempted to install
+_processed_integrations = set()  # type: Set[str]
+
+# Set of all integration identifiers we have actually installed
 _installed_integrations = set()  # type: Set[str]
 
 
@@ -121,7 +126,7 @@ def setup_integrations(
 
     for identifier, integration in iteritems(integrations):
         with _installer_lock:
-            if identifier not in _installed_integrations:
+            if identifier not in _processed_integrations:
                 logger.debug(
                     "Setting up previously not enabled integration %s", identifier
                 )
@@ -144,8 +149,16 @@ def setup_integrations(
                     logger.debug(
                         "Did not enable default integration %s: %s", identifier, e
                     )
+                else:
+                    _installed_integrations.add(identifier)
+
+                _processed_integrations.add(identifier)
 
-                _installed_integrations.add(identifier)
+    integrations = {
+        identifier: integration
+        for identifier, integration in iteritems(integrations)
+        if identifier in _installed_integrations
+    }
 
     for identifier in integrations:
         logger.debug("Enabling integration %s", identifier)


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -30,7 +30,7 @@
 import sentry_sdk
 from sentry_sdk._compat import iteritems, reraise, string_types
 from sentry_sdk.envelope import Envelope
-from sentry_sdk.integrations import _installed_integrations  # noqa: F401
+from sentry_sdk.integrations import _processed_integrations  # noqa: F401
 from sentry_sdk.profiler import teardown_profiler
 from sentry_sdk.transport import Transport
 from sentry_sdk.utils import capture_internal_exceptions
@@ -187,8 +187,8 @@ def reset_integrations():
     with a clean slate to ensure monkeypatching works well,
     but this also means some other stuff will be monkeypatched twice.
     """
-    global _installed_integrations
-    _installed_integrations.clear()
+    global _processed_integrations
+    _processed_integrations.clear()
 
 
 @pytest.fixture
diff --git a/tests/test_basics.py b/tests/test_basics.py
--- a/tests/test_basics.py
+++ b/tests/test_basics.py
@@ -18,8 +18,13 @@
     Hub,
 )
 from sentry_sdk._compat import reraise
-from sentry_sdk.integrations import _AUTO_ENABLING_INTEGRATIONS
+from sentry_sdk.integrations import (
+    _AUTO_ENABLING_INTEGRATIONS,
+    Integration,
+    setup_integrations,
+)
 from sentry_sdk.integrations.logging import LoggingIntegration
+from sentry_sdk.integrations.redis import RedisIntegration
 from sentry_sdk.scope import (  # noqa: F401
     add_global_event_processor,
     global_event_processors,
@@ -28,6 +33,36 @@
 from sentry_sdk.tracing_utils import has_tracing_enabled
 
 
+def _redis_installed():  # type: () -> bool
+    """
+    Determines whether Redis is installed.
+    """
+    try:
+        import redis  # noqa: F401
+    except ImportError:
+        return False
+
+    return True
+
+
+class NoOpIntegration(Integration):
+    """
+    A simple no-op integration for testing purposes.
+    """
+
+    identifier = "noop"
+
+    @staticmethod
+    def setup_once():  # type: () -> None
+        pass
+
+    def __eq__(self, __value):  # type: (object) -> bool
+        """
+        All instances of NoOpIntegration should be considered equal to each other.
+        """
+        return type(__value) == type(self)
+
+
 def test_processors(sentry_init, capture_events):
     sentry_init()
     events = capture_events()
@@ -59,8 +94,8 @@ def test_auto_enabling_integrations_catches_import_error(sentry_init, caplog):
     sentry_init(auto_enabling_integrations=True, debug=True)
 
     for import_string in _AUTO_ENABLING_INTEGRATIONS:
-        # Ignore redis in the test case, because it is installed as a
-        # dependency for running tests, and therefore always enabled.
+        # Ignore redis in the test case, because it does not raise a DidNotEnable
+        # exception on import; rather, it raises the exception upon enabling.
         if _AUTO_ENABLING_INTEGRATIONS[redis_index] == import_string:
             continue
 
@@ -686,3 +721,18 @@ def test_functions_to_trace_with_class(sentry_init, capture_events):
     assert len(event["spans"]) == 2
     assert event["spans"][0]["description"] == "tests.test_basics.WorldGreeter.greet"
     assert event["spans"][1]["description"] == "tests.test_basics.WorldGreeter.greet"
+
+
+@pytest.mark.skipif(_redis_installed(), reason="skipping because redis is installed")
+def test_redis_disabled_when_not_installed(sentry_init):
+    sentry_init()
+
+    assert Hub.current.get_integration(RedisIntegration) is None
+
+
+def test_multiple_setup_integrations_calls():
+    first_call_return = setup_integrations([NoOpIntegration()], with_defaults=False)
+    assert first_call_return == {NoOpIntegration.identifier: NoOpIntegration()}
+
+    second_call_return = setup_integrations([NoOpIntegration()], with_defaults=False)
+    assert second_call_return == {NoOpIntegration.identifier: NoOpIntegration()}