Repository: netbox-community/netbox

Issue:
Allow Redis Connections to Use Sentinel
<!--
    NOTE: This form is only for proposing specific new features or enhancements.
    If you have a general idea or question, please post to our mailing list
    instead of opening an issue:

        https://groups.google.com/forum/#!forum/netbox-discuss

    NOTE: Due to an excessive backlog of feature requests, we are not currently
    accepting any proposals which significantly extend NetBox's feature scope.

    Please describe the environment in which you are running NetBox. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report.
-->
### Environment
* Python version:  3.7.3
* NetBox version:  2.7.2

<!--
    Describe in detail the new functionality you are proposing. Include any
    specific changes to work flows, data models, or the user interface.
-->
### Proposed Functionality

My proposal is to allow `SENTINELS` and `SERVICE` as optional keys to the `REDIS` dictionaries, like so. If these are both provided and non-empty, then the application would attempt to use the Redis Sentinel connection information. Note that the current way of configuring normal Redis would not change.

```python
# Using the current method 
REDIS = {
    'webhooks': {
        'HOST': 'webhooks_redis',
        'PORT': 6379,
        'PASSWORD': '',
        'DATABASE': 0,
        'DEFAULT_TIMEOUT': 300,
        'SSL': False,
    },
    'caching': {
        'HOST': 'cache_redis',
        'PORT': 6379,
        'PASSWORD': '',
        'DATABASE': 0,
        'DEFAULT_TIMEOUT': 300,
        'SSL': False,
    }
}

# Using a Sentinel instance
REDIS = {
    'webhooks': {
        'SENTINELS': (
            ('sentinel.redis.example.com', 1234),  # Each is defined as (HOST, PORT) tuple
        ),
        'SERVICE': 'webhooks_redis',                   # Sentinel service to connect to
        'PASSWORD': '',
        'DATABASE': 0,
        'DEFAULT_TIMEOUT': 300,
        'SSL': False,
    },
    'caching': {
        'SENTINELS': (
            ('sentinel.redis.example.com', 1234),  # Each is defined as (HOST, PORT) tuple
        ),
        'SERVICE': 'webhooks_redis',                   # Sentinel service to connect to
        'PASSWORD': '',
        'DATABASE': 0,
        'DEFAULT_TIMEOUT': 300,
        'SSL': False,
    }
}
```

<!--
    Convey an example use case for your proposed feature. Write from the
    perspective of a NetBox user who would benefit from the proposed
    functionality and describe how.
--->
### Use Case

It was nice that the webhooks and cache Redis connection settings were separated, but it would be ever nicer to be able to configure one (or both) to use Redis Sentinel connections for the sake of High Availability. Not only may this be preferred, but in some environments, it might be necessary due to the availability of developer environments.

<!--
    Note any changes to the database schema necessary to support the new
    feature. For example, does the proposal require adding a new model or
    field? (Not all new features require database changes.)
--->
### Database Changes
None.

<!--
    List any new dependencies on external libraries or services that this new
    feature would introduce. For example, does the proposal require the
    installation of a new Python package? (Not all new features introduce new
    dependencies.)
-->
### External Dependencies
None.


Discussion:
Both of django-cacheops and django-rq support redis sentinel. So I would accept this FR. Let me start to figure out how to add them to settings.py, then make PR when accepted by other maintainers.

`configuration.py` is the same as @dstarner described and default comment out

```python
# REDIS_SENTINEL = {
#     'webhooks': {
#         'SENTINELS': (
#             ('sentinel.redis.example.com', 1234),  # Each is defined as (HOST, PORT) tuple
#         ),
#         'MASTER': 'webhooks_redis',                   # Sentinel service to connect to
#         'PASSWORD': '',
#         'DATABASE': 0,
#         'SOCKET_TIMEOUT': 300,
#         'SSL': False,
#     },
#     'caching': {
#         'SENTINELS': (
#             ('sentinel.redis.example.com', 1234),  # Each is defined as (HOST, PORT) tuple
#         ),
#         'MASTER': 'webhooks_redis',                   # Sentinel service to connect to
#         'PASSWORD': '',
#         'DATABASE': 0,
#         'SOCKET_TIMEOUT': 300,
#         'SSL': False,
#     }
# }
```

`settings.py`
```python
REDIS_SENTINEL = getattr(configuration, 'REDIS_SENTINEL', None)
if REDIS_SENTINEL:
    WEBHOOK_SENTINEL = REDIS_SENTINEL.get('webhooks', None)
    WEBHOOK_SENTINEL_SENTINELS = WEBHOOK_SENTINEL.get('SENTINELS', None)
    if not WEBHOOK_SENTINEL_SENTINELS:
        raise ImproperlyConfigured(
            "REDIS_SENTINEL section in configuration.py is missing SENTINELS subsection."
        )
    WEBHOOK_SENTINEL_MASTER = WEBHOOK_SENTINEL.get('MASTER', None)
    if not WEBHOOK_SENTINEL_MASTER:
        raise ImproperlyConfigured(
            "REDIS_SENTINEL section in configuration.py is missing MASTER subsection of webhooks."
        )
    WEBHOOK_SENTINEL_DB = WEBHOOK_SENTINEL.get('DATABASE', 0)
    WEBHOOK_SENTINEL_PASSWORD = WEBHOOK_SENTINEL.get('PASSWORD', None)
    WEBHOOK_SENTINEL_SOCKET_TIMEOUT = WEBHOOK_SENTINEL.get('SOCKET_TIMEOUT', None)
    WEBHOOK_SENTINEL_SSL = WEBHOOK_SENTINEL.get('SSL', False)

    RQ_QUEUES.extend({
        'SENTINELS': WEBHOOK_SENTINEL_SENTINELS,
        'MASTER_NAME': WEBHOOK_SENTINEL_MASTER,
        'DB': WEBHOOK_SENTINEL_DB,
        'SOCKET_TIMEOUT': WEBHOOK_SENTINEL_SOCKET_TIMEOUT,
        'PASSWORD': WEBHOOK_SENTINEL_PASSWORD,
        'SSL': WEBHOOK_SENTINEL_SSL,
    })
    
    CACHING_SENTINEL = REDIS_SENTINEL.get('caching', None)
    if CACHING_SENTINEL:
        # CACHEOPS_REDIS needs to be None for cacheops to enable redis sentinel
        CACHEOPS_REDIS = None
        CACHING_SENTINEL_LOCATIONS = CACHING_SENTINEL.get('SENTINELS', None)
        if not CACHEOPS_SENTINEL_LOCATIONS:
            raise ImproperlyConfigured(
                "REDIS_SENTINEL section in configuration.py is missing SENTINELS subsection of cacheops.")
        CACHING_SENTINEL_SERVICE_NAME = CACHING_SENTINEL.get('MASTER', None)
        if not CACHEOPS_SENTINEL_SERVICE_NAME:
            raise ImproperlyConfigured(
                "REDIS_SENTINEL section in configuration.py is missing MASTER subsection of cacheops.")
        CACHING_SENTINEL_SOCKET_TIMEOUT = CACHING_SENTINEL.get('SOCKET_TIMEOUT', None)
        CACHING_SENTINEL_DB = CACHING_SENTINEL.get('DATABASE', 0)
        CACHING_SENTINEL_PASSWORD = CACHING_SENTINEL.get('PASSWORD', None)
        CACHING_SENTINEL_SSL = CACHING_SENTINEL.get('SSL', False)
        CACHEOPS_SENTINEL = {
            'locations': [CACHING_SENTINEL_LOCATIONS],
            'service_name': CACHING_SENTINEL_SERVICE_NAME,
            'db': CACHING_SENTINEL_DB,
            'socket_timeout': CACHING_SENTINEL_SOCKET_TIMEOUT,
            'password': CACHING_SENTINEL_PASSWORD,
            'ssl': CACHING_SENTINEL_SSL,
        }

```
@kobayashi I can probably turn my hack into the solution you have above. I already have something similar implemented, I would just need to clean it up and upstream it into the main project.

Resolving patch:
diff --git a/netbox/netbox/configuration.example.py b/netbox/netbox/configuration.example.py
--- a/netbox/netbox/configuration.example.py
+++ b/netbox/netbox/configuration.example.py
@@ -28,6 +28,9 @@
     'webhooks': {
         'HOST': 'localhost',
         'PORT': 6379,
+        # Comment out `HOST` and `PORT` lines and uncomment the following if using Redis Sentinel
+        # 'SENTINELS': [('mysentinel.redis.example.com', 6379)],
+        # 'SENTINEL_SERVICE': 'netbox',
         'PASSWORD': '',
         'DATABASE': 0,
         'DEFAULT_TIMEOUT': 300,
@@ -36,6 +39,9 @@
     'caching': {
         'HOST': 'localhost',
         'PORT': 6379,
+        # Comment out `HOST` and `PORT` lines and uncomment the following if using Redis Sentinel
+        # 'SENTINELS': [('mysentinel.redis.example.com', 6379)],
+        # 'SENTINEL_SERVICE': 'netbox',
         'PASSWORD': '',
         'DATABASE': 1,
         'DEFAULT_TIMEOUT': 300,
diff --git a/netbox/netbox/settings.py b/netbox/netbox/settings.py
--- a/netbox/netbox/settings.py
+++ b/netbox/netbox/settings.py
@@ -170,14 +170,27 @@ def _setting(name, default=None):
 WEBHOOKS_REDIS = REDIS.get('webhooks', {})
 WEBHOOKS_REDIS_HOST = WEBHOOKS_REDIS.get('HOST', 'localhost')
 WEBHOOKS_REDIS_PORT = WEBHOOKS_REDIS.get('PORT', 6379)
+WEBHOOKS_REDIS_SENTINELS = WEBHOOKS_REDIS.get('SENTINELS', [])
+WEBHOOKS_REDIS_USING_SENTINEL = all([
+    isinstance(WEBHOOKS_REDIS_SENTINELS, (list, tuple)),
+    len(WEBHOOKS_REDIS_SENTINELS) > 0
+])
+WEBHOOKS_REDIS_SENTINEL_SERVICE = WEBHOOKS_REDIS.get('SENTINEL_SERVICE', 'default')
 WEBHOOKS_REDIS_PASSWORD = WEBHOOKS_REDIS.get('PASSWORD', '')
 WEBHOOKS_REDIS_DATABASE = WEBHOOKS_REDIS.get('DATABASE', 0)
 WEBHOOKS_REDIS_DEFAULT_TIMEOUT = WEBHOOKS_REDIS.get('DEFAULT_TIMEOUT', 300)
 WEBHOOKS_REDIS_SSL = WEBHOOKS_REDIS.get('SSL', False)
 
+
 CACHING_REDIS = REDIS.get('caching', {})
 CACHING_REDIS_HOST = CACHING_REDIS.get('HOST', 'localhost')
 CACHING_REDIS_PORT = CACHING_REDIS.get('PORT', 6379)
+CACHING_REDIS_SENTINELS = CACHING_REDIS.get('SENTINELS', [])
+CACHING_REDIS_USING_SENTINEL = all([
+    isinstance(CACHING_REDIS_SENTINELS, (list, tuple)),
+    len(CACHING_REDIS_SENTINELS) > 0
+])
+CACHING_REDIS_SENTINEL_SERVICE = CACHING_REDIS.get('SENTINEL_SERVICE', 'default')
 CACHING_REDIS_PASSWORD = CACHING_REDIS.get('PASSWORD', '')
 CACHING_REDIS_DATABASE = CACHING_REDIS.get('DATABASE', 0)
 CACHING_REDIS_DEFAULT_TIMEOUT = CACHING_REDIS.get('DEFAULT_TIMEOUT', 300)
@@ -394,28 +407,35 @@ def _setting(name, default=None):
 #
 # Caching
 #
-
-if CACHING_REDIS_SSL:
-    REDIS_CACHE_CON_STRING = 'rediss://'
+if CACHING_REDIS_USING_SENTINEL:
+    CACHEOPS_SENTINEL = {
+        'locations': CACHING_REDIS_SENTINELS,
+        'service_name': CACHING_REDIS_SENTINEL_SERVICE,
+        'db': CACHING_REDIS_DATABASE,
+    }
 else:
-    REDIS_CACHE_CON_STRING = 'redis://'
-
-if CACHING_REDIS_PASSWORD:
-    REDIS_CACHE_CON_STRING = '{}:{}@'.format(REDIS_CACHE_CON_STRING, CACHING_REDIS_PASSWORD)
-
-REDIS_CACHE_CON_STRING = '{}{}:{}/{}'.format(
-    REDIS_CACHE_CON_STRING,
-    CACHING_REDIS_HOST,
-    CACHING_REDIS_PORT,
-    CACHING_REDIS_DATABASE
-)
+    if CACHING_REDIS_SSL:
+        REDIS_CACHE_CON_STRING = 'rediss://'
+    else:
+        REDIS_CACHE_CON_STRING = 'redis://'
+
+    if CACHING_REDIS_PASSWORD:
+        REDIS_CACHE_CON_STRING = '{}:{}@'.format(REDIS_CACHE_CON_STRING, CACHING_REDIS_PASSWORD)
+
+    REDIS_CACHE_CON_STRING = '{}{}:{}/{}'.format(
+        REDIS_CACHE_CON_STRING,
+        CACHING_REDIS_HOST,
+        CACHING_REDIS_PORT,
+        CACHING_REDIS_DATABASE
+    )
+    CACHEOPS_REDIS = REDIS_CACHE_CON_STRING
 
 if not CACHE_TIMEOUT:
     CACHEOPS_ENABLED = False
 else:
     CACHEOPS_ENABLED = True
 
-CACHEOPS_REDIS = REDIS_CACHE_CON_STRING
+
 CACHEOPS_DEFAULTS = {
     'timeout': CACHE_TIMEOUT
 }
@@ -534,6 +554,15 @@ def _setting(name, default=None):
         'PASSWORD': WEBHOOKS_REDIS_PASSWORD,
         'DEFAULT_TIMEOUT': WEBHOOKS_REDIS_DEFAULT_TIMEOUT,
         'SSL': WEBHOOKS_REDIS_SSL,
+    } if not WEBHOOKS_REDIS_USING_SENTINEL else {
+        'SENTINELS': WEBHOOKS_REDIS_SENTINELS,
+        'MASTER_NAME': WEBHOOKS_REDIS_SENTINEL_SERVICE,
+        'DB': WEBHOOKS_REDIS_DATABASE,
+        'PASSWORD': WEBHOOKS_REDIS_PASSWORD,
+        'SOCKET_TIMEOUT': None,
+        'CONNECTION_KWARGS': {
+            'socket_connect_timeout': WEBHOOKS_REDIS_DEFAULT_TIMEOUT
+        },
     }
 }
 


Test patch: