Repository: celery/kombu

Issue:
How to best set max_retries for system? AKA How to implement circuit breaker pattern?
I'm interested in implementing a [circuit breaker design pattern](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern) with Celery in my codebase. This means that my application should be able to start if the broker is down and it should be able to raise an exception (within a reasonable window) when attempting to schedule a task. My setup is written so that there is a strong separation between task providers and task consumers (they are separate codebases). This issue is only about task providers, I accept that a task consumer should crash if it can't connect to a broker.

My task provider codebase is a Django server. I'm using SQS as a broker. It is configured to have a virtual topic exchange that schedules tasks to be the worker queues and a queue that's read from by a process responsible for recording scheduled tasks in a database. To ensure that the virtual exchange is registered at server startup, I run the following code:

```python
from django.apps import AppConfig


class TasksConfig(AppConfig):
    name = 'tasks'

    def ready(self):
        from .celery import app
        app.autodiscover_tasks(force=True)
        with app.producer_or_acquire() as P:
            # Ensure all queues are noticed
            for q in app.amqp.queues.values():
                P.maybe_declare(q)
```

This code goes through `kombu.connection.Channel.default_channel` which runs `self.ensure_connection()` with no `max_retries` specified:

```python
class Connection(object):
    # ...
    @property
    def default_channel(self):
        # make sure we're still connected, and if not refresh.
        self.ensure_connection()
        if self._default_channel is None:
            self._default_channel = self.channel()
        return self._default_channel
```

This allows `utils.functional.retry_over_time()` to retry indefinitely, causing the application to hang on startup. A very similar chain of actions occurs when calling `.delay()` on a task function. It feels like the simplest solution here is to provide a `max_retries` arguments.  However, **my question is "how should I best deliver this configuration to the channel?"**  I don't believe the `Channel` has access to the Celery app. It's feeling like providing it as a `transport_options` entry will be the simplest/cleanest method of configuration.  Any input on this before I make a PR to support this `transport_options` attribute?
How to best set max_retries for system? AKA How to implement circuit breaker pattern?
I'm interested in implementing a [circuit breaker design pattern](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern) with Celery in my codebase. This means that my application should be able to start if the broker is down and it should be able to raise an exception (within a reasonable window) when attempting to schedule a task. My setup is written so that there is a strong separation between task providers and task consumers (they are separate codebases). This issue is only about task providers, I accept that a task consumer should crash if it can't connect to a broker.

My task provider codebase is a Django server. I'm using SQS as a broker. It is configured to have a virtual topic exchange that schedules tasks to be the worker queues and a queue that's read from by a process responsible for recording scheduled tasks in a database. To ensure that the virtual exchange is registered at server startup, I run the following code:

```python
from django.apps import AppConfig


class TasksConfig(AppConfig):
    name = 'tasks'

    def ready(self):
        from .celery import app
        app.autodiscover_tasks(force=True)
        with app.producer_or_acquire() as P:
            # Ensure all queues are noticed
            for q in app.amqp.queues.values():
                P.maybe_declare(q)
```

This code goes through `kombu.connection.Channel.default_channel` which runs `self.ensure_connection()` with no `max_retries` specified:

```python
class Connection(object):
    # ...
    @property
    def default_channel(self):
        # make sure we're still connected, and if not refresh.
        self.ensure_connection()
        if self._default_channel is None:
            self._default_channel = self.channel()
        return self._default_channel
```

This allows `utils.functional.retry_over_time()` to retry indefinitely, causing the application to hang on startup. A very similar chain of actions occurs when calling `.delay()` on a task function. It feels like the simplest solution here is to provide a `max_retries` arguments.  However, **my question is "how should I best deliver this configuration to the channel?"**  I don't believe the `Channel` has access to the Celery app. It's feeling like providing it as a `transport_options` entry will be the simplest/cleanest method of configuration.  Any input on this before I make a PR to support this `transport_options` attribute?


Discussion:

Resolving patch:
diff --git a/kombu/connection.py b/kombu/connection.py
--- a/kombu/connection.py
+++ b/kombu/connection.py
@@ -815,8 +815,20 @@ def default_channel(self):
             a connection is passed instead of a channel, to functions that
             require a channel.
         """
+        conn_opts = {}
+        transport_opts = self.transport_options
+        if transport_opts:
+            if 'max_retries' in transport_opts:
+                conn_opts['max_retries'] = transport_opts['max_retries']
+            if 'interval_start' in transport_opts:
+                conn_opts['interval_start'] = transport_opts['interval_start']
+            if 'interval_step' in transport_opts:
+                conn_opts['interval_step'] = transport_opts['interval_step']
+            if 'interval_max' in transport_opts:
+                conn_opts['interval_max'] = transport_opts['interval_max']
+
         # make sure we're still connected, and if not refresh.
-        self.ensure_connection()
+        self.ensure_connection(**conn_opts)
         if self._default_channel is None:
             self._default_channel = self.channel()
         return self._default_channel


Test patch:
diff --git a/t/unit/test_connection.py b/t/unit/test_connection.py
--- a/t/unit/test_connection.py
+++ b/t/unit/test_connection.py
@@ -380,6 +380,32 @@ def test_revive_when_default_channel(self):
         defchan.close.assert_called_with()
         assert conn._default_channel is None
 
+    def test_default_channel_no_transport_options(self):
+        conn = self.conn
+        conn.ensure_connection = Mock()
+
+        assert conn.default_channel
+        conn.ensure_connection.assert_called_with()
+
+    def test_default_channel_transport_options(self):
+        conn = self.conn
+        conn.transport_options = options = {
+            'max_retries': 1,
+            'interval_start': 2,
+            'interval_step': 3,
+            'interval_max': 4,
+            'ignore_this': True
+        }
+        conn.ensure_connection = Mock()
+
+        assert conn.default_channel
+        conn.ensure_connection.assert_called_with(**{
+            k: v for k, v in options.items()
+            if k in ['max_retries',
+                     'interval_start',
+                     'interval_step',
+                     'interval_max']})
+
     def test_ensure_connection(self):
         assert self.conn.ensure_connection()