Repository: openwisp/openwisp-utils (language: python)

Issue:
[feature] Add celery task class for regular background tasks with default soft and hard time limit set
We should look into whether it's possible to create a celery task class in which we can set the hard and soft time limit to make sure tasks never become stale and once we achieve that we have to base all celery tasks (except the long running tasks which must in any case define some kind of soft or hard time limit) of all openwisp modules on this class.


Discussion / hints:

Fix patch:
diff --git a/README.rst b/README.rst
index 63103550..f4d0e46e 100644
--- a/README.rst
+++ b/README.rst
@@ -998,7 +998,6 @@ If you want to print a string in ``Red Bold``, you can do it as below.
 
 You may also provide the ``end`` arguement similar to built-in print method.
 
-
 ``openwisp_utils.utils.SorrtedOrderedDict``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -1006,6 +1005,30 @@ Extends ``collections.SortedDict`` and implements logic to sort inserted
 items based on ``key`` value. Sorting is done at insert operation which
 incurs memory space overhead.
 
+``openwisp_utils.tasks.OpenwispCeleryTask``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A custom celery task class that sets hard and soft time limits of celery tasks
+using `OPENWISP_CELERY_HARD_TIME_LIMIT <#openwisp_celery_hard_time_limit>`_
+and `OPENWISP_CELERY_SOFT_TIME_LIMIT <#openwisp_celery_soft_time_limit>`_
+settings respectively.
+
+Usage:
+
+.. code-block:: python
+
+    from celery import shared_task
+
+    from openwisp_utils.tasks import OpenwispCeleryTask
+
+    @shared_task(base=OpenwispCeleryTask)
+    def your_celery_task():
+        pass
+
+**Note:** This task class should be used for regular background tasks
+but not for complex background tasks which can take a long time to execute
+(eg: firmware upgrades, network operations with retry mechanisms).
+
 Storage utilities
 -----------------
 
@@ -1606,6 +1629,30 @@ This setting allows to change the logo which is displayed in HTML version of the
 **Note**: Provide a URL which points to the logo on your own web server. Ensure that the URL provided is
 publicly accessible from the internet. Otherwise, the logo may not be displayed in the email.
 
+``OPENWISP_CELERY_SOFT_TIME_LIMIT``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++---------+---------------------+
+| type    | ``int``             |
++---------+---------------------+
+| default | ``30`` (in seconds) |
++---------+---------------------+
+
+Sets the soft time limit for celery tasks using
+`OpenwispCeleryTask <#openwisp_utilstasksopenwispcelerytask>`_.
+
+``OPENWISP_CELERY_HARD_TIME_LIMIT``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++---------+----------------------+
+| type    | ``int``              |
++---------+----------------------+
+| default | ``120`` (in seconds) |
++---------+----------------------+
+
+Sets the hard time limit for celery tasks using
+`OpenwispCeleryTask <#openwisp_utilstasksopenwispcelerytask>`_.
+
 Installing for development
 --------------------------
 
diff --git a/openwisp_utils/settings.py b/openwisp_utils/settings.py
index da9879ce..dbffeddc 100644
--- a/openwisp_utils/settings.py
+++ b/openwisp_utils/settings.py
@@ -13,3 +13,6 @@
         'description': 'OpenWISP REST API',
     },
 )
+
+CELERY_HARD_TIME_LIMIT = getattr(settings, 'OPENWISP_CELERY_HARD_TIME_LIMIT', 120)
+CELERY_SOFT_TIME_LIMIT = getattr(settings, 'OPENWISP_CELERY_SOFT_TIME_LIMIT', 30)
diff --git a/openwisp_utils/tasks.py b/openwisp_utils/tasks.py
new file mode 100644
index 00000000..1235ebae
--- /dev/null
+++ b/openwisp_utils/tasks.py
@@ -0,0 +1,8 @@
+from celery import Task
+
+from . import settings as app_settings
+
+
+class OpenwispCeleryTask(Task):
+    soft_time_limit = app_settings.CELERY_SOFT_TIME_LIMIT
+    time_limit = app_settings.CELERY_HARD_TIME_LIMIT
diff --git a/setup.py b/setup.py
index 453bfb9b..b3e633c2 100644
--- a/setup.py
+++ b/setup.py
@@ -59,6 +59,7 @@
             'django-filter>=2.2.0<2.4.0',
             'drf-yasg~=1.20.0',
         ],
+        'celery': ['celery~=5.2.3'],
     },
     classifiers=[
         'Development Status :: 3 - Alpha',


Test patch:
diff --git a/tests/test_project/tests/test_tasks.py b/tests/test_project/tests/test_tasks.py
new file mode 100644
index 00000000..fd46d4f5
--- /dev/null
+++ b/tests/test_project/tests/test_tasks.py
@@ -0,0 +1,14 @@
+from unittest import TestCase
+
+from celery import shared_task
+from openwisp_utils.tasks import OpenwispCeleryTask
+
+
+class TestOpenwispCeleryTask(TestCase):
+    def test_default_time_limits(self):
+        @shared_task(base=OpenwispCeleryTask)
+        def do_nothing():
+            pass
+
+        self.assertEqual(do_nothing.soft_time_limit, 30)
+        self.assertEqual(do_nothing.time_limit, 120)