Repository: getredash/redash

Issue:
Rewrite Extensions registry not to depend on Flask app
- [ ] Rewrite Extensions registry not to depend on Flask app
- [ ] Bring back support for `add_periodic_tasks` (disabled in #3465)
Rewrite Extensions registry not to depend on Flask app
- [ ] Rewrite Extensions registry not to depend on Flask app
- [ ] Bring back support for `add_periodic_tasks` (disabled in #3465)


Discussion:
Please allow me to specify again why the backend extensions need the Flask app instance (at the moment) passed directly (and can't use `flask.current_app`):

- traditional Flask extensions require manual initialization of extension instances, Flask blueprints are not enough to cover the full life cycle of Flask apps
- our Docker based deployment workflow (contrary to e.g. a Python package based one) requires us to have a mechanism in place that extends Redash via separate dependencies (refs #2810) that are installed into the container
- to be able to extend the Redash backend system with own API endpoints, we're currently requiring the use of Flask-Restful's API endpoint registry)
- once we've moved to something else (#2981) the app instance is still needed to since the request handler api in anything based on Flask uses the app instance to store the handlers
- `flask.current_app` isn't set yet when extensions are loaded in `redash.create_app` since that's where the flask runtime looks for the app instance

Some ideas how to solve this:

- move the extension initialization into a signal handler for [Flask.before_first_request](http://flask.pocoo.org/docs/0.12/api/#flask.Flask.before_first_request) which is called [just before the first request is handled](https://github.com/pallets/flask/blob/a74864ec229141784374f1998324d2cbac837295/flask/app.py#L1826)
- simply rely on flask.current_app in extensions entrypoint callbacks
- move to using a Flask integration layer for initializing the Celery app, e.g. CERN's [flask-celeryext](https://github.com/inveniosoftware/flask-celeryext)

For some examples, feel free to check out https://github.com/mozilla/redash-stmo which contains a few extensions already, including examples that:

- extend a built-in query runner
- extend built-in API endpoints
- add new API endpoints
- bundle additional React components
- use a 3rd party Flask extension
- additional Celery tasks

I would be extremely happy if we were not to break any of those features 😬 But I'm also happy to work on this if you'd like and make sure the extension API becomes rock solid.
> Please allow me to specify again why the backend extensions need the Flask app instance (at the moment) passed directly (and can't use `flask.current_app`):

I should've elaborated more in the issue description... The idea here is not to prevent existing use cases, but to improve the code architecture. The registry not being coupled with Flask doesn't mean we can't use it with Flask, it just means we don't need Flask to make it work in Celery.

Currently `redash.extensions.init_extensions` both iterates on `iter_entry_points` and updates the Flask app object. Once it's done the only way to get the list of extensions is to query Flask's app. We just need to introduce some other object in the middle which we can query for extensions.
Please allow me to specify again why the backend extensions need the Flask app instance (at the moment) passed directly (and can't use `flask.current_app`):

- traditional Flask extensions require manual initialization of extension instances, Flask blueprints are not enough to cover the full life cycle of Flask apps
- our Docker based deployment workflow (contrary to e.g. a Python package based one) requires us to have a mechanism in place that extends Redash via separate dependencies (refs #2810) that are installed into the container
- to be able to extend the Redash backend system with own API endpoints, we're currently requiring the use of Flask-Restful's API endpoint registry)
- once we've moved to something else (#2981) the app instance is still needed to since the request handler api in anything based on Flask uses the app instance to store the handlers
- `flask.current_app` isn't set yet when extensions are loaded in `redash.create_app` since that's where the flask runtime looks for the app instance

Some ideas how to solve this:

- move the extension initialization into a signal handler for [Flask.before_first_request](http://flask.pocoo.org/docs/0.12/api/#flask.Flask.before_first_request) which is called [just before the first request is handled](https://github.com/pallets/flask/blob/a74864ec229141784374f1998324d2cbac837295/flask/app.py#L1826)
- simply rely on flask.current_app in extensions entrypoint callbacks
- move to using a Flask integration layer for initializing the Celery app, e.g. CERN's [flask-celeryext](https://github.com/inveniosoftware/flask-celeryext)

For some examples, feel free to check out https://github.com/mozilla/redash-stmo which contains a few extensions already, including examples that:

- extend a built-in query runner
- extend built-in API endpoints
- add new API endpoints
- bundle additional React components
- use a 3rd party Flask extension
- additional Celery tasks

I would be extremely happy if we were not to break any of those features 😬 But I'm also happy to work on this if you'd like and make sure the extension API becomes rock solid.
> Please allow me to specify again why the backend extensions need the Flask app instance (at the moment) passed directly (and can't use `flask.current_app`):

I should've elaborated more in the issue description... The idea here is not to prevent existing use cases, but to improve the code architecture. The registry not being coupled with Flask doesn't mean we can't use it with Flask, it just means we don't need Flask to make it work in Celery.

Currently `redash.extensions.init_extensions` both iterates on `iter_entry_points` and updates the Flask app object. Once it's done the only way to get the list of extensions is to query Flask's app. We just need to introduce some other object in the middle which we can query for extensions.

Resolving patch:
diff --git a/redash/extensions.py b/redash/extensions.py
--- a/redash/extensions.py
+++ b/redash/extensions.py
@@ -1,30 +1,103 @@
-import os
-from pkg_resources import iter_entry_points, resource_isdir, resource_listdir
+# -*- coding: utf-8 -*-
+import logging
+from collections import OrderedDict as odict
 
+from importlib_metadata import entry_points
 
-def init_app(app):
-    """
-    Load the Redash extensions for the given Redash Flask app.
-    """
-    if not hasattr(app, 'redash_extensions'):
-        app.redash_extensions = {}
+# The global Redash extension registry
+extensions = odict()
+
+# The periodic Celery tasks as provided by Redash extensions.
+# This is separate from the internal periodic Celery tasks in
+# celery_schedule since the extension task discovery phase is
+# after the configuration has already happened.
+periodic_tasks = odict()
+
+logger = logging.getLogger(__name__)
+
+
+def load_extensions(app):
+    """Load the Redash extensions for the given Redash Flask app.
 
-    for entry_point in iter_entry_points('redash.extensions'):
-        app.logger.info('Loading Redash extension %s.', entry_point.name)
+    The extension entry point can return any type of value but
+    must take a Flask application object.
+
+    E.g.::
+
+        def extension(app):
+            app.logger.info("Loading the Foobar extenions")
+            Foobar(app)
+
+    """
+    for entry_point in entry_points().get("redash.extensions", []):
+        app.logger.info('Loading Redash extension "%s".', entry_point.name)
         try:
-            extension = entry_point.load()
-            app.redash_extensions[entry_point.name] = {
-                "entry_function": extension(app),
-                "resources_list": []
+            # Then try to load the entry point (import and getattr)
+            obj = entry_point.load()
+        except (ImportError, AttributeError):
+            # or move on
+            app.logger.error(
+                'Redash extension "%s" could not be found.', entry_point.name
+            )
+            continue
+
+        if not callable(obj):
+            app.logger.error(
+                'Redash extension "%s" is not a callable.', entry_point.name
+            )
+            continue
+
+        # then simply call the loaded entry point.
+        extensions[entry_point.name] = obj(app)
+
+
+def load_periodic_tasks(logger):
+    """Load the periodic tasks as defined in Redash extensions.
+
+    The periodic task entry point needs to return a set of parameters
+    that can be passed to Celery's add_periodic_task:
+
+        https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#entries
+
+    E.g.::
+
+        def add_two_and_two():
+            return {
+                'name': 'add 2 and 2 every 10 seconds'
+                'sig': add.s(2, 2),
+                'schedule': 10.0,  # in seconds or a timedelta
             }
-        except ImportError:
-            app.logger.info('%s does not have a callable and will not be loaded.', entry_point.name)
-            (root_module, _) = os.path.splitext(entry_point.module_name)
-            content_folder_relative = os.path.join(entry_point.name, 'bundle')
-
-            # If it's a frontend extension only, store a list of files in the bundle directory.
-            if resource_isdir(root_module, content_folder_relative):
-                app.redash_extensions[entry_point.name] = {
-                    "entry_function": None,
-                    "resources_list": resource_listdir(root_module, content_folder_relative)
-                }
+
+    and then registered with an entry point under the "redash.periodic_tasks"
+    group, e.g. in your setup.py::
+
+        setup(
+            # ...
+            entry_points={
+                "redash.periodic_tasks": [
+                    "add_two_and_two = calculus.addition:add_two_and_two",
+                ]
+                # ...
+            },
+            # ...
+        )
+    """
+    for entry_point in entry_points().get("redash.periodic_tasks", []):
+        logger.info(
+            'Loading periodic Redash tasks "%s" from "%s".',
+            entry_point.name,
+            entry_point.value,
+        )
+        try:
+            periodic_tasks[entry_point.name] = entry_point.load()
+        except (ImportError, AttributeError):
+            # and move on if it couldn't load it
+            logger.error(
+                'Periodic Redash task "%s" could not be found at "%s".',
+                entry_point.name,
+                entry_point.value,
+            )
+
+
+def init_app(app):
+    load_extensions(app)
diff --git a/redash/worker.py b/redash/worker.py
--- a/redash/worker.py
+++ b/redash/worker.py
@@ -1,5 +1,4 @@
 from __future__ import absolute_import
-
 from datetime import timedelta
 from random import randint
 
@@ -8,15 +7,20 @@
 from celery import Celery
 from celery.schedules import crontab
 from celery.signals import worker_process_init
+from celery.utils.log import get_logger
 
-from redash import create_app, settings
+from redash import create_app, extensions, settings
 from redash.metrics import celery as celery_metrics  # noqa
 
 
+logger = get_logger(__name__)
+
+
 celery = Celery('redash',
                 broker=settings.CELERY_BROKER,
                 include='redash.tasks')
 
+# The internal periodic Celery tasks to automatically schedule.
 celery_schedule = {
     'refresh_queries': {
         'task': 'redash.tasks.refresh_queries',
@@ -69,18 +73,21 @@ def __call__(self, *args, **kwargs):
 celery.Task = ContextTask
 
 
-# Create Flask app after forking a new worker, to make sure no resources are shared between processes.
 @worker_process_init.connect
 def init_celery_flask_app(**kwargs):
+    """Create the Flask app after forking a new worker.
+
+    This is to make sure no resources are shared between processes.
+    """
     app = create_app()
     app.app_context().push()
 
 
-# Commented until https://github.com/getredash/redash/issues/3466 is implemented.
-# Hook for extensions to add periodic tasks.
-# @celery.on_after_configure.connect
-# def add_periodic_tasks(sender, **kwargs):
-#     app = safe_create_app()
-#     periodic_tasks = getattr(app, 'periodic_tasks', {})
-#     for params in periodic_tasks.values():
-#         sender.add_periodic_task(**params)
+@celery.on_after_configure.connect
+def add_periodic_tasks(sender, **kwargs):
+    """Load all periodic tasks from extensions and add them to Celery."""
+    # Populate the redash.extensions.periodic_tasks dictionary
+    extensions.load_periodic_tasks(logger)
+    for params in extensions.periodic_tasks.values():
+        # Add it to Celery's periodic task registry, too.
+        sender.add_periodic_task(**params)


Test patch: