Repository: ckan/ckan

Issue:
Templates not overriden due to plugin ordering issues in 2.9
Tl;DR:

* `PluginImplementations` return plugins in the order that their class (extending `SingletonPlugin`) is first evaluated, not when the plugin is loaded
* This causes issues like the template folders always being loaded in a particular position
* Let's ignore how pyutilib returns the plugins and always enforce the order in the ini file

---

This is related to https://github.com/ckan/ckan/issues/5731 but a different issue. The initial symptom was that some templates from `plugin1` that I was overriding from `plugin2` never got applied regardless of the ordering of `plugin2` in `ckan.plugins`. Inspecting the `config['computed_template_paths']` you could tell that the templates from `plugin1` always were loaded first than the `plugin2`, regardless of the order of the plugins in the ini file.

As with all plugin ordering issues, this order eventually boils down to the order in which plugins are returned by the [PluginImplementations](https://github.com/ckan/ckan/blob/72180ea38ca70c3178379fe0a430a2cb0e492ac2/ckan/plugins/core.py#L74) iterator (in this particular case in the one called in [`environment.py`](https://github.com/ckan/ckan/blob/72180ea38ca70c3178379fe0a430a2cb0e492ac2/ckan/config/environment.py#L173) to call the `update_config()` hook where the template dirs are registered).

pyutilib uses an internal [`_id property`](https://github.com/PyUtilib/pyutilib/blob/d99406f2af1fb62268c34453a2fbe6bd4a7348f0/pyutilib/component/core/core.py#L292) to order the output list of plugins that implement a particular interface when using `PluginImplementations`. Our assumption has always been that this order was determined by the order in which plugins were [loaded](https://github.com/ckan/ckan/blob/72180ea38ca70c3178379fe0a430a2cb0e492ac2/ckan/plugins/core.py#L158) by CKAN, and in most cases tsat is true. But the actual time this internal `_id` property is set is when the [plugin class is initialized](https://github.com/PyUtilib/pyutilib/blob/master/pyutilib/component/core/core.py#L845), ie when we define a plugin class that implements `SingletonPlugin`, eg:

```python

import ckan.plugins as p


class MyPlugin1(p.SingletonPlugin):

    p.implements(ITemplateHelpers)

    def get_helpers(self):
        return {'plugin1_helper': plugin1_helper}


def plugin1_helper():
    pass

```

That is fine if plugins don't import modules from other plugins, but if they do, when evaluating the imports they can initialize the other plugin class, and so that one will be always loaded first.

For instance:

```python
import ckan.plugins as p

from ckanext.plugin1.plugin import plugin1_helper


class MyPlugin2(p.SingletonPlugin):
    
    #...
```

When loading `plugin2`, the imports will get processed and `MyPlugin1` will get evaluated, receiving a lower `_id` than `plugin1` and so always being returned first than `plugin1` regardless of the ordering in the ini file.
This particular case might seem like an edge case but note this is a simplified example, the imports could be in another module of `plugin2`, not necessarily `plugin.py`. The same ordering issue is happening eg when using `text_view`, which loads `resourceproxy` causing it to always come up first.

Considering how plugin ordering is and how difficult these issues are to track down, my suggestion to address this and future issues is that we enforce on the CKAN side that the order of plugins returned by `PluginImplementations` matches the ones defined in `ckan.plugins` (as in reordering the iterator returned by pyutilib based on the value of `ckan.plugins`):

```python

# on ckan/plugins/core.py

from pyutilib.component.core import ExtensionPoint


class PluginImplementations(ExtensionPoint):

    def __iter__(self):

        iterator = super(PluginImplementations, self).__iter__()

        plugins_found = list(iterator)

        plugins_in_config = config.get('ckan.plugins', '').split()

        # TODO reorder plugins_found based on plugins_in_config :)

        return plugins_found
```






Discussion:

Resolving patch:
diff --git a/ckan/lib/search/__init__.py b/ckan/lib/search/__init__.py
--- a/ckan/lib/search/__init__.py
+++ b/ckan/lib/search/__init__.py
@@ -9,6 +9,7 @@
 
 import requests
 
+from ckan.common import asbool, config
 import ckan.model as model
 import ckan.plugins as p
 import ckan.logic as logic
@@ -117,7 +118,8 @@ class SynchronousSearchPlugin(p.SingletonPlugin):
     p.implements(p.IDomainObjectModification, inherit=True)
 
     def notify(self, entity, operation):
-        if not isinstance(entity, model.Package):
+        if (not isinstance(entity, model.Package) or
+                not asbool(config.get('ckan.search.automatic_indexing', True))):
             return
         if operation != model.domain_object.DomainObjectOperation.deleted:
             dispatch_by_operation(
diff --git a/ckan/plugins/core.py b/ckan/plugins/core.py
--- a/ckan/plugins/core.py
+++ b/ckan/plugins/core.py
@@ -83,7 +83,23 @@ def __iter__(self):
 
         iterator = super(PluginImplementations, self).__iter__()
 
-        return reversed(list(iterator))
+        plugin_lookup = {pf.name: pf for pf in iterator}
+
+        plugins_in_config = (
+            config.get('ckan.plugins', '').split() + find_system_plugins())
+
+        ordered_plugins = []
+        for pc in plugins_in_config:
+            if pc in plugin_lookup:
+                ordered_plugins.append(plugin_lookup[pc])
+                plugin_lookup.pop(pc)
+
+        if plugin_lookup:
+            # Any oustanding plugin not in the ini file (ie system ones),
+            # add to the end of the iterator
+            ordered_plugins.extend(plugin_lookup.values())
+
+        return iter(ordered_plugins)
 
 
 class PluginNotFoundException(Exception):
@@ -145,12 +161,6 @@ def load_all():
     unload_all()
 
     plugins = config.get('ckan.plugins', '').split() + find_system_plugins()
-    # Add the synchronous search plugin, unless already loaded or
-    # explicitly disabled
-    if 'synchronous_search' not in plugins and \
-            asbool(config.get('ckan.search.automatic_indexing', True)):
-        log.debug('Loading the synchronous search plugin')
-        plugins.append('synchronous_search')
 
     load(*plugins)
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -68,16 +68,11 @@
         'solr = ckan.lib.search.solr_backend:SolrSearchBackend',
     ],
     'ckan.plugins': [
-        'synchronous_search = ckan.lib.search:SynchronousSearchPlugin',
         'stats = ckanext.stats.plugin:StatsPlugin',
-        'publisher_form = ckanext.publisher_form.forms:PublisherForm',
-        'publisher_dataset_form = ckanext.publisher_form.forms:PublisherDatasetForm',
         'multilingual_dataset = ckanext.multilingual.plugin:MultilingualDataset',
         'multilingual_group = ckanext.multilingual.plugin:MultilingualGroup',
         'multilingual_tag = ckanext.multilingual.plugin:MultilingualTag',
         'multilingual_resource = ckanext.multilingual.plugin:MultilingualResource',
-        'organizations = ckanext.organizations.forms:OrganizationForm',
-        'organizations_dataset = ckanext.organizations.forms:OrganizationDatasetForm',
         'expire_api_token = ckanext.expire_api_token.plugin:ExpireApiTokenPlugin',
         'chained_functions = ckanext.chained_functions.plugin:ChainedFunctionsPlugin',
         'datastore = ckanext.datastore.plugin:DatastorePlugin',
@@ -163,6 +158,7 @@
         'example_humanizer = ckanext.example_humanizer.plugin:ExampleHumanizerPlugin',
     ],
     'ckan.system_plugins': [
+        'synchronous_search = ckan.lib.search:SynchronousSearchPlugin',
         'domain_object_mods = ckan.model.modification:DomainObjectModificationExtension',
     ],
     'ckan.test_plugins': [


Test patch:
diff --git a/ckan/tests/plugins/test_core.py b/ckan/tests/plugins/test_core.py
--- a/ckan/tests/plugins/test_core.py
+++ b/ckan/tests/plugins/test_core.py
@@ -19,3 +19,19 @@ def test_plugins_order_in_pluginimplementations():
             u"example_idatasetform_v3"
         ]
     )
+
+
+@pytest.mark.usefixtures(u"with_plugins")
+@pytest.mark.ckan_config(
+    u"ckan.plugins",
+    u"example_idatasetform_v1 example_idatasetform_v3 example_idatasetform_v2")
+def test_plugins_order_in_pluginimplementations_matches_config():
+
+    assert (
+        [plugin.name for plugin in p.PluginImplementations(p.IDatasetForm)] ==
+        [
+            u"example_idatasetform_v1",
+            u"example_idatasetform_v3",
+            u"example_idatasetform_v2"
+        ]
+    )