Repository: ckan/ckan

Issue:
Pytest ckan_config does not override configurations from test.ini
## CKAN version
2.9, master

## Describe the bug
By configuring some variable in test.ini like `ckan.plugins` reduces duplication in tests themselves. However if test requires some other variable like different plugins, using `ckan_config` decorator does not override the variable from test.ini

### Steps to reproduce
Create tests with ckan.plugins set to something in test.ini
Implement a test with decorator:
@pytest.mark.ckan_config('ckan.plugins', 'some_plugin')

### Expected behavior
Test should use plugin given in decorator and not in test.ini





Discussion:
Example of this issue can be found here:

Plugins configured in test.ini
https://github.com/vrk-kpa/ckanext-xroad_integration/blob/4e7ebf0f9a3f3a21196018f7b94d4d70aae649dc/test.ini#L17

failing test:
https://github.com/vrk-kpa/ckanext-xroad_integration/blob/4e7ebf0f9a3f3a21196018f7b94d4d70aae649dc/ckanext/xroad_integration/tests/test_plugin.py#L134-L138
The failing test run can be seen here https://github.com/vrk-kpa/ckanext-xroad_integration/actions/runs/3938128022/jobs/6736463077 at least of awhile, the test fails as it tries to include templates from the extensions defined in test.ini
Turned out that the `ckan_config` mark actually overrides the configuration. The problem lies much deeper.

Every test uses [a snapshot of configuration object](https://github.com/ckan/ckan/blob/master/ckan/tests/pytest_ckan/ckan_setup.py#L79-L80) which is created [before the first test](https://github.com/ckan/ckan/blob/master/ckan/tests/pytest_ckan/ckan_setup.py#L49-L50). At this moment, CKAN loads all the plugins from the config file and saves template paths as `config['extra_template_paths']`. 

Then, even if you load no plugins via the `ckan_config` mark, the config snapshot already contains all the template paths. It behaves like a global variable. If, for example, you add `@pytest.mark.ckan_config("extra_template_paths", "")` decorator to the problematic test, it will pass.

It's still a bug, but it has a different scale. In order to solve this issue, we have to compute template paths in realtime or re-compute `extra_template_paths` whenever `with_plugins` fixture is applied(this option sounds like a better plan)

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -116,6 +116,14 @@ def update_config() -> None:
 
     webassets_init()
 
+    # these are collections of all template/public paths registered by
+    # extensions. Each call to `tk.add_template_directory` or
+    # `tk.add_public_directory` updates these collections. We have to reset
+    # them in order to remove templates/public files that came from plugins
+    # that were once enabled but are disabled right now.
+    config["plugin_template_paths"] = []
+    config["plugin_public_paths"] = []
+
     for plugin in reversed(list(p.PluginImplementations(p.IConfigurer))):
         # must do update in place as this does not work:
         # config = plugin.update_config(config)


Test patch:
diff --git a/ckan/tests/config/data/home/about.html b/ckan/tests/config/data/home/about.html
new file mode 100644
--- /dev/null
+++ b/ckan/tests/config/data/home/about.html
@@ -0,0 +1 @@
+YOU WILL NOT FIND ME
diff --git a/ckan/tests/config/test_environment.py b/ckan/tests/config/test_environment.py
--- a/ckan/tests/config/test_environment.py
+++ b/ckan/tests/config/test_environment.py
@@ -89,3 +89,11 @@ def test_siteurl_removes_backslash(ckan_config):
 def test_missing_timezone():
     with pytest.raises(CkanConfigurationException):
         environment.update_config()
+
+
+@pytest.mark.ckan_config("plugin_template_paths", [
+    os.path.join(os.path.dirname(__file__), "data")
+])
+def test_plugin_template_paths_reset(app):
+    resp = app.get("/about")
+    assert "YOU WILL NOT FIND ME" not in resp