Repository: ckan/ckan

Issue:
Don't use paste.deploy appconfig to parse the ini file
Right now we are using paste.deploy's `appconfig` to parse the INI file and get the `config` object that CKAN uses. It offers some functionality that we don't use (like the ability to parse an INI file hosted elsewhere via HTTP), but basically it's a wrapper around `ConfigParser`. The one feature that we need to keep for backwards compatibility is interpolation (eg things like `cache_dir = /tmp/%(ckan.site_id)s/`), which is also supported in the [standard library](https://docs.python.org/3/library/configparser.html#interpolation-of-values).

#### Approach

Write our own simpler version of the configuration file parser and replace the `appconfig` calls in the CLI code.


#### Story points

5


Discussion:
@amercader can you please add some notes of where to check existing code, who is the user of this code (to test that the "contract" is kept) etc? I have a good understanding of what needs to be done, as I've written some config parsers. Looking for where exactly to start.
We use the config parser whenever we start the application to parse the ini file and pass the config object to `load_environment()`, which will set everything up.

This can be done more explicitly or hidden as part of a higher level execution. The only place we are doing this directly right now is in the CLI, and the only ones that should be updated are:

* [Old paster CLI](https://github.com/ckan/ckan/blob/master/ckan/lib/cli.py#L205)
* [New click CLI](https://github.com/ckan/ckan/blob/master/ckan/cli/__init__.py#L25). This one covers the new development server (`ckan run`) 

In here you need to replace usage of `appconfig` with our own parser. Check the paste source to see what is special about `appconfig` but as I said in the description I think we only need to keep the interpolation support.

Other places where the config is parsed include the tests. Currently the parsing is done via a Pylons plugin (`nose --with-pylons`) but the new pytest based tests use [our own](https://github.com/ckan/ckan/pull/4996/files#diff-023eefdbe76a226e831eebffefa86b9eR15) parser.

Another place where this will be used will be the wsgi entrypoint for production servers, like the current [Apache one](https://github.com/ckan/ckan-packaging/blob/master/common/etc/ckan/default/apache.wsgi) (which will be replaced by uswgi or gunicorn).

But for now as I said the goal is to parse the ini file and get a config object, and refactor the cli methods to use our own parser.

Let me know if this makes sense.

One important functionality that we need to keep is support for the interpolated variables eg (`%(ckan.site_id)s`). So this ini file:

```
# This is /etc/ckan/production.ini

cache_dir = /tmp/%(ckan.site_id)s/ 
ckan.site_id = default
who.config_file = %(here)s/who.ini
who.log_file = %(cache_dir)s/who_log.ini
```

Should become this config object

```python
config = {
  'cache_dir': '/tmp/default',
  'ckan.site_id': 'default
  'who.config_file': '/etc/ckan/who.ini
  'who.log_file': '/tmp/default/who_log.ini
}
```

[This module](https://github.com/Pylons/pastedeploy/blob/9cbc584a86bd1ae425bd0c0c3c00617d7e59475d/paste/deploy/loadwsgi.py) contains the source code of paster's paster in case you want to check the implementation (but that one is way more complex than what we need)

Resolving patch:
diff --git a/ckan/cli/__init__.py b/ckan/cli/__init__.py
--- a/ckan/cli/__init__.py
+++ b/ckan/cli/__init__.py
@@ -4,19 +4,67 @@
 
 import click
 import logging
-from logging.config import fileConfig as loggingFileConfig
+from configparser import ConfigParser
 
 
 log = logging.getLogger(__name__)
 
 
+class CKANConfigLoader(object):
+    def __init__(self, filename):
+        self.filename = filename = filename.strip()
+        self.parser = ConfigParser()
+        self.section = u'app:main'
+        self.read_config_files(filename)
+
+        defaults = {
+            u'here': os.path.dirname(os.path.abspath(filename)),
+            u'__file__': os.path.abspath(filename)
+        }
+        self._update_defaults(defaults)
+
+    def read_config_files(self, filename):
+        '''
+        Read and parses a config file. If the config file has
+        'use=config:<filename>' then it parses both files. Automatically
+        applies interpolation if needed.
+        '''
+        self.parser.read(filename)
+
+        schema, path = self.parser.get(self.section, u'use').split(u':')
+        if schema == u'config':
+            path = os.path.join(
+                os.path.dirname(os.path.abspath(filename)), path)
+            self.parser.read([path, filename])
+
+    def _update_defaults(self, new_defaults):
+        for key, value in new_defaults.items():
+            self.parser._defaults[key] = value
+
+    def get_config(self):
+        global_conf = self.parser.defaults().copy()
+        local_conf = {}
+        options = self.parser.options(self.section)
+
+        for option in options:
+            if option in global_conf:
+                continue
+            local_conf[option] = self.parser.get(self.section, option)
+
+        return CKANLoaderContext(global_conf, local_conf)
+
+
+class CKANLoaderContext(object):
+    def __init__(self, global_conf, local_conf):
+        self.global_conf = global_conf
+        self.local_conf = local_conf
+
+
 def error_shout(exception):
     click.secho(str(exception), fg=u'red', err=True)
 
 
 def load_config(ini_path=None):
-    from paste.deploy import appconfig
-
     if ini_path:
         if ini_path.startswith(u'~'):
             ini_path = os.path.expanduser(ini_path)
@@ -42,6 +90,7 @@ def load_config(ini_path=None):
         msg += u'\n(Given by: %s)' % config_source
         exit(msg)
 
-    loggingFileConfig(filename)
+    config_loader = CKANConfigLoader(filename)
     log.info(u'Using configuration file {}'.format(filename))
-    return appconfig(u'config:' + filename)
+
+    return config_loader.get_config()
diff --git a/ckan/lib/cli.py b/ckan/lib/cli.py
--- a/ckan/lib/cli.py
+++ b/ckan/lib/cli.py
@@ -29,6 +29,7 @@
 from paste.registry import Registry
 from paste.script.util.logging_config import fileConfig
 import click
+from ckan.cli import load_config as _get_config
 
 from ckan.config.middleware import make_app
 import ckan.logic as logic
@@ -183,36 +184,6 @@ def ungettext(self, singular, plural, n):
         return singular
 
 
-def _get_config(config=None):
-    from paste.deploy import appconfig
-
-    if config:
-        filename = os.path.abspath(config)
-        config_source = '-c parameter'
-    elif os.environ.get('CKAN_INI'):
-        filename = os.environ.get('CKAN_INI')
-        config_source = '$CKAN_INI'
-    else:
-        default_filename = 'development.ini'
-        filename = os.path.join(os.getcwd(), default_filename)
-        if not os.path.exists(filename):
-            # give really clear error message for this common situation
-            msg = 'ERROR: You need to specify the CKAN config (.ini) '\
-                'file path.'\
-                '\nUse the --config parameter or set environment ' \
-                'variable CKAN_INI or have {}\nin the current directory.' \
-                .format(default_filename)
-            exit(msg)
-
-    if not os.path.exists(filename):
-        msg = 'Config file not found: %s' % filename
-        msg += '\n(Given by: %s)' % config_source
-        exit(msg)
-
-    fileConfig(filename)
-    return appconfig('config:' + filename)
-
-
 def load_config(config, load_site_user=True):
     conf = _get_config(config)
     assert 'ckan' not in dir()  # otherwise loggers would be disabled
@@ -249,7 +220,7 @@ def load_config(config, load_site_user=True):
         pylons.c.userobj = model.User.get(site_user['name'])
 
     ## give routes enough information to run url_for
-    parsed = urlparse(conf.get('ckan.site_url', 'http://0.0.0.0'))
+    parsed = urlparse(conf.local_conf.get('ckan.site_url', 'http://0.0.0.0'))
     request_config = routes.request_config()
     request_config.host = parsed.netloc + parsed.path
     request_config.protocol = parsed.scheme


Test patch:
diff --git a/ckan/tests/cli/data/test-core.ini.tpl b/ckan/tests/cli/data/test-core.ini.tpl
new file mode 100644
--- /dev/null
+++ b/ckan/tests/cli/data/test-core.ini.tpl
@@ -0,0 +1,17 @@
+[server:main]
+use = egg:Paste#http
+host = 0.0.0.0
+port = 5000
+
+[app:main]
+use = egg:ckan
+full_stack = true
+cache_dir = /tmp/%(ckan.site_id)s/
+debug = false
+testing = true
+
+# Specify the Postgres database for SQLAlchemy to use
+sqlalchemy.url = postgresql://ckan_default:pass@localhost/ckan_test
+
+ckan.site_id = default
+
diff --git a/ckan/tests/cli/data/test.ini.tpl b/ckan/tests/cli/data/test.ini.tpl
new file mode 100644
--- /dev/null
+++ b/ckan/tests/cli/data/test.ini.tpl
@@ -0,0 +1,13 @@
+[DEFAULT]
+debug = true
+smtp_server = localhost
+error_email_from = ckan-errors@example.com
+
+[server:main]
+use = egg:Paste#http
+host = 0.0.0.0
+port = 5000
+
+[app:main]
+use = config:test-core.ini.tpl
+faster_db_test_hacks = True
\ No newline at end of file
diff --git a/ckan/tests/cli/test_cli.py b/ckan/tests/cli/test_cli.py
--- a/ckan/tests/cli/test_cli.py
+++ b/ckan/tests/cli/test_cli.py
@@ -1,8 +1,10 @@
 # -*- coding: utf-8 -*-
 
 import pytest
+import os
 
 from ckan.cli.cli import ckan
+from ckan.cli import CKANConfigLoader
 
 
 def test_without_args(cli):
@@ -70,3 +72,20 @@ def test_command_from_extension_is_available_when_all_requirements_satisfied(cli
     """
     result = cli.invoke(ckan, [u'example-iclick-hello'])
     assert not result.exit_code
+
+
+def test_ckan_config_loader_parse_files():
+    """
+    CKANConfigLoader should parse both 'test.ini.tpl' and 'test-core.ini.tpl'
+    files since test.ini.ptl has a use = config:test-core.ini.tpl config.
+    """
+    filename = os.path.join(os.path.dirname(__file__), u'data/test.ini.tpl')
+    conf = CKANConfigLoader(filename).get_config()
+
+    assert conf.global_conf[u'debug'] == u'true'
+    assert conf.global_conf[u'smtp_server'] == u'localhost'
+    assert conf.local_conf[u'ckan.site_id'] == u'default'
+    assert conf.local_conf[u'faster_db_test_hacks'] == u'True'
+    assert conf.local_conf[u'cache_dir'] == u'/tmp/default/'
+    assert (conf.local_conf[u'sqlalchemy.url'] ==
+            u'postgresql://ckan_default:pass@localhost/ckan_test')
diff --git a/test.ini b/test.ini
--- a/test.ini
+++ b/test.ini
@@ -21,7 +21,7 @@ use = config:test-core.ini
 # Here we hard-code the database and a flag to make default tests
 # run fast.
 faster_db_test_hacks = True
-sqlalchemy.url = sqlite:///
+
 # NB: other test configuration should go in test-core.ini, which is
 #     what the postgres tests use.
 
@@ -41,13 +41,13 @@ handlers = console
 
 [logger_ckan]
 qualname = ckan
-handlers = 
+handlers =
 level = INFO
 
 [logger_sqlalchemy]
 handlers =
 qualname = sqlalchemy.engine
-level = WARN  
+level = WARN
 
 [handler_console]
 class = StreamHandler