Repository: ckan/ckan

Issue:
[DOCKER] docker-compose up not working
Running `docker-compose up -d` on the `docker-compose.yml` file in `contrib/docker` seems to result in the `ckan` container not running? When inspecting the `ckan` container logs it shows me this:
```
Distribution already installed:
  ckan 2.7.0a0 from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
  /etc/ckan/default/ckan.ini
2017-05-18 17:46:43,354 ERROR [ckan.lib.redis] Redis is not available
Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/redis.py", line 61, in is_redis_available
    return redis_conn.ping()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/redis/client.py", line 682, in ping
    return self.execute_command('PING')
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
2017-05-18 17:46:43,355 CRITI [ckan.config.environment] Could not connect to Redis.
Traceback (most recent call last):
  File "/usr/local/bin/ckan-paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 334, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 303, in _load_config
    load_config(self.options.config, load_site_user)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 221, in load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 103, in load_environment
    p.load_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 129, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 266, in update_config
    model.init_model(engine)
  File "/usr/lib/ckan/default/src/ckan/ckan/model/__init__.py", line 157, in init_model
    version_table = Table('migrate_version', meta.metadata, autoload=True)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 352, in __new__
    table._init(name, metadata, *args, **kw)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 425, in _init
    self._autoload(metadata, autoload_with, include_columns)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 448, in _autoload
    self, include_columns, exclude_columns
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1670, in run_callable
    with self.contextual_connect() as conn:
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1737, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 332, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 630, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 433, in checkout
    rec = pool._do_get()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 949, in _do_get
    return self._create_connection()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 278, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 404, in __init__
    self.connection = self.__connect()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 530, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 95, in connect
    connection_invalidated=invalidated
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 89, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 376, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (OperationalError) could not connect to server: Connection refused
	Is the server running on host "172.17.0.3" and accepting
	TCP/IP connections on port 5432?
 None None
```

Any ideas on what goes wrong?


Discussion:
@mattfullerton is this related to the ports issue you were discussion recently?
Ok; so I made one little change which I didn't tell you all about, and apparently reverting this change solved my problem. It does pose a different challenge though. The change I made was to the binding of port 80 to port 5000 for the `ckan` container. I changed this line to:
```
      - "8083:5000"
```
Apparently this breaks the whole setup. Which is a bad thing because you can't assume CKAN to be installed on a server which doesn't already use port 80 (e.g., we already bind port 80 via Docker to an Nginx load balancer).

So my next question is, how can I change `docker-compose.yml` to not be dependent on binding port 80?
@amercader yes - as you can see its trying to reach redis at localhost (this is in the code as a default - it can and should be overriden in the .ini for docker-compose)

What I can't explain off the top of my head is why CKAN is trying to reach Postgres at 172.17.0.3 - that's a docker IP... but ideally it should be using the container name.

I will try to get a look at this on a suitable machine ASAP.
@mattfullerton I guess you see the Postgres docker container IP address because it already resolved it from the container name? I'm still not sure why changing the binding of port 80 causes the whole thing the break down.
https://github.com/ckan/ckan/pull/3572
you should port docker-compose to version 3. now its in version 1 so binding works only via env vars
@mattfullerton given your pull request, what values should I set for the environment variables in order to run `docker-compose up -d` without binding port 80?
For that it's even easier: just change this line
https://github.com/ckan/ckan/blob/master/contrib/docker/docker-compose.yml#L10

to something else instead of 80.

That said, we need a way of piping through a site url. This is bad:
https://github.com/ckan/ckan/blob/master/Dockerfile#L9


@mattfullerton I tried changing/removing that line but that still results in the errors shown at the top of this issue.

update: I didn't try out your change in a correct way, so it might work. I'll try to test it later again.

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -95,13 +95,13 @@ def find_controller(self, controller):
     for msg in msgs:
         warnings.filterwarnings('ignore', msg, sqlalchemy.exc.SAWarning)
 
+    # load all CKAN plugins
+    p.load_all()
+
     # Check Redis availability
     if not is_redis_available():
         log.critical('Could not connect to Redis.')
 
-    # load all CKAN plugins
-    p.load_all()
-
     app_globals.reset()
 
     # issue #3260: remove idle transaction


Test patch: