Repository: ckan/ckan

Issue:
CKAN's state table clashes with PostGIS generated TIGER state table
### CKAN Version if known (or site URL)

2.8.0a HEAD [e219c2] (although observed earlier tags)

### Please describe the expected behaviour

On running `docker-compose up` in a freshly cloned repo, all services start and database migrations run successfully.

### Please describe the actual behaviour

`ckan` container exits during start-up with errors (from `db` and `ckan` containers):

```
ckan          | Traceback (most recent call last):
ckan          |   File "/usr/local/bin/ckan-paster", line 11, in <module>
ckan          |     sys.exit(run())
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
ckan          |     invoke(command, command_name, options, args[1:])
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
ckan          |     exit_code = runner.run(args)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
ckan          |     result = self.command()
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 356, in command
ckan          |     model.repo.init_db()
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/model/__init__.py", line 187, in init_db
db            | ERROR:  column "id" referenced in foreign key constraint does not exist
db            | STATEMENT:  
db            |         CREATE TABLE package (
db            |                 id SERIAL NOT NULL, 
db            |                 name VARCHAR(100) NOT NULL, 
db            |                 title TEXT, 
db            |                 version VARCHAR(100), 
db            |                 url TEXT, 
db            |                 download_url TEXT, 
db            |                 notes TEXT, 
db            |                 license_id INTEGER, 
ckan          |     self.upgrade_db()
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/model/__init__.py", line 284, in upgrade_db
ckan          |     mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
db            |                 state_id INTEGER, 
db            |                 revision_id INTEGER, 
db            |                 PRIMARY KEY (id), 
db            |                 UNIQUE (name), 
db            |                 FOREIGN KEY(license_id) REFERENCES license (id), 
db            |                 FOREIGN KEY(state_id) REFERENCES state (id), 
db            |                 FOREIGN KEY(revision_id) REFERENCES revision (id)
db            |         )
db            | 
db            | 
ckan          |     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
ckan          |   File "<decorator-gen-16>", line 2, in _migrate
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
ckan          |     return f(*a, **kw)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
ckan          |     schema.runchange(ver, change, changeset.step)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
ckan          |     change.run(self.engine, step)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
ckan          |     script_func(engine)
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/migration/versions/001_add_existing_tables.py", line 105, in upgrade
ckan          |     meta.create_all()
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3934, in create_all
ckan          |     tables=tables)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1929, in _run_visitor
ckan          |     conn._run_visitor(visitorcallable, element, **kwargs)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1538, in _run_visitor
ckan          |     **kwargs).traverse_single(element)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
ckan          |     return meth(obj, **kw)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 733, in visit_metadata
ckan          |     _is_metadata_operation=True)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
ckan          |     return meth(obj, **kw)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 767, in visit_table
ckan          |     include_foreign_key_constraints=include_foreign_key_constraints
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
ckan          |     return meth(self, multiparams, params)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
ckan          |     return connection._execute_ddl(self, multiparams, params)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1002, in _execute_ddl
ckan          |     compiled
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
ckan          |     context)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
ckan          |     exc_info
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
ckan          |     reraise(type(exception), exception, tb=exc_tb, cause=cause)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
ckan          |     context)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
ckan          |     cursor.execute(statement, parameters)
ckan          | sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "id" referenced in foreign key constraint does not exist
ckan          |  [SQL: '\nCREATE TABLE package (\n\tid SERIAL NOT NULL, \n\tname VARCHAR(100) NOT NULL, \n\ttitle TEXT, \n\tversion VARCHAR(100), \n\turl TEXT, \n\tdownload_url TEXT, \n\tnotes TEXT, \n\tlicense_id INTEGER, \n\tstate_id INTEGER, \n\trevision_id INTEGER, \n\tPRIMARY KEY (id), \n\tUNIQUE (name), \n\tFOREIGN KEY(license_id) REFERENCES license (id), \n\tFOREIGN KEY(state_id) REFERENCES state (id), \n\tFOREIGN KEY(revision_id) REFERENCES revision (id)\n)\n\n']
ckan exited with code 1

```

### What steps can be taken to reproduce the issue? 

Clone a fresh repository, with no existing docker images or volumes, then go to `./contrib/docker` and run:

```$ COMPOSE_HTTP_TIMEOUT=120 CKAN_SITE_URL=http://localhost CKAN_PORT=5000 docker-compose up```

### Notes

This seems to be caused by a clash of relation names in PostGIS, where the `mdillon/postgis` image creates a `state` table in the `tiger` namespace, and SQLAlchemy consequently does not create a table for the CKAN model ([./ckan/migration/versions/001_add_existing_tables.py:10](https://github.com/ckan/ckan/blob/e219c2352050d413392e33c560406a5b4e8557ac/ckan/migration/versions/001_add_existing_tables.py#L10)).

Adding `public` as a schema argument to the MetaData constructor ([./ckan/migration/versions/001_add_existing_tables.py:8](https://github.com/ckan/ckan/blob/e219c2352050d413392e33c560406a5b4e8557ac/ckan/migration/versions/001_add_existing_tables.py#L8)) seems to solve this. As a workaround, I have added a `DATABASE_SCHEMA=public` variable to pass to the constructor, but happy to produce a PR if I could have some guidance on the preferred means of configuring/passing this schema choice to the migration scripts.


Discussion:
@philtweir a PR would be great!
The other workaround is to remove tiger
I'd imagine that removing tiger would take an extra step ootb, or a separate image, as it's bundled with mdillon's PostGIS? (or customizing in docker-compose build, I suppose)

Schema as an environment variable, default "public", brought in via pylons_config sound OK then?
Yes, where I'm working we automatically remove it at startup; your solution sounds better :-)
I am getting the same error but adding the `DATABASE_SCHEMA=public` in 001_add_existing_tables.py:9 did not resolve it for me. Now I'm getting:
`NameError: global name 'public' is not defined`
Anything I might have missed? 
Sorry, that wasn't well described - that was thinking through passing the schema name in consistently (and a configurable patch is currently WIP), but the specific parameter you want is `schema` so you would have ``MetaData(schema='public')``, for example - this [minimal work-around](https://github.com/flaxandteal/ckan/blob/0ed116f3258cacac723978769d7607326aa082da/ckan/migration/versions/001_add_existing_tables.py#L9) may make it clearer.

Also, I did setup from the database from scratch before it all worked, to make sure the tables got created again in the right order, but if you really needed to, you could probably pick around that.
This worked fine. Thanks!
@philtweir This is such a particular case that I'm inclined to just use your minimal patch for now.
(but perhaps read it from an env var to make it customizable in case someone wants to use another schema name)
 To be consistent and do it "properly", if we wanted to made the schema customizable via a documented config option we would have to make sure that [every time](https://gist.github.com/amercader/47f45481eef5f2e617281883885038fc) a `MetaData` instance is created the schema config option is used. For that we would probably need to use our own wrapper around `MetaData` that just accesses this config option once. But as I said, the first approach should be enough if you don't want to follow that route.
Yep, that was the issue I was finding going down that path - particularly directly executed SQL statements, which would need the schema injected into the SQL, or a default schema set, with syntax that would need to switch for a sqlite test database. If you're happy, will go with that more targeted approach.
Having the same trouble on master (5ee0e074c6bfe7a288a8f35f2e1ca5f1af7ce59d) even after adding the schema fix to `ckan/migration/versions/001_add_existing_tables.py` that others mentioned above.  Docker on Windows 10.

```
$ docker-compose logs -f ckan
Attaching to ckan
ckan          | Traceback (most recent call last):
ckan          |   File "/usr/local/bin/ckan-paster", line 11, in <module>
ckan          |     sys.exit(run())
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
ckan          |     invoke(command, command_name, options, args[1:])
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
ckan          |     exit_code = runner.run(args)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
ckan          |     result = self.command()
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 356, in command
ckan          |     model.repo.init_db()
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/model/__init__.py", line 187, in init_db
ckan          |     self.upgrade_db()
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/model/__init__.py", line 284, in upgrade_db
ckan          |     mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
ckan          |     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
ckan          |   File "<decorator-gen-16>", line 2, in _migrate
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_eng
ine
ckan          |     return f(*a, **kw)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
ckan          |     schema.runchange(ver, change, changeset.step)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
ckan          |     change.run(self.engine, step)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
ckan          |     script_func(engine)
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/migration/versions/001_add_existing_tables.py", line 105, in upgrade
ckan          |     meta.create_all()
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3934, in create_all
ckan          |     tables=tables)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1929, in _run_visitor
ckan          |     conn._run_visitor(visitorcallable, element, **kwargs)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1538, in _run_visitor
ckan          |     **kwargs).traverse_single(element)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
ckan          |     return meth(obj, **kw)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 733, in visit_metadata
ckan          |     _is_metadata_operation=True)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
ckan          |     return meth(obj, **kw)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 767, in visit_table
ckan          |     include_foreign_key_constraints=include_foreign_key_constraints
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
ckan          |     return meth(self, multiparams, params)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
ckan          |     return connection._execute_ddl(self, multiparams, params)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1002, in _execute_ddl
ckan          |     compiled
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
ckan          |     context)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exc
eption
ckan          |     exc_info
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
ckan          |     reraise(type(exception), exception, tb=exc_tb, cause=cause)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
ckan          |     context)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
ckan          |     cursor.execute(statement, parameters)
ckan          | sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "id" referenced in foreign key constraint does not e
xist
ckan          |  [SQL: '\nCREATE TABLE package (\n\tid SERIAL NOT NULL, \n\tname VARCHAR(100) NOT NULL, \n\ttitle TEXT, \n\tversion VARC
HAR(100), \n\turl TEXT, \n\tdownload_url TEXT, \n\tnotes TEXT, \n\tlicense_id INTEGER, \n\tstate_id INTEGER, \n\trevision_id INTEGER, \n
\tPRIMARY KEY (id), \n\tUNIQUE (name), \n\tFOREIGN KEY(license_id) REFERENCES license (id), \n\tFOREIGN KEY(state_id) REFERENCES state (
id), \n\tFOREIGN KEY(revision_id) REFERENCES revision (id)\n)\n\n']
ckan exited with code 1
```
  
@carlsonp have you pruned your volumes and rebuild your instance? Else the source code won't update properly
@gerbyzation Thanks for reply.  I believe so, I ran these which I think clears everything out and starts from scratch.  https://gist.github.com/JeffBelback/5687bb02f3618965ca8f
I then rebuilt: `docker-compose up -d --build`
Same behaviour here.
Commit 5ee0e074c (HEAD -> master, origin/master, origin/HEAD), removed docker_<image-name> images, all things pruned 
```
git pull origin master
docker container rm ckan
docker container rm solr
docker container rm redis
docker container rm datapusher
docker container rm db
docker image rm docker_ckan
docker image rm docker_db
docker image rm docker_solr
docker image prune
docker volumes prune
```
docker-compose up --build
```
ckan          |     context)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
ckan          |     cursor.execute(statement, parameters)
ckan          | sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "id" referenced in foreign key constraint does not exist
ckan          |  [SQL: '\nCREATE TABLE package (\n\tid SERIAL NOT NULL, \n\tname VARCHAR(100) NOT NULL, \n\ttitle TEXT, \n\tversion VARCHAR(100), \n\turl TEXT, \n\tdownload_url TEXT, \n\tnotes TEXT, \n\tlicense_id INTEGER, \n\tstate_id INTEGER, \n\trevision_id INTEGER, \n\tPRIMARY KEY (id), \n\tUNIQUE (name), \n\tFOREIGN KEY(license_id) REFERENCES license (id), \n\tFOREIGN KEY(state_id) REFERENCES state (id), \n\tFOREIGN KEY(revision_id) REFERENCES revision (id)\n)\n\n']
ckan exited with code 1
```
  
@carlsonp @giafar a slightly easier way to do this is to run the following from the `contrib/docker` directory
```
$ docker-compose down
$ docker volume prune
```
This will shut down and remove the containers run through compose and then prune all unattached volumes. After this you can (re)build if necessary and use `docker-compose up` again. 

I'm afraid I can't be of further help at the moment as my internet is extremely poor at the moment and I am unable to rebuild.
  
I might be being daft, but I can't see any autolink/notification here to the PR itself, so just for reference: https://github.com/ckan/ckan/pull/3965
In #4016 I've done a PR for the simpler version @ad-m proposed as an alternative to PR #3965. I think that we shouldn't go to lengths to make the namespace configurable - no-one has asked for that. Let's just fix the specific clash with this one-liner, and backport it to the last release, so that the docker install works again for people. What do you think @amercader ?
I'm certainly fine with that (as #3965 author) - the configurability came from suggestion in https://github.com/ckan/ckan/issues/3929#issuecomment-349459395 , not as an additional option, but rather than hard-coding the Postgres setting into each migration, as I didn't think that would be mergeable (as standard as it is).

However, at present, docker-compose is still broken, so anything that gets it fixed in master would be helpful for me too :)
Are you saying there's a half-way house? If the schema/namespace needs to be configurable, then don't we need your full PR plus also the MetaData @amercader spotted, plus the same for any extensions used? I just don't see anyone ever saying they need that configurability. It might be more proper, but it's a lot of addition.

So I'm simply suggesting we are simply explicit about the schema/namespace, for this one case where sqlalchemy needs bringing into line, to fix the issue at hand.
What you're suggesting is more about making the schema configurable across the whole application. That's broader, beyond this issue. I spent some time taking a look but, as mentioned there, it would involve explicitly a lot more modification than would be worthwhile.

As it stands, CKAN will install to and read from whatever is set as the default schema (public or otherwise, set in postgresql.conf) _except_, due to the bug above, when a relation name exists in another schema. Naming the default schema explicitly then stops that.

My understanding of @amercader's comment above is that allowing that to be settable via env beats hard-coding `public`. I've done that in each migration (as a two-liner per migration) for consistency, so that if mdillon/postgis changes (or you have anything in PG alongside CKAN) this doesn't randomly break again - there's nothing specific about the "state" table, other than it's a popular name.

Either way - I'm not disagreeing - that's not necessarily better, if it increases complexity. As far as I can see there are three key differences:
 - I have set the schema consistently in every migration, rather than one,
 - the schema name can be overridden by env,
 - but, thanks to CkanMetaData, that happens solely in one place.

However, that means all the migrations get touched in the PR, which is a lot of files for a single issue - #4016 may be better trade-off.
Ok that's really good to know that my change could screw things if someone changed their default schema in postgresql.conf. On balance, for that little bit of safety, I'd be tempted to use your change then.

Resolving patch:
diff --git a/ckan/migration/versions/001_add_existing_tables.py b/ckan/migration/versions/001_add_existing_tables.py
--- a/ckan/migration/versions/001_add_existing_tables.py
+++ b/ckan/migration/versions/001_add_existing_tables.py
@@ -2,10 +2,17 @@
 
 from sqlalchemy import *
 from migrate import *
+from ckan.common import config as ckan_config
 
 
 def upgrade(migrate_engine):
-    meta = MetaData()
+    schema = ckan_config.get(u'ckan.migrations.target_schema') or 'public'
+    # we specify the schema here because of a clash with another 'state' table
+    # in the mdillon/postgis container. You only need to change the value in the
+    # config if you've altered the default schema from 'public' in your
+    # postgresql.conf. Because this is such a rarely needed option, it is
+    # otherwise undocumented.
+    meta = MetaData(schema=schema)
 
     state = Table('state', meta,
       Column('id', Integer() ,  primary_key=True, nullable=False),


Test patch: