Repository: ckan/ckan

Issue:
paster db upgrade failure 2.1->2.3
bumped into this when trying to test another issue. If I run `paster db upgrade` on master with a db from 2.1 I get this traceback:

``` pytb
Traceback (most recent call last):
  File "/home/ian/ckan2env26/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/home/ian/git/ckan/ckan/lib/cli.py", line 192, in command
    self._load_config()
  File "/home/ian/git/ckan/ckan/lib/cli.py", line 150, in _load_config
    self.site_user = logic.get_action('get_site_user')({'ignore_auth': True}, {})
  File "/home/ian/git/ckan/ckan/logic/__init__.py", line 424, in wrapped
    result = _action(context, data_dict, **kw)
  File "/home/ian/git/ckan/ckan/logic/action/get.py", line 2185, in get_site_user
    user = model.User.get(site_id)
  File "/home/ian/git/ckan/ckan/model/user.py", line 64, in get
    return query.first()
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2156, in first
    ret = list(self[0:1])
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2023, in __getitem__
    return list(res)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/home/ian/ckan2env26/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.state does not exist
LINE 1: ..._notifications, "user".sysadmin AS user_sysadmin, "user".sta...
```



Discussion:
This is pretty serious as if I'm not mistaken, it also prevents 2.1 -> 2.2.

The problem seems to have been introduced in #1178. A new column `state` was added to the users table, and vdm was thrown into the mix. It now seems that when calling `paster db upgrade` and getting the site user the model is initialized as if the user table had a `state` column (I guess [this](https://github.com/ckan/ckan/blob/d680a44bff3ef6fb865f3c41e9554a85222b00ac/ckan/model/user.py#L36) does it) but that's still not the case physically in the DB.

Not sure how to tackle this one, apart from (more) horrible hacks on [cli.py](https://github.com/ckan/ckan/blob/d680a44bff3ef6fb865f3c41e9554a85222b00ac/ckan/lib/cli.py#L106) to prevent getting the site user on `paster db upgrade`.

And to make things worse the [migration script](https://github.com/ckan/ckan/blob/master/ckan/migration/versions/071_add_state_column_to_user_table.py) is not wrapped in a transaction so even if you add the field manually the migration hangs :(

I was able to upgrade from 2.1 to 2.2 (note that it took about 24 hours to complete). I just started the migration from 2.2->master

After letting it run for more than a week I think it's safe to assume that my migration failed, so I still think there might be issues when doing 2.1 -> 2.2

i have the same trouble when i doing 2.2a -> 2.2.1.


Resolving patch:
diff --git a/ckan/lib/cli.py b/ckan/lib/cli.py
--- a/ckan/lib/cli.py
+++ b/ckan/lib/cli.py
@@ -135,7 +135,7 @@ def _get_config(self):
         fileConfig(self.filename)
         return appconfig('config:' + self.filename)
 
-    def _load_config(self):
+    def _load_config(self, load_site_user=True):
         conf = self._get_config()
         assert 'ckan' not in dir() # otherwise loggers would be disabled
         # We have now loaded the config. Now we can import ckan for the
@@ -149,7 +149,7 @@ def _load_config(self):
         self.translator_obj = MockTranslator()
         self.registry.register(pylons.translator, self.translator_obj)
 
-        if model.user_table.exists():
+        if model.user_table.exists() and load_site_user:
             # If the DB has already been initialized, create and register
             # a pylons context object, and add the site user to it, so the
             # auth works as in a normal web request
@@ -199,11 +199,12 @@ class ManageDb(CkanCommand):
     min_args = 1
 
     def command(self):
-        self._load_config()
+        cmd = self.args[0]
+
+        self._load_config(cmd!='upgrade')
         import ckan.model as model
         import ckan.lib.search as search
 
-        cmd = self.args[0]
         if cmd == 'init':
 
             model.repo.init_db()


Test patch: