Repository: privacyidea/privacyidea

Issue:
Migration from LinOTP fails on NOT NULL constraint on `token.id`
Attempting to use the privacyidea-migrate-linotp.py script with postgresql+psycopg2 for both the databases:

```
Adding 40 tokens...
Insert token records 0 to 39 ...
Failed to insert chunk: (psycopg2.errors.NotNullViolation) null value in column "id" violates not-null constraint
DETAIL:  Failing row contains (...)

...

(Background on this error at: http://sqlalche.me/e/13/gkpj)
Traceback (most recent call last):
  File "./privacyidea-migrate-linotp.py", line 579, in <module>
    main()
  File "./privacyidea-migrate-linotp.py", line 569, in main
    migrate(config_obj)
  File "./privacyidea-migrate-linotp.py", line 471, in migrate
    token_values[i]["id"] = token_serial_id_map[token_values[i]["serial"]]
KeyError: '...'
```

Looking in the new PrivacyIDEA database (created by `pi-manage createdb`), the table `token` has an `id` column which is set NOT NULL, but has no default value.  The migration script is trying to insert rows into this table, but is not providing a value for `id` fields.

If the `id` column is supposed to contain a synthetic primary key, it should probably be created with the type `serial` or `bigserial` in Postgres.

It turns out that the database does have sequences defined for these `id` columns, but they are not set as the column DEFAULT in the database.  To fix this, you can do the following for each table:

```
ALTER TABLE token ALTER COLUMN id SET DEFAULT nextval('token_seq');
ALTER SEQUENCE token_seq OWNED BY token.id;
```

I went through and set up the column defaults for `token`, `tokenrealm`, `tokenowner` and `tokeninfo`, and then the migration script was able to complete successfully.


Discussion:
The script is not intended and tested to run with postgres.
Yes, I can see that :)  It's pretty close though, we might be able to get it to work perfectly well with Postgres, with just some small tweaks.

I dug a little deeper and noticed that your models.py for the main app declares a `Sequence` for the `id` columns, e.g. `Sequence("token_seq")`, but the models baked in to the migration script are missing this.  It could be as simple as adding those Sequences to the models in the migration script.  I'd be happy to give it a try, and raise a PR if it works. 

Resolving patch:
diff --git a/tools/privacyidea-migrate-linotp.py b/tools/privacyidea-migrate-linotp.py
--- a/tools/privacyidea-migrate-linotp.py
+++ b/tools/privacyidea-migrate-linotp.py
@@ -52,7 +52,7 @@
 from sqlalchemy import create_engine
 from sqlalchemy.orm import sessionmaker
 from sqlalchemy.sql import select
-from sqlalchemy.schema import ForeignKey
+from sqlalchemy.schema import ForeignKey, Sequence
 from sqlalchemy import (Table, MetaData, Column, Integer, Unicode, Boolean,
                         UnicodeText)
 import sys
@@ -175,7 +175,8 @@ def migrate(config_obj):
     metadata = MetaData()
 
     token_table = Table("token", metadata,
-                        Column("id", Integer, primary_key=True, nullable=False),
+                        Column("id", Integer, Sequence('token_seq'),
+                               primary_key=True, nullable=False),
                         Column("description", Unicode(80), default=u''),
                         Column("serial", Unicode(40), default=u'', unique=True,
                                nullable=False, index=True),
@@ -201,7 +202,8 @@ def migrate(config_obj):
                         Column("rollout_state", Unicode(10), default=u''))
 
     tokenowner_table = Table("tokenowner", metadata,
-                             Column("id", Integer, primary_key=True, nullable=False),
+                             Column("id", Integer, Sequence('tokenowner_seq'),
+                                    primary_key=True, nullable=False),
                              Column("token_id", Integer, ForeignKey("token.id"), nullable=False),
                              Column("resolver", Unicode(120), default=u"", index=True),
                              Column("user_id", Unicode(320), default=u"", index=True),
@@ -209,7 +211,8 @@ def migrate(config_obj):
                              )
 
     tokeninfo_table = Table("tokeninfo", metadata,
-                            Column("id", Integer, primary_key=True),
+                            Column("id", Integer, Sequence('tokeninfo_seq'),
+                                   primary_key=True),
                             Column("Key", Unicode(255), nullable=False),
                             Column("Value", UnicodeText(), default=u''),
                             Column("Type", Unicode(100), default=u''),
@@ -217,23 +220,29 @@ def migrate(config_obj):
                             Column("token_id", Integer()))
 
     tokenrealm_table = Table("tokenrealm", metadata,
-                             Column("id", Integer(), primary_key=True),
+                             Column("id", Integer(),
+                                    Sequence('tokenrealm_seq'),
+                                    primary_key=True),
                              Column("token_id", Integer()),
                              Column("realm_id", Integer()))
 
     realm_table = Table("realm", metadata,
-                        Column("id", Integer, primary_key=True),
+                        Column("id", Integer, Sequence('realm_seq'),
+                               primary_key=True),
                         Column("name", Unicode(255), default=u''),
                         Column("default",  Boolean(), default=False),
                         Column("option", Unicode(40), default=u''))
 
     resolver_table = Table("resolver", metadata,
-                           Column("id", Integer, primary_key=True),
+                           Column("id", Integer, Sequence('resolver_seq'),
+                                  primary_key=True),
                            Column("name", Unicode(255), default=u""),
                            Column("rtype", Unicode(255), default=u""))
 
     resolver_config_table = Table("resolverconfig", metadata,
-                                  Column("id", Integer, primary_key=True),
+                                  Column("id", Integer,
+                                         Sequence('resolverconf_seq'),
+                                         primary_key=True),
                                   Column("resolver_id", Integer),
                                   Column("Key", Unicode(255), default=u""),
                                   Column("Value", Unicode(2000), default=u""),
@@ -242,7 +251,9 @@ def migrate(config_obj):
                                   )
 
     resolverrealm_table = Table("resolverrealm", metadata,
-                                Column("id", Integer, primary_key=True),
+                                Column("id", Integer,
+                                       Sequence('resolverrealm_seq'),
+                                       primary_key=True),
                                 Column("resolver_id", Integer),
                                 Column("realm_id", Integer),
                                 Column("priority", Integer))


Test patch: