Repository: privacyidea/privacyidea

Issue:
tune up migration script
The migration script ``privacyidea-migrate-linotp.py`` runs slow for a big number of tokens.

We need to identify the problem (maybe due to tokeninfo, resolvers and realms) and speed it up.
@fredreichbier I would like to discuss this.

Things that can be done are:

1. Check if it is better to first **read** all information from the old database to a dictionary, before writing the new database.

1. add a parameter to simply ignore all tokeninfo or only migrate certain tokeninfo field.
    https://github.com/privacyidea/privacyidea/blob/master/tools/privacyidea-migrate-linotp.py#L200
    https://github.com/privacyidea/privacyidea/blob/master/tools/privacyidea-migrate-linotp.py#L227

1. skip resolver and realm definitions. In contrast we could configure resolvers and realms in the
   destination privacyIDEA server and add a mapping to the script, that maps old resolvers to new resolvers and old realm names to new realmnames.






Discussion:
I had a quick look at the code and I think point (1) seems like a good a idea which can already bring a nice speedup. For example, if I read the code correctly, [this line](https://github.com/privacyidea/privacyidea/blob/master/tools/privacyidea-migrate-linotp.py#L225) executes an INSERT for every migrated token. We should probably aggregate the new tokens and bulk-insert them.
OK, the bulk operation seem to work with the ORM.
http://docs.sqlalchemy.org/en/rel_1_0/orm/persistence_techniques.html#bulk-operations

So we would have to rewrite from using the table statements like 

    token_table.insert.values()

to using the ORM with ``Token`` database classes.

    session.bulk_save_objects([Token(), Token(), Token()])

We still would need to check, how we could write tokeninfos for those token objects into the tokeninfo table.


There might even be a way to get a similar effect without using the ORM by wrapping all token and token info INSERTs into one transaction. I can have a look into this later.
[This example](http://docs.sqlalchemy.org/en/latest/faq/performance.html#i-m-inserting-400-000-rows-with-the-orm-and-it-s-really-slow) is probably interesting. There, they use:
```python
    engine.execute(
        Customer.__table__.insert(),
        [{"name": 'NAME ' + str(i)} for i in xrange(n)]
    )
```
Here is a mysqldump of 12.000 Tokens in the linotp token database. 

[linotp-token-table.txt](https://github.com/privacyidea/privacyidea/files/1706908/linotp-token-table.txt)

Some migration tests with 12.000 tokens

tokens | tokeninfo | realms | resolvers | time 
--- | --- | --- | --- | ---
| Y | N  | N | N | 40 secs | 
| Y | Y | N | N | 80 secs | 

The realm assignment of the tokens was not migrated, yet!
The problem is, we loose the assigement if we do NOT fill the table "tokenrealm".
With a458207 applied, importing 12000 tokens (token + tokeninfo) goes down from 30s to 2s on my system. Maybe that's already enough?
I like this!
I will export the resolvers and realms and check with this patch!


This is lighning fast. But acutally we have a small problem: 

This only works with an empty token database, as we also import the the ``id`` into the ``token`` table.
It might be necessary to import into a database with existing tokens. Then the ``id`` would have to be rewritten. 

I will check this.

Resolving patch:
diff --git a/tools/privacyidea-migrate-linotp.py b/tools/privacyidea-migrate-linotp.py
--- a/tools/privacyidea-migrate-linotp.py
+++ b/tools/privacyidea-migrate-linotp.py
@@ -1,354 +1,407 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
 #
+#  2018-02-09 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#             Speedup thanks to Friedrich Weber.
+#             Add flexible resolver mapping
 #  2016-09-14 Cornelius Kölbel <cornelius.koelbel@netknights.it>
 #
 """
 You can use this script to migrate a LinOTP database to privacyIDEA.
 All tokens and token assignments are transferred to privacyIDEA.
 
+You only need an export of the LinOTP "Token" table.
+Please configure below, how your database URIs look like,
+what should be migrated and what should be mapped.
 """
-# You need to change this!
-LINOTP_URI = "mysql://linotp2:testtest@localhost/LinOTP2"
-PRIVACYIDEA_URI = "mysql://pi:pi@localhost/pi"
+# ========================================================
+# You need to change the following values:
+#  1. set the database URIs for LinOTP and privacyIDEA
+#  2. configure, what you want to migrate in the
+#     MIGRATE dictionary
+#  3. Set the ASSIGNMENTS dictionary, to define, which
+#     realms you want to have the tokens in.
+# =========================================================
+LINOTP_URI = "mysql://linotp2:linotp2@localhost/linotp2"
+PRIVACYIDEA_URI = "mysql://pitest:pitest@localhost/pitest"
 
+"""
+Define, what should be migrated:
+
+"tokens" should always be True.
+If "assignments" is False, the imported tokens will be unassigned!
+
+"""
+MIGRATE = {"tokens": True,
+           "tokeninfo": True,
+           "assignments": True}
+
+"""
+This assigns the token to the new resolver
+
+The "resolver" key maps LinOTP-resolvers to privacyIDEA-Resolvers. This value is
+rewritten in the token tabel.
+
+The "realm" key puts tokens in the privacyIDEA-Resolver into this privacyIDEA realm.
+
+In this example the resolvername "lokal" from LinOTP gets imported to
+privacyIDEA with the resolver "PIResolver" in realm "pirealm".
+
+Tokens, that were not assigned to a user will be assigned to the realms "realm1" and "realm2"
+in privacyIDEA.
+"""
+ASSIGNMENTS = {
+    "resolver": {"lokal": "PIResolver"},
+    "realm": {"PIResolver": "pirealm"},
+    "unassigned_tokens": ["realm1", "realm2"]
+}
+
+"""
+This option splits INSERT statements into chunks of a given size, which is useful
+if the number of migrated tokens is so large that it cannot fit in one INSERT
+statement.
+"""
+INSERT_CHUNK_SIZE = 10000
+
+# Do not change anything after this line
+# ============================================================================
+#
 from sqlalchemy import Table, MetaData, Column
 from sqlalchemy import Integer, Unicode, Boolean, UnicodeText
-metadata = MetaData()
-
-token_table = Table("token", metadata,
-                    Column("id", Integer, primary_key=True, nullable=False),
-                    Column("description", Unicode(80), default=u''),
-                    Column("serial", Unicode(40), default=u'', unique=True,
-                           nullable=False, index=True),
-                    Column("tokentype", Unicode(30), default=u'HOTP',
-                           index=True),
-                    Column("user_pin", Unicode(512), default=u''),
-                    Column("user_pin_iv", Unicode(32), default=u''),
-                    Column("so_pin", Unicode(512), default=u''),
-                    Column("so_pin_iv", Unicode(32), default=u''),
-                    Column("resolver", Unicode(120), default=u'', index=True),
-                    Column("resolver_type",  Unicode(120), default=u''),
-                    Column("user_id", Unicode(320), default=u'', index=True),
-                    Column("pin_seed", Unicode(32), default=u''),
-                    Column("otplen", Integer(), default=6),
-                    Column("pin_hash", Unicode(512), default=u''),
-                    Column("key_enc", Unicode(1024), default=u''),
-                    Column("key_iv", Unicode(32), default=u''),
-                    Column("maxfail", Integer(), default=10),
-                    Column("active", Boolean(), nullable=False, default=True),
-                    Column("revoked", Boolean(), default=False),
-                    Column("locked", Boolean(), default=False),
-                    Column("failcount", Integer(), default=0),
-                    Column("count", Integer(), default=0),
-                    Column("count_window", Integer(), default=10),
-                    Column("sync_window", Integer(), default=1000),
-                    Column("rollout_state", Unicode(10), default=u''))
-
-tokeninfo_table = Table("tokeninfo", metadata,
-                  Column("id", Integer, primary_key=True),
-                  Column("Key", Unicode(255), nullable=False),
-                  Column("Value", UnicodeText(), default=u''),
-                  Column("Type", Unicode(100), default=u''),
-                  Column("Description", Unicode(2000), default=u''),
-                  Column("token_id", Integer()))
-
-tokenrealm_table = Table("tokenrealm", metadata,
-                         Column("id", Integer(), primary_key=True),
-                         Column("token_id", Integer()),
-                         Column("realm_id", Integer()))
-
-realm_table = Table("realm", metadata,
-                    Column("id", Integer, primary_key=True),
-                    Column("name", Unicode(255), default=u''),
-                    Column("default",  Boolean(), default=False),
-                    Column("option", Unicode(40), default=u''))
-
-resolver_table = Table("resolver", metadata,
-                       Column("id", Integer, primary_key=True),
-                       Column("name", Unicode(255), default=u""),
-                       Column("rtype", Unicode(255), default=u""))
-
-resolver_config_table = Table("resolverconfig", metadata,
-                              Column("id", Integer, primary_key=True),
-                              Column("resolver_id", Integer),
-                              Column("Key", Unicode(255), default=u""),
-                              Column("Value", Unicode(2000), default=u""),
-                              Column("Type", Unicode(2000), default=u""),
-                              Column("Description", Unicode(2000), default=u""),
-                              )
-
-resolverrealm_table = Table("resolverrealm", metadata,
-                            Column("id", Integer, primary_key=True),
-                            Column("resolver_id", Integer),
-                            Column("realm_id", Integer),
-                            Column("priority", Integer))
+import sys
+import getopt
 
-#
-# LinOTP table definitions
-#
-linotp_config_table = Table("Config", metadata,
-                            Column("Key", Unicode(255)),
-                            Column("Value", Unicode(2000)),
-                            Column("Type", Unicode(2000)),
-                            Column("Description", Unicode(2000))
-                            )
-
-linotp_tokenrealm_table = Table("TokenRealm", metadata,
-                         Column("id", Integer(), primary_key=True),
-                         Column("token_id", Integer()),
-                         Column("realm_id", Integer()))
-
-linotp_realm_table = Table("Realm", metadata,
-                    Column("id", Integer, primary_key=True),
-                    Column("name", Unicode(255), default=u''),
-                    Column("default",  Boolean(), default=False),
-                    Column("option", Unicode(40), default=u''))
-
-linotp_token_table = Table('Token',metadata,
-                           Column('LinOtpTokenId', Integer(),
-                                  primary_key=True, nullable=False),
-                           Column(
-                               'LinOtpTokenDesc', Unicode(80), default=u''),
-                           Column('LinOtpTokenSerialnumber', Unicode(
-                               40), default=u'', unique=True, nullable=False,
-                                  index=True),
-                           Column(
-                               'LinOtpTokenType', Unicode(30), default=u'HMAC',
-                               index=True),
-                           Column(
-                               'LinOtpTokenInfo', Unicode(2000), default=u''),
-                           Column(
-                               'LinOtpTokenPinUser', Unicode(512), default=u''),
-                           Column(
-                               'LinOtpTokenPinUserIV', Unicode(32),
-                               default=u''),
-                           Column(
-                               'LinOtpTokenPinSO', Unicode(512), default=u''),
-                           Column(
-                               'LinOtpTokenPinSOIV', Unicode(32), default=u''),
-                           Column(
-                               'LinOtpIdResolver', Unicode(120), default=u'',
-                               index=True),
-                           Column(
-                               'LinOtpIdResClass', Unicode(120), default=u''),
-                           Column(
-                               'LinOtpUserid', Unicode(320), default=u'',
+
+def migrate():
+
+    # This maps the resolver types. You must not change this!
+    resolver_map = {"LDAPIdResolver": "ldapresolver",
+                    "SQLIdResolver": "sqlresolver",
+                    "PasswdIdResolver": "passwdresolver"}
+
+    token_serial_id_map = {}
+    realm_id_map = {}
+    resolver_id_map = {}
+
+    metadata = MetaData()
+
+    token_table = Table("token", metadata,
+                        Column("id", Integer, primary_key=True, nullable=False),
+                        Column("description", Unicode(80), default=u''),
+                        Column("serial", Unicode(40), default=u'', unique=True,
+                               nullable=False, index=True),
+                        Column("tokentype", Unicode(30), default=u'HOTP',
                                index=True),
-                           Column(
-                               'LinOtpSeed', Unicode(32), default=u''),
-                           Column(
-                               'LinOtpOtpLen', Integer(), default=6),
-                           Column(
-                               'LinOtpPinHash', Unicode(512), default=u''),
-                           Column(
-                               'LinOtpKeyEnc', Unicode(1024), default=u''),
-                           Column(
-                               'LinOtpKeyIV', Unicode(32), default=u''),
-                           Column(
-                               'LinOtpMaxFail', Integer(), default=10),
-                           Column(
-                               'LinOtpIsactive', Boolean(), default=True),
-                           Column(
-                               'LinOtpFailCount', Integer(), default=0),
-                           Column('LinOtpCount', Integer(), default=0),
-                           Column(
-                               'LinOtpCountWindow', Integer(), default=10),
-                           Column(
-                               'LinOtpSyncWindow', Integer(), default=1000)
-                           )
-
-
-from sqlalchemy import create_engine
-from sqlalchemy.orm import sessionmaker
-from sqlalchemy.sql import select
-import json
-
-linotp_engine = create_engine(LINOTP_URI)
-privacyidea_engine = create_engine(PRIVACYIDEA_URI)
-
-linotp_session = sessionmaker(bind=linotp_engine)()
-privacyidea_session = sessionmaker(bind=privacyidea_engine)()
-
-conn_linotp = linotp_engine.connect()
-conn_pi = privacyidea_engine.connect()
-
-resolver_map = {"LDAPIdResolver": "ldapresolver",
-                "SQLIdResolver": "sqlresolver",
-                "PasswdIdResolver": "passwdresolver"}
-
-# Process Tokens
-
-s = select([linotp_token_table])
-result = conn_linotp.execute(s)
-for r in result:
-    print("migrating token {0!s}".format(r["LinOtpTokenSerialnumber"]))
-    # Adapt type
-    type = r["LinOtpTokenType"]
-    if type.lower() == "hmac":
-        type = "HOTP"
-    # Adapt resolver
-    resolver = r["LinOtpIdResolver"].split(".")[-1]
-
-    # Adapt resolver type
-    resolver_type = ""
-    if r["LinOtpIdResClass"]:
-        resolver_type = r['LinOtpIdResClass'].split(".")[1]
-        resolver_type = resolver_map.get(resolver_type)
-
-    # Adapt tokeninfo
-    ti = {}
-    if r["LinOtpTokenInfo"]:
-        ti = json.loads(r["LinOtpTokenInfo"])
-
-    ins = token_table.insert().values(
-        id=r["LinOtpTokenId"],
-        description=r["LinOtpTokenDesc"],
-        serial=r["LinOtpTokenSerialnumber"],
-        tokentype=type,
-        user_pin=r['LinOtpTokenPinUser'],
-        user_pin_iv=r['LinOtpTokenPinUserIV'],
-        so_pin=r['LinOtpTokenPinSO'],
-        so_pin_iv=r['LinOtpTokenPinSOIV'],
-        resolver=resolver,
-        resolver_type=resolver_type,
-        user_id=r['LinOtpUserid'],
-        pin_seed=r['LinOtpSeed'],
-        key_enc=r['LinOtpKeyEnc'],
-        key_iv=r['LinOtpKeyIV'],
-        maxfail=r['LinOtpMaxFail'],
-        active=r['LinOtpIsactive'],
-        failcount=r['LinOtpFailCount'],
-        count=r['LinOtpCount'],
-        count_window=r['LinOtpCountWindow'],
-        sync_window=r['LinOtpSyncWindow'])
-    conn_pi.execute(ins)
-
-    if ti:
-        # Add tokeninfo for this token
-        for k, v in ti.iteritems():
-            ins = tokeninfo_table.insert().values(
-                Key=k, Value=v, token_id=r["LinOtpTokenId"])
-            conn_pi.execute(ins)
-            print(" +--- adding tokeninfo {0!s}".format(k))
-
-# Process Realms
-
-s = select([linotp_realm_table])
-result = conn_linotp.execute(s)
-for r in result:
-    print("migrating realm {0!s}".format(r["name"]))
-    ins = realm_table.insert().values(
-        id=r["id"],
-        name=r["name"],
-        default=r["default"],
-        option=r["option"]
-    )
-    conn_pi.execute(ins)
-
-# Process Tokenrealms
-
-s = select([linotp_tokenrealm_table])
-result = conn_linotp.execute(s)
-for r in result:
-    print("migrating tokenrealm {0!s}".format(r["id"]))
-    ins = tokenrealm_table.insert().values(
-        id=r["id"],
-        token_id=r["token_id"],
-        realm_id=r["realm_id"]
-    )
-    conn_pi.execute(ins)
-
-# Process Resolvers
-
-s = select([linotp_config_table])
-result = conn_linotp.execute(s)
-resolvers = {}
-for r in result:
-    config_key = r["Key"]
-    # TODO: We need to supprt the other resolvers!
-    entry = config_key.split(".")
-    if entry[1] == "ldapresolver":
-        name = entry[3]
-        resolver_key = entry[2]
-        value = r["Value"]
-        desc = r["Description"]
-        type = r["Type"]
-
-        if name not in resolvers:
-            resolvers[name] = {"type": "ldapresolver",
-                               "config": [{"Key": "CACHE_TIMEOUT",
-                                           "Value": "120"},
-                                          {"Key": "SCOPE",
-                                           "Value": "SUBTREE",
-                                           "Type": "string"},
-                                          {"Key": "EDITABLE",
-                                           "Value": "0",
-                                           "Type": "bool"}]}
-        resolvers[name]["config"].append({
-            "Key": resolver_key,
-            "Value": value,
-            "Description": desc,
-            "Type": type
-        })
-
-if resolvers:
-    # Write the resolvers
-    for name, config in resolvers.iteritems():
-        print("Migrating Resolver {0!s}".format(name))
-
-        ins = resolver_table.insert().values(
-                name=name,
-                rtype=config.get("type")
-        )
-        res_id = conn_pi.execute(ins)
-        resolver_id = res_id.inserted_primary_key[0]
-        for resolver_entry in config.get("config"):
-            # save the list of resolver configuration
-            ins = resolver_config_table.insert().values(
-                resolver_id=resolver_id,
-                Key=resolver_entry.get("Key") or "",
-                Value=resolver_entry.get("Value") or "",
-                Type=resolver_entry.get("Type") or "",
-                Description=resolver_entry.get("Description") or ""
-            )
-            conn_pi.execute(ins)
-
-
-# Put resolvers into realms
-s = select([linotp_config_table])
-result = conn_linotp.execute(s)
-realms = {}
-for r in result:
-    config_key = r["Key"]
-    entry = config_key.split(".")
-    if entry[1] == "useridresolver" and entry[2] == "group":
-        resolvers = r["Value"].split()
-        realms[entry[3]] = [x.split(".")[3] for x in resolvers]
-
-print("Migrating realm definitions {0}".format(realms))
-
-# Save the resolvers in the realms
-"""For this we need to get the IDs of the realmname and the resolvernames"""
-for realmname, resolvers in realms.iteritems():
-    print("Migrating realm {0!s}".format(realmname))
-    s = select([realm_table]).where(realm_table.c.name == realmname)
-    result = conn_pi.execute(s)
-    for r in result:
-        realm_id = r["id"]
-
-    for resolvername in resolvers:
-        s = select([resolver_table]).where(resolver_table.c.name ==
-                                           resolvername)
+                        Column("user_pin", Unicode(512), default=u''),
+                        Column("user_pin_iv", Unicode(32), default=u''),
+                        Column("so_pin", Unicode(512), default=u''),
+                        Column("so_pin_iv", Unicode(32), default=u''),
+                        Column("resolver", Unicode(120), default=u'', index=True),
+                        Column("resolver_type",  Unicode(120), default=u''),
+                        Column("user_id", Unicode(320), default=u'', index=True),
+                        Column("pin_seed", Unicode(32), default=u''),
+                        Column("otplen", Integer(), default=6),
+                        Column("pin_hash", Unicode(512), default=u''),
+                        Column("key_enc", Unicode(1024), default=u''),
+                        Column("key_iv", Unicode(32), default=u''),
+                        Column("maxfail", Integer(), default=10),
+                        Column("active", Boolean(), nullable=False, default=True),
+                        Column("revoked", Boolean(), default=False),
+                        Column("locked", Boolean(), default=False),
+                        Column("failcount", Integer(), default=0),
+                        Column("count", Integer(), default=0),
+                        Column("count_window", Integer(), default=10),
+                        Column("sync_window", Integer(), default=1000),
+                        Column("rollout_state", Unicode(10), default=u''))
+
+    tokeninfo_table = Table("tokeninfo", metadata,
+                      Column("id", Integer, primary_key=True),
+                      Column("Key", Unicode(255), nullable=False),
+                      Column("Value", UnicodeText(), default=u''),
+                      Column("Type", Unicode(100), default=u''),
+                      Column("Description", Unicode(2000), default=u''),
+                      Column("token_id", Integer()))
+
+    tokenrealm_table = Table("tokenrealm", metadata,
+                             Column("id", Integer(), primary_key=True),
+                             Column("token_id", Integer()),
+                             Column("realm_id", Integer()))
+
+    realm_table = Table("realm", metadata,
+                        Column("id", Integer, primary_key=True),
+                        Column("name", Unicode(255), default=u''),
+                        Column("default",  Boolean(), default=False),
+                        Column("option", Unicode(40), default=u''))
+
+    resolver_table = Table("resolver", metadata,
+                           Column("id", Integer, primary_key=True),
+                           Column("name", Unicode(255), default=u""),
+                           Column("rtype", Unicode(255), default=u""))
+
+    resolver_config_table = Table("resolverconfig", metadata,
+                                  Column("id", Integer, primary_key=True),
+                                  Column("resolver_id", Integer),
+                                  Column("Key", Unicode(255), default=u""),
+                                  Column("Value", Unicode(2000), default=u""),
+                                  Column("Type", Unicode(2000), default=u""),
+                                  Column("Description", Unicode(2000), default=u""),
+                                  )
+
+    resolverrealm_table = Table("resolverrealm", metadata,
+                                Column("id", Integer, primary_key=True),
+                                Column("resolver_id", Integer),
+                                Column("realm_id", Integer),
+                                Column("priority", Integer))
+
+    #
+    # LinOTP table definitions
+    #
+
+    linotp_token_table = Table('Token',metadata,
+                               Column('LinOtpTokenId', Integer(),
+                                      primary_key=True, nullable=False),
+                               Column(
+                                   'LinOtpTokenDesc', Unicode(80), default=u''),
+                               Column('LinOtpTokenSerialnumber', Unicode(
+                                   40), default=u'', unique=True, nullable=False,
+                                      index=True),
+                               Column(
+                                   'LinOtpTokenType', Unicode(30), default=u'HMAC',
+                                   index=True),
+                               Column(
+                                   'LinOtpTokenInfo', Unicode(2000), default=u''),
+                               Column(
+                                   'LinOtpTokenPinUser', Unicode(512), default=u''),
+                               Column(
+                                   'LinOtpTokenPinUserIV', Unicode(32),
+                                   default=u''),
+                               Column(
+                                   'LinOtpTokenPinSO', Unicode(512), default=u''),
+                               Column(
+                                   'LinOtpTokenPinSOIV', Unicode(32), default=u''),
+                               Column(
+                                   'LinOtpIdResolver', Unicode(120), default=u'',
+                                   index=True),
+                               Column(
+                                   'LinOtpIdResClass', Unicode(120), default=u''),
+                               Column(
+                                   'LinOtpUserid', Unicode(320), default=u'',
+                                   index=True),
+                               Column(
+                                   'LinOtpSeed', Unicode(32), default=u''),
+                               Column(
+                                   'LinOtpOtpLen', Integer(), default=6),
+                               Column(
+                                   'LinOtpPinHash', Unicode(512), default=u''),
+                               Column(
+                                   'LinOtpKeyEnc', Unicode(1024), default=u''),
+                               Column(
+                                   'LinOtpKeyIV', Unicode(32), default=u''),
+                               Column(
+                                   'LinOtpMaxFail', Integer(), default=10),
+                               Column(
+                                   'LinOtpIsactive', Boolean(), default=True),
+                               Column(
+                                   'LinOtpFailCount', Integer(), default=0),
+                               Column('LinOtpCount', Integer(), default=0),
+                               Column(
+                                   'LinOtpCountWindow', Integer(), default=10),
+                               Column(
+                                   'LinOtpSyncWindow', Integer(), default=1000)
+                               )
+
+
+    from sqlalchemy import create_engine
+    from sqlalchemy.orm import sessionmaker
+    from sqlalchemy.sql import select
+    import json
+
+    linotp_engine = create_engine(LINOTP_URI)
+    privacyidea_engine = create_engine(PRIVACYIDEA_URI)
+
+    linotp_session = sessionmaker(bind=linotp_engine)()
+    privacyidea_session = sessionmaker(bind=privacyidea_engine)()
+
+    conn_linotp = linotp_engine.connect()
+    conn_pi = privacyidea_engine.connect()
+
+    def insert_chunks(conn, table, values, chunk_size=INSERT_CHUNK_SIZE):
+        """
+        Split **values** into chunks of size **chunk_size** and insert them sequentially.
+        """
+        values_length = len(values)
+        for i in xrange(0, values_length, chunk_size):
+            print 'Insert records {} to {} ...'.format(i, min(i + chunk_size, values_length) - 1)
+            conn.execute(table.insert(), values[i:i+chunk_size])
+
+
+    # Values to be imported
+    token_values = []
+    tokeninfo_values = []
+    tokenrealm_values = []
+
+    warnings = []
+
+    # Process Assignments in table "tokenrealm"
+    if MIGRATE.get("assignments"):
+        s = select([realm_table])
         result = conn_pi.execute(s)
         for r in result:
-            resolver_id = r["id"]
-        # insert each resolver_id for the realm_id
-        ins = resolverrealm_table.insert().values(
-            realm_id=realm_id,
-            resolver_id=resolver_id
-        )
-        print(" +--- Adding resolver {0!s} to realm {1!s}".format(
-            resolvername, realmname))
-        conn_pi.execute(ins)
+            realm_id_map[r["name"]] = r["id"]
 
+        s = select([resolver_table])
+        result = conn_pi.execute(s)
+        for r in result:
+            resolver_id_map[r["name"]] = r["id"]
+
+        print("Realm-Map: {}".format(realm_id_map))
+        print("Resolver-Map: {}".format(resolver_id_map))
+
+    # Process Tokens
+
+    if MIGRATE.get("tokens"):
+        s = select([linotp_token_table])
+        result = conn_linotp.execute(s)
+
+        i = 0
+        for r in result:
+            i = i + 1
+            print("processing token #{1!s}: {0!s}".format(r["LinOtpTokenSerialnumber"], i))
+            # Adapt type
+            type = r["LinOtpTokenType"]
+            if type.lower() == "hmac":
+                type = "HOTP"
+            # Adapt resolver
+            linotp_resolver = r["LinOtpIdResClass"].split(".")[-1]
+
+            # Adapt resolver type
+            resolver_type = ""
+            if r["LinOtpIdResClass"]:
+                resolver_type = r['LinOtpIdResClass'].split(".")[1]
+                resolver_type = resolver_map.get(resolver_type)
+
+            # Adapt tokeninfo
+            ti = {}
+            if r["LinOtpTokenInfo"]:
+                ti = json.loads(r["LinOtpTokenInfo"])
+
+            if MIGRATE.get("assignments"):
+                user_pin = r['LinOtpTokenPinUser']
+                user_pin_iv = r['LinOtpTokenPinUserIV']
+                # Map the LinOTP-Resolver to the PI-Resolver
+                resolver = ASSIGNMENTS.get("resolver").get(linotp_resolver)
+                if not resolver and linotp_resolver:
+                    warnings.append(u"No mapping defined for the LinOTP resolver: {0!s}".format(linotp_resolver))
+                resolver_type = resolver_type
+                user_id = r['LinOtpUserid']
+            else:
+                user_pin = None
+                user_pin_iv = None
+                resolver = None
+                resolver_type = None
+                user_id = None
+
+            token_values.append(dict(
+                description=r["LinOtpTokenDesc"],
+                serial=r["LinOtpTokenSerialnumber"],
+                tokentype=type,
+                user_pin=user_pin,
+                user_pin_iv=user_pin_iv,
+                so_pin=r['LinOtpTokenPinSO'],
+                so_pin_iv=r['LinOtpTokenPinSOIV'],
+                resolver=resolver,
+                resolver_type=resolver_type,
+                user_id=user_id,
+                pin_seed=r['LinOtpSeed'],
+                pin_hash=r['LinOtpPinHash'],
+                key_enc=r['LinOtpKeyEnc'],
+                key_iv=r['LinOtpKeyIV'],
+                maxfail=r['LinOtpMaxFail'],
+                active=r['LinOtpIsactive'],
+                failcount=r['LinOtpFailCount'],
+                count=r['LinOtpCount'],
+                count_window=r['LinOtpCountWindow'],
+                sync_window=r['LinOtpSyncWindow']))
+
+            if MIGRATE.get("tokeninfo") and ti:
+                # Add tokeninfo for this token
+                for k, v in ti.iteritems():
+                    tokeninfo_values.append(dict(
+                        serial=r["LinOtpTokenSerialnumber"],
+                        Key=k, Value=v,
+                        token_id=r["LinOtpTokenId"]))
+                    print(" +--- processing tokeninfo {0!s}".format(k))
+
+        print
+        print("Adding {} tokens...".format(len(token_values)))
+        insert_chunks(conn_pi, token_table, token_values)
+
+        # fetch the new token_id's in privacyIDEA and write them to the
+        # token serial id map.
+        s = select([token_table])
+        result = conn_pi.execute(s)
+        for r in result:
+            token_serial_id_map[r["serial"]] = r["id"]
+
+        # rewrite the id's in the token_values list
+        for i in range(0, len(token_values)):
+            token_values[i]["id"] = token_serial_id_map[token_values[i]["serial"]]
+
+        if MIGRATE.get("tokeninfo"):
+            # Now we have to rewrite the token_id in the tokeninfo_values
+            for ti in tokeninfo_values:
+                ti["token_id"] = token_serial_id_map[ti["serial"]]
+                del ti["serial"]
+
+            print("Adding {} token infos...".format(len(tokeninfo_values)))
+            insert_chunks(conn_pi, tokeninfo_table, tokeninfo_values)
+
+    if MIGRATE.get("assignments"):
+        # If the token is assigned, we also need to create an entry for tokenrealm
+        # We need to determine the realm_id for this resolver!
+        for token in token_values:
+            token_id = token.get("id")
+            resolver = token.get("resolver")
+            if resolver:
+                realm = ASSIGNMENTS.get("realm").get(resolver)
+                realm_id = realm_id_map.get(realm)
+                print("Assigning token {} for resolver {} to realm_id {} (realm {})".format(token_id,
+                                                                                            resolver,
+                                                                                            realm_id,
+                                                                                            realm))
+                tokenrealm_values.append(dict(token_id=token_id,
+                                              realm_id=realm_id))
+            else:
+                # The token has no resolver and thus is not assigned
+                for tokenrealm in ASSIGNMENTS.get("unassigned_tokens", []):
+                    realm_id = realm_id_map.get(tokenrealm)
+                    if realm_id:
+                        tokenrealm_values.append(dict(token_id=token_id,
+                                                      realm_id=realm_id))
+
+        print("Adding {} tokenrealms...".format(len(tokenrealm_values)))
+        insert_chunks(conn_pi, tokenrealm_table, tokenrealm_values)
+
+    if warnings:
+        print("We need to inform you about the following WARNINGS:")
+        for warning in warnings:
+            print(warning)
+
+
+def usage():
+    print("You need to configure some mapping within the script and then pass the parameter '-c'.")
+
+
+try:
+    opts, args = getopt.getopt(sys.argv[1:], "c", ["configured"])
+except getopt.GetoptError as e:
+    print(str(e))
+    sys.exit(1)
+
+for o, a in opts:
+    if o in ("-c", "--configured"):
+        migrate()
+        sys.exit(0)
+
+usage()
+sys.exit(1)


Test patch: