Package: anybox.migration.openerp
WarningReminder
This tool is complex and highly experimental and is distributed in the hope
that itmightbe useful, but WITHOUT ANY WARRANTY of FITNESS FOR A
PARTICULAR PURPOSE. In particular, if you’re using OpenERP for your company,
you should consider purchasing an OpenERP Enterprise contract that will provide
you with the best and riskless migration path for your database and custom
developments.ContentsInstallationWith virtualenvWith BuildoutUsageInternalsMapping fileCopying dataCopying all columns of a tableCopying all columns to a different tableCopying everythingSplitting one source line to several tables (table splitting)Data moved to another table (table merging)Not migrating a columnTransforming data with Python codeFeeding a new columnMerging with existing dataForeign keys without constraintsReference fieldsHandle cyclic dependant tablesrunning SQL requests during migrationField size limitOverall migration processBefore migrationMigrationAfter migrationUnderstanding errorsContributeAuthors and contributorsCode repository and bug trackerChanges0.9 (2013-12-31)0.8 (2013-09-30)0.7 (2013-09-24)0.6 (2013-08-24)0.5 (2013-08-02)0.4 (2013-07-28)0.3 (2013-07-11)0.2 (2013-07-01)This tool has been developped with these initial goals in mind, in this
priority order:Merging2 different OpenERP databases into a single multicompany dbMigratingfrom OpenERP 6.1 to 7.0Migrating data from a legacy business application (Access, Delphi, etc.)Migrating from Dolibarr to OpenERPThe principle of this tool is to export CSV data from an old application (only
OpenERP for now), then to process CSV files in order to import them into a
freshly installed OpenERP database. This is a completely different strategy
than the in-place migration of OpenERP or OpenUpgrade and allows to start over
from a clean database, while keeping history.  Import and export are done with
the PostgreSQL-specific COPY command, and results in extremely fast exports and
imports. Combined with a pure in-memory Python csv processing, this tool can
often achieve overall migration rates over 2000 lines/sec.InstallationThis tool only works withPython 2.7!With virtualenv$ virtualenv sandbox
$ sandbox/bin/pip install anybox.migration.openerp
$ sandbox/bin/migrate -hWith BuildoutIf you’re using Buildout, you may add this tool in a new part like this:[migration]
recipe = zc.recipe.egg
eggs = anybox.migration.openerpThen, don’t forget to add themigrationsection in thepartsof the[buildout]section. After relaunchingbin/buildout, themigratescript will appear in thebindirectory of the buildout.Starting from the version 1.7 of theOpenERP buildout recipeYou may also install
this tool directly in your[openerp]part, just by adding:eggs = anybox.migration.openerp
scripts = migrateUsageThis tool offers a singlemigratescript:$ migrate -hYou can list the available default mapping files:$ migrate -l
openerp6.1-openerp7.0.ymlYou should specify the source and target DBs, a selection of the source tables
to migrate, and the mapping files to use.  The tool then takes care of
selecting the dependant tables:$ migrate -s source_dbname -t target_dbname -r res_partner account_move -p openerp6.1-openerp7.0.yml custom.ymlIf you want to inspect the temporary CSV files created, use the--keepcsvoption. They will be stored in a temporary directory under the current
directory.This script won’t actually write anything in the target database unless you
specify the-woption to commit the transaction at the end.The most important part of the migration process is the YML mapping file, which
describes how to handle data, table by table and column by column. A default
mapping file is provided and is being used as a real mapping for a migration
consisting in migrating two 6.1 databases into a single 7.0 multicompany
database.  You can mix the default 6.1 to 7.0 file provided, and augment it
with other custom yml files, they will be merged.InternalsThis tool was very loosely inspired from:the external_referential OpenERP modulethe OpenUpgrade projectTalend Open StudioThe different internal steps are:Exporting CSV from the old databaseTransforming CSV to match the target databaseDetect data existing in the target DB with discriminatorsPostprocessing CSV files to fix foreign keysReinjecting into OpenERPUpdating possible pre-existing data with incoming dataThe processing of CSV files is done using a mapping file written in Yaml.
Initial versions of the mapping file have been written with the help of the
OpenUpgrade analysis files.Mapping fileYou should keep in mind that this migration tool is only dealing with database
tables and columns: the OpenERP fields are unknown to it. Each table,
each line, each cell of the source database is handled independently and the
mapping file tells what to do with the current cell. This leads to limitations
and this tool won’t be able to handle extremely complex migration.  But it
is powerful enough to allow to simultaneously merge and migrate two 6.1
databases into a 7.0 multicompany database.For a real-life example, you can have a look at the OpenERP 6.1 to 7.0 mapping
file provided in themappingsdirectory of this tool.Copying dataThe most simple and basic YML statement for a column mapping is the following:module:
    table1.column1:
        table2.column2: __copy__It tells that, if the OpenERPmoduleis installed in thetargetdatabase, thecolumn1of thetable1from the source DB should be copied
to thecolumn2of thetable2in the target DB.The__copy__instruction can even be omitted and the previous statement is
equivalent to this one:module:
    table1.column1:
        table2.column2:Internally, this statement is actually converted to a Python dict:{'module':
    {'table1.column1':
        {'table2.column2': '__copy__'}}And the whole yml file is converted to a large mapping dict whose leafs are
statements or functions which are able to process data.Copying all columns of a tableIf your target table has the same structure as the source table, you can avoid
specifying one mapping statement for each column and use a wildcard:module:
    table1.*:It means: copy all the columns of table1 from the source db to table1 in the
target db.  This kind of mapping is often used as a starting point when source
and table structures are similar. You can then add mapping statements for
specific columns to override this wildcard.Copying all columns to a different tableIf the source table has only been renamed, you can copy all the columns of the
source table1 to the target table2:module:
    table1.*:
        table2.*:Copying everythingIf the source and target db have exactly the same structure and you just want
to transfer data, you may use a global wildcard (but we have not had the
opportunity to try this one for real yet):module:
    .*:It means: copy all tables to the target database without processing. It may
seem unuseful compared to a bare dump and restore, but remind that this way you
can append data to the target DB, not only replace it. In that case you should
take care of existing data, if the table has constraints (see discriminators
below)Splitting one source line to several tables (table splitting)For a single source line coming from a source table, you can feed data in
several target tables. This can be done just by putting several target lines
like this:module:
    table1.column1:
        table2.column2:
        table3.column3:It means: for eachcolumn1in thetable1of the source DB, create two
target lines: one fortable2and one fortable3.During the processing of the current line, other mapping statements
can feed the same target lines. Take this example:module:
    table1.column1:
        table2.column2:
        table3.column3:
    table1.column2:
        table2.column2:
        table3.column4:In this case, data in thetable1will be directed totable2andtable3. You can then add more lines to handle all the columns oftable1However in the example above, there is a conflict since two source cells are directed
to the same target cell (table2.column2). In this scenario, there is no way to
predict which one will be used (because the mapping is a Pythondictand a dict is not
ordered). You should avoid this kind of conflicts.In case of an OpenERP 6.1 to 7.0 migration, this kind of mapping is actually
used to migrate one sourceres_usersline to three different lines: one inres_users+ one inres_partner+ one inmail_alias. See the default
mapping for a real example.Data moved to another table (table merging)When input lines must move to a different table, you want the foreign keys
pointing to them to be kept so that they point to the new table after
migration, you should use the__moved__statement.The only current situation in OpenERP is for theres_partner_addressdata
moving to theres_partnertable:base:
    res_partner_address.id:
        res_partner.id: __moved__This statement must be accompanied with a__fk__statement for all the
foreign keys pointing to the moved table (See the__fk__chapter).Not migrating a columnIf you want to get rid of a specific column in a table, use the__forget__statement:module:
    table1.column1: __forget__This statement is useful if you defined a wildcard, to prevent from migrating a
specific column.Transforming data with Python codeInstead of just copying data with the__copy__statement, you can use any
Python code. The Python code should be written in a literal Yaml block and is
executed as is, as a function body, so that you have to insert areturnstatement somewhere.Example from themailmodule:mail:
    mail_message.type:
        mail_message.type: return 'email'It means thetypecolumn of themail_messagetable will be filled with'email'strings, whatever data the source column had.The eventual signature of the function constructed using the Python code block isdef mapping_function(self, source_row, target_rows):It means that in the function body you can access the fullsource_row,
which is a dict containing all the keys (column names) and values of the
current line being processed. But keep in mind that at this time, you are
dealing with one specific cell of this line, and you should return the value
that will be inserted in the corresponding cell of the target table. This can
be used to aggregate data from two source cells into a target cell:base:
    table1.firstname: __forget__
    table1.name:
        table1.name: return source_row['firstname'] + ' ' + source_row['name']You can also access thetarget_rowsbeeing filled during the processing of
the line, so that data coming from a source cell can influence several cells in
the target lines, or even different target tables. Here is an example:base:
    table1.id:
        table1.id:
        table2.id:
    table1.name:
        table1.name: |
            name = source_row['firstname'] + ' ' + source_row['name']
            target_rows['table1']['display_name'] = name
            target_rows['table2']['display_name'] = name
            return name
        table2.nameNote that in the example above, the Python code spans on several lines, and you
should define a Yaml literal block using|. The example above eventually
means: appendfirstnametonamecoming from thetable1, and put it
in thedisplay_namecell of the targettable1andtable2. The
targetnamecell will contain a copy of the sourcenamecell.If the target line is not supposed to have the sameidas the source line,
you can create a newidwith the newid() function. This function returns a
different value at each call and is responsible of incrementing theid. Here
is an example:base:
    res_users.id:
        res_users.id:
        res_users.partner_id:
        res_partner.notification_email_send: return 'comment'
        res_partner.id: |
            i = newid()
            target_rows['res_users']['partner_id'] = i
            target_rows['res_partner']['id'] = i
            target_rows['res_partner']['name'] = source_row['name']
            target_rows['res_partner']['email'] = source_row['user_email']
            return iEachres_usersline will generate a newres_partnerline with a newid, while theres_usersidwill be the same as the source. (Actually it
will not be the same, because an offset is applied to all ids).Feeding a new columnIf a target column should contain data but has no equivalent in the source
table, you can use ‘_’ as a substitute to the not existing source column name:base:
    res_partner._:
        res_partner.is_company: return FalseMerging with existing dataWhen data is inserted in the target table, you may want to merge it with
existing data.Imagine the targetres_userstable already contains anadminaccount, and you don’t want to duplicate this account by migrating
data from the sourceres_userstable. In this case you should tell the
mapping how to recognize existing data. This is done by replacing the
source column name with the__discriminator__statement, and by providing a
list of column names that will be used to recognize existing data:base:
    res_users.__discriminator__:
        - loginUsing this statement, you can install a new OpenERP database with its admin
account, and merge all existing accounts with data coming from the source
table. Thelogincolumn will be used to match data. The preexistingadminaccount won’t be duplicated but will be updated with theadminaccount from
the source table.Another use case in a multicompany scenario is to merge partners existing in
the target database, but keep them separate for the two companies:base:
    res_partner.__discriminator__:
        - name
        - company_idForeign keys without constraintsThe first step of the migration is to automatically detect all the foreign keys
of the source and target tables. Sometimes, OpenERP defines foreign keys
without constraints. This mainly happens withrelatedfields withstore=True, which create a column of integers without constraints. If you
don’t want to__forget__such columns, you have to tell the mapping what
the target of the foreign key is, like in the real example below:account:
    account_move.company_id:
        account_move.company_id: __fk__ res_companyHere is another example for thecrm_leadtable, which may contain a field
coming from a__moved__table. Imagine you want thepartner_idfield of
the CRM leads in OpenERP 7.0 to come from thepartner_address_idfield of
the same table in OpenERP 6.1.  The new field is a foreign key tores_partner, while the old one was pointing tores_partner_address. You
can tell this with the following statement:crm_lead.partner_address_id:
    crm_lead.partner_id: __fk__ res_partner_addressHowever you should also not forget to forget the partner_id field, or you will
have a conflict an mix data badly if you used a wildcard for the table:crm_lead.*:
crm_lead.partner_id: __forget__Reference fieldsSometimes columns define a dynamic reference id to another table, just like a
foreign key, except that the name of the table is actually stored in another
column.idmodelres_id1cr.claim232cr.claim353base.action.rule27In the example above, since theres_idis not a real foreign key, its value
won’t be fixed to correspond to the target database. In that case you should
use the__ref__statement, followed by the name of the column holding the
table or model name. This statement assumes the model-to-table transformation
of OpenERP is used (replacing ‘.’ with ‘_’):mail_message.res_id:
    mail_message.res_id: __ref__ modelHandle cyclic dependant tablesDuring the last step, the migrated CSV files are imported one by one.  Some
tables depend on other tables through foreign key constraints, and such
dependencies sometimes happen to be cyclic. In that case, there is no way to
import tables because they all depend on another one. One solution is to__forget__the column, which is rarely desirable because you lose data. To
be able to keep such data, you should use the__defer__statement, so that
the column will be updated after all the data is imported:base:
    res_users.create_uid:
        res_users.create_uid: __defer__
    res_users.write_uid:
        res_users.write_uid: __defer__running SQL requests during migrationIn case the wanted migration is too complex to be handled by regular
statements, you can run SQL queries on both the source and target database.
This should be used in limited cases because the queries will be executed for
each source cell for which the mapping defines it, and the migration may be
slowed down, unless you limit the queries with manual caching. (See the
workflow migration in the mapping).A simple sql() function is available in the mapping file, and has the following signature:sql(db, query, args)

where:
- db is the string 'source' or 'target'
- query is the SQL query
- args is the arguments to insert in the query
The query is actually executed with: cursor.execute(query, args)Here is an example:base:
    res_users._:
        (...)
        mail_alias.alias_model_id: return sql('target', "select id from ir_model where model='res.users'")[0][0]Field size limitWhen running migration, you may encounter acsv.Error: field larger than field limit. This is due to the csv module limiting the csv field size to
128k by default. The default value has been increased to 20MB. If this is not
enough for your migration, you can increase the limit by inserting a direct
call tocsv.field_size_limit().For example:module:
    table1.column1:
        table2.column2: |
            import csv
            csv.field_size_limit(262144)
            return source_row['column1']Overall migration processMigrating need several steps described below. If you need, you can easily write
a small script to automate this full process.Before migrationThe different steps before migration are the following. All of them are
important for the migration to be successful and should be done on the target db:Create a clean target database without demo data, using the latest migrated codeInstall the expected modulesRename the target company so that its name exactly match the company in the source databaseRemove the company of all internal sequences by running the following SQL:update ir_sequence set company_id=NULL;. This will allow to remove duplicate after migration.MigrationThe migration consists in running themigratescript by selecting the
correct options. If the data in the target database are not the one you expect,
you must adapt the options and the mapping file to obtain what you want.Here is a real example../bin/migrate -s sourcedb -t targetdb -p openerp6.1-openerp7.0.yml custom.yml
-r res_partner account_move res_users pos_order pos_order_line account_move_line
account_journal sale_order_line stock_inventory_line account_tax
product_supplierinfo wkf_instance wkf_workitem wkf_triggers -wAfter migrationThemigratescript alone may not be sufficient for your database to be
clean and usable.  You may have to handle additional corrections. Please test
your instance thoroughly!  Since version 0.6 you shouldn’t have to manually fix
the internal sequences, as they are now handled by the mapping file. You may
just clean them up to remove duplicates (from the menu
Settings/Technical/Sequences&Identifiers/Sequences).  However one of the
required remaining fixes consists in dropping someparent_leftandparent_rightcolumns. Here is the example with the accounting module:psql targetdb -c 'alter table account_account drop parent_left;'
psql targetdb -c 'alter table account_account drop parent_right;'You might also need to force a recalculation of new or changed related fields
that are persisted in the database (store=True).  Here is an example with
the account_report_company module:psql targetdb -c 'alter table account_invoice drop commercial_partner_id;'At the end, you should run a final global update of the database.
If you’re using thebuildout recipeit should look like this:../bin/start_openerp -u all -d targetdb --stop-after-initUnderstanding errorsThe most difficult part of using this tool is to understand the errors during
the processing, as it requires a deep knowledge of how it internally works.
Most errors generally come from an erroneous mapping file. Errors can happen
during the processing of the CSV files, but the most difficult ones come from
the last import step, because some tables may fail to be imported. In this
case, you should carefully look at the logging messages at the end, and try to
understand the constraint errors or why tables cannot be imported. You also
should use the--keepcsvoption, and inspect the intermediate CSV files to
understand the problem. By using this option, you will end up with a directory
containing five CSV files for each table.For instance, for theres_partnertable you will find these files:res_partner.csvis the original data exported from the source
databaseres_partner.target.csvcontains data after the first processing with
the mapping file, but wrong foreign keysres_partner.target2.csvcontains final data with fixed foreign keys,
that will eventually be imported at the endres_partner.update.csvcontains data which have been detected as
existing in the target database, with wrong foreign keys.res_partner.update2.csvcontains the final existing data with fixed
foreign keys, that will be used to update the target table after import.If you’re going into trouble during the import step with foreign key errors,
please have a look at this log, as it contains most of the common
encountered issues to solve:https://bitbucket.org/anybox/anybox.migration.openerp/issue/3/foreign-key-constraintsContributeAuthors and contributorsChristophe CombellesFlorent JouatteGuy-Clovis NzouendjouStéphane BidoulCode repository and bug trackerSee here:https://bitbucket.org/anybox/anybox.migration.openerpPlease don’t hesitate to give us feedback, report bugs or contribute the mapping files
on Bitbucket.Changes0.9 (2013-12-31)cleaned up and added missing tablesMany mapping improvements, fixes and cleanupAdded mapping for HR migration0.8 (2013-09-30)restored mapping for users/company assignmentimproved message migration0.7 (2013-09-24)Fixed CRM migrationMigrate email_templateMigrate email_messageSupport references with__ref__statementsFixed and improved the mappingIncreased the default csv field size limit to 20MB0.6 (2013-08-24)Migrate ir_sequence without needing post-migration scriptFixed workflow instance and workitem migrationMajor performance improvement (x3) in case of db mergingFixed unwanted merging due to bad offset of foreign key discriminatorsBreak some dependency loops and other mapping improvements0.5 (2013-08-02)Fixed foreign keys pointing to a__moved__table with existing dataUpdated doc0.4 (2013-07-28)Fixed migration of leads and purchase orderssimplified__moved__statement handlingimproved workflow migrationmigrate employees and expensesset suppliers as companies by defaulthow to install in a buildoutupdated doc0.3 (2013-07-11)Lots of improvements for the 6.1 to 7.0 migrationFixed a bug during import due to bad quotingAllow m2o to m2m migration without custom codeAdded mapping for project, crm and auth_ldap modulesFixed move linesAllow to request the source db as wellImproved documentationMigration of running workflows0.2 (2013-07-01)initial release