Package: ilrt.contentmigrator
ILRT Content MigratorEd Crewe,ILRTat University of Bristol, October 2014The core functionality should allow migration between any Plone versions.The XML format should allow migration to most other CMS, assuming any modern CMS
should cope with XML content import, one way or another.There is a tool to migrate plone exported sites to
Google sites or vice versa, seehttp://sites.google.comEither moving legacy plone sites to free hosting, or increasing the
customisation level of a site, beyond Google sites remit. Either way I hope this
tool may prove mutally beneficial.TODO: Add import/export handling of XML format to the CMIS schema for more
generic standard compliance. Plus maybe Wordpress export format, etc.Seehttp://bitbucket.org/edcrewe/ilrt.contentmigratorfor mercurial source
repository, issue tracker etc.NOTE: Export should work (to some extent) for all plone versions.Updated from plone 3.* version 0.6 to plone 4.* compatible version 1.6
There were no functional changes between 0.6 and 1.6, in terms of the
plone export / import just extra version compatibility tweaks and update of
the test suite.Versions 1.7 - 1.13 Further feature tweaks and XML compatibility changes.Tested with Plone from 2.0 to 4.3.3OverviewThis egg and the companion Product it contains was written to migrate content
from pre-Archetypes plone 2.0 sites (or later) to current plone.The ilrt.contentmigrator egg extends the generic setup content import system
to handle binary files and custom content. Hence a fully populated
site can be generated from file system content held in a profiles structure
folder.The egg follows the paradigm of the existing generic setup, but adds workflow
state to the properties metadata.
It also adds..ini files for each binary content item so that these can
have all their associated metadata imported and exported.It contains a companion old-style plone product. This can be dropped into the
Products folder in an old plone site.
The site gains a portal_exportcontent tool. Running the export from this
tool exports the content to a structure folder in the var directory ready
for using to populate a current plone site, and hence migrate the content.ConceptThe code was arrived at due to the need to migrate a large number of obselete
plone sites and having researched the issue, found that most tools assumed a plone
version within the last few years, where Archetypes, Five, Marshall and XML, or in
place content migration is viable.Instead the code applies the methodology discussed in Andreas Jungs’ blog postingPlone migration fails - doing content-migration onlyUsing the Content MigratorCopy ilrt/contentmigrator/ContentMigrator to the Products directory of
the old plone site. Restart and you should have a ‘Content Migrator Tool’ listed
in the right hand content drop down. Pick this and add it to the portal.There will be a new portal_exportcontent tool in your site. Select this and choose
the Export content tab.
Click export and wait whilst you site becomes files in var/zope/structure
If you only wish to export a subsection of your site then specify the path in the
textbox at the top of the page.Go to your new plone install. Add ilrt.contentmigrator to your buildout config
eggs and zcml sections then run bin/buildout.To do a full import you must first install ilrt.contentmigrator via the quick installer.Add a plone site if you are not importing to an existing one. Go to the ZMI
viahttp://host/Plone/manageand click on the portal_quickinstaller tool
Select Content Migrator Tool Install  check the box and click Install.
You should then have /Plone/portal_setupcontent available. Click on that
to access the migrator interface.Copy (or symlink) the exported structure folder to a profile folder either in the
ilrt.contentmigrator egg or in the main theme egg for your new plone site
and restart, eg. ilrt.contentmigrator/ilrt/contentmigrator/profiles/import/structureExport Formats - CSV, XML, HTMLNote that although there are three export formats only the default CSV format
works for import into a newer Plone site.(CSV format may perhaps better be called YAML these days,
although maybe not strictly compliant to the YAML 1.2 specification)The XML export format was added as a more universal format for use by migration tools into other CMS.
The Addition of the HTML dump was for archival purposes.Further detailsWhen the contentmigrator tool is installed, the content adapter for generic setup will
be modified so that the content import step will now add all content and set workflow states.Hence generic setup, ie Plone/portal_setup is required as the base for this tool.When you go to the new portal_setupcontent tool you can run a further enhanced
version of the generic setup content step that also sets up users, groups and
memberdata and provides fuller logging to screen.
In addition the tool provides access to the exporter so that you can re-export a site
or a subfolder of its content.If you wish to specify another path for the structure folder import just adjust
the directory in the profile that you are using e.g. directory=”c:\import”
in profiles.zcmlIf a default profile is used then generic setup will automatically create
the content when the egg with the profile is reinstalled or selected for Plone site
creation. Where as if another profile is used (such as /import above) then it has to
be manually selected first and then run via the setup tool or this migrator tool.
For large content imports this is likely to be preferable.Standard generic setup runs the adapter in CMFCore.exportimport.content which will
only populate content for HTML documents, and no properties or workflow states will
be added.The ilrt.contentmigrator modifies the generic setup site creation to do the followingPopulate binary content formats and archetypes if matching ones are found.Use Marshall’s RFC822 marshaller to extract and apply the properties data.Apply workflow state transitions.
NB: The workflow migration requires theilrt.migrationtoolegg.Translate old content types and add memberdata (see below)Please note that the import takes much longer than the export.
So for example a Gigabyte of content might only take 5 minutes to export, but take
an hour to import!Content Types TranslationThere is a mapping of old types to newer archetypes for old plone sites.
Currently this just handles ‘Calendar Item’ to ‘Event’ and ‘Link’ to ‘ATLink’.
It is in the ilrt/contentmigrator/ContentMigrator/config.py file.
By modifying the TYPEMAP and NONATPROPS dictionaries of configuration data you can map
other old custom types to new content, or even use it to migrate content from one new type
to another.User migrationThe contentmigrator will also export and import zope held users, including passwords.
It does so by generating the user, roles and groups data from GRUF or
PAS based sites as generic setup xml files in the /structure/acl_users folder.
Memberdata is saved as a csv file for each member in the portal_memberdata directory
within acl_users.Google sites importThe Google sites import is outside of plone.
It just requires command line python with Googles gdata library installed.
Please see the ilrt.contentmigrator/ilrt/contentmigrator/google/README.txt
NB: Currently it doesnt handle custom content typesXML FormatThere is an option to export content as XML for importing into other systems,
for reimporting into current Plone, stick with the default CSV format.Google Data API import / exportEd Crewe, Feb 2011OverviewAdd utility import and export scripts for migrating plone sites to or from
Google sites. Or migrating folders of content to or from Google docs.Initial use case was to archive simple old plone sites which had no money for hosting anymore - so even a static dump would be more costly than exporting to
a free Google site, which also still provides a working CMS.Future use cases may be migrating simple Google sites to Plone when they needed to grow substantially wrt. customisation and features.Requires the install of gdata fromhttp://code.google.com/p/gdata-python-client/into 2.2 or later python. Otherwise it is entirely independent of the plone installation since it just requires or creates the structure dump folder.
So the google folder code is command-line only at this time.UsageCopy demo_config.py to google_config.py and edit it to point at your google account and site.Plone to Google sites exportThis first version just handles google site folders and pages.NB: Google sites have two other content types, announcements (ie simple news / events) and lists (small data customizable content types).Install ilrt.contentmigrator (or ContentMigrator if plone 2) in your plone instance and run the export to a structure directory on the file system.Set up a google site and add your google credentials and the site details to
the google_config.py file along with the path to the exported structure folder then run …> python export_to_google.pyGoogle sites to Plone importThis first version just handles google site folders and pages.For the google site you wish to migrate to plone add the google credentials and the site details to
the google_config.py file along with the path to where you want the structure folder created as IMPORT_FILES then run …> python import_from_google.pyInstall ilrt.contentmigrator in your plone instance and run the import from the structure directory created on the file system by the previous step.Google docs to Plone or vice versaThere is already an application for integrating google docs and plone athttp://pypi.python.org/pypi/collective.googlesharing/1.0.0However it may be useful to have a one off import / export tool too, but for the moment that may remain on my TODO list until I get a real need to do it.Changelog for ilrt.contentmigratorilrt.contentmigrator - 1.14 Released - (2014-11-09)Fixed tests for slight changes in Plone templatesCreated new test account for Google site export - since old one lockedFixed relatedItems causing an unecessary log warning on import[Ed Crewe]ilrt.contentmigrator - 1.13 Released - (2014-10-30)Fix the edge case of folders with content in them with id = ‘data’ not being binary objects[Ed Crewe]ilrt.contentmigrator - 1.12 Released - (2014-04-17)Add TEXT_GETTERS config of custom text methods to check for HTML renderingAdd option to dump out content as HTML, with content type properties as meta tags
For use if the site needs to be generated as static HTML for archival etc.
NB: This export is not a format for re-import[Ed Crewe]Language all specification in query, so foreign content is included[Tessa Alexander]Make sure properties are reset if empty, so it doesnt get wrongly copied to another object[Marek Kralewski]ilrt.contentmigrator - 1.11 Released - (2013-07-22)Add test of textual content types to see if they are actually folders with
hidden items (even if they have isPrincipiaFolderish = False)
Export any hidden items found to the file system as filename.content[Ed Crewe]Update egg dependencies to be compatible with Plone 4.2[Tessa Alexander]ilrt.contentmigrator - 1.10 Released - (2013-05-03)Escape HTML body that already contains CDATA tags for XML format export[Damian Steer]ilrt.contentmigrator - 1.9 Released - (2012-10-26)Close the CDATA tags for XML format export[Ed Crewe]ilrt.contentmigrator - 1.8 Released - (2012-01-04)Manifest failed to include docs directory[Ed Crewe]ilrt.contentmigrator - 1.7 Released - (2011-12-21)Added optional XML format export modeRefactored export to use add element function and doc iostream approach
to more easily cater for format variationsMake sure setText is used with text/html mimetype set[Ed Crewe]ilrt.contentmigrator - 1.6 Released - (2011-02-25)Tested against plone 4.0 - fixed tests and added more version compatibilityFixed issue with ZMI tabs to use consistent zcmlAdded Google Sites migration utility[Ed Crewe]ilrt.contentmigrator - 0.7 (2010-04-07 unreleased)Move code from svn to mercurial and make public on bitbucket[Ed Crewe]ilrt.contentmigrator - 0.6 (2010-02-10)Replace user export page template xml generation with DTML
to be more compatible with old zopeFix adding of empty portal.REQUEST attribute causing error with site when
exporting users from plone 2[Ed Crewe]Use indexObject not reindexObject so modified date is preservedDocument how to change import path[David Sandy]ilrt.contentmigrator - 0.5 (2009-11-20)Export any folderish object by defaultMake user export and reindexing optionalJust log failed object deletions and continueAdd the os.O_BINARY flag to all file writing to stop line ending tampering on Windows[Ed Crewe]Check for all string types when doing export[Dominic Hiles]ilrt.contentmigrator - 0.4Add conversion of old links to archetype links.[Ed Crewe]0.3 release was missing some files, doh![Jerry Van Baren]ilrt.contentmigrator - 0.3Bug fixes of utils AT types conversion methods -
setting empty dates to current date (hence expiring most content!)
and lines fields not converted correctly to tuples.[Ed Crewe]ilrt.contentmigrator - 0.2Contains stand alone ContentMigrator product for exporting content
from old plone sitesUses generic setup style exportimport/content for importing contentImports file and image contentSets workflow state of content (requires ilrt.migrationtool)Imports users, groups and rolesTranslates old calendar item type to ATEvent[Ed Crewe]ilrt.contentmigrator - 0.1 UnreleasedInitial package structure.[zopeskel]TO DOAdd Google site announcements and list types export/import to Plone news and custom typesFix old plone quick installer addition for the ContentMigrator Product part of the eggAdd ATReference handling for export / import like GSXMLFix AT file field handling to work if object also has default file data attributeAdd portrait handling to memberdata export / importMore old plone type translations?Adapt to use for old zope only based content, eg. add html filters to grab
the body text from ZPT content.