Package: redturtle.importer.base
Tool to migrate contents between Plone sites based on transmogrifier.This tool works in addition withredturtle.exporter.baseDependenciesThis product is made over other useful tools:collective.jsonmigratorcollective.transmogrifiertransmigrify.dexterityThese tools are not yet actively maintained, so we moved useful parts into this
project to have a working Python 3 importer based on transmogrifier.FeaturesHandle migration for basic content-typesDiscussions migrationCustomizable import procedure via blueprintsExtensible with more specific blueprintsPossibility to customize specific step options with custom adaptersReview view after migration with process resultsInstallationInstall redturtle.importer.base by adding it to your buildout:[buildout]

...

eggs =
    redturtle.importer.baseand then runningbin/buildoutYou don’t have to install it. In this way, after the data migration, you can
remove it from the buildout and everything is clean.UsageMigration viewTo start a migration, you only need to call@@data-migrationview on site root.In this view you can see the blueprint configuration (base and overrided), and start the process.Pipelines customizationThis tool is based on transmogrifier and works with blueprints.
A blueprint is basically a config file that lists all the steps needed for the migration.This product has adefault blueprintfor basic migrations, that can be used as is.Default blueprint can be easily customized using a.migrationconfig.cfgfile located in buildout root folder.In this file you can override already present parts/variables (likepipelinesinto[transmogrifier]section) or
add new ones (for example a new step).For example, catalogsource step can be configured with some queries like this:[catalogsource]
catalog-query = {'portal_type': ['Document', 'Event', 'News Item']}
...In.migrationconfig.cfgfile, under[catalogsource]section, you also need to set some settings about how to retrieve data on source site:[catalogsource]
...
remote-url = http://localhost:8080
remote-root = /Plone
catalog-path = /Plone/portal_catalog
remote-username = username
remote-password = password
...Before running a migration, you can check the final configuration in@@data-migrationview.catalogsource configurationThis is an example of[catalogsource]part:[catalogsource]
...
remote-url = http://localhost:8080
remote-root = /Plone
catalog-path = /Plone/portal_catalog
remote-username = username
remote-password = passwordRequired options are:remote-url: The url of source Plone siteremote-root: The path of Plone site that we want to migrateremote-username: Credentials to access to source siteremote-password: Credentials to access to source siteAdditional options are:default-local-path: A path where save migrate contents in destination Site. This path will replace item’s root path. Destination root path is not needed in this path.skip-private: Boolean to migrate or not private items into destination. Default isFalse.remote_skip_paths: A list of paths from source site that will be skipped during migration process.incremental-migration: Boolean value. If a content already migrate hasn’t been modified since last migration, don’t override it. Default isFalse.ignore-cache: Boolean value. If True, ignore local cache and always get content data from source site.cache-dir: Local folder where migration data cache will be stored. Default is/tmp/migration/migration_cache.migration-dir: Local fodler where migration support files (for final summary for example) will be saved. Default is ‘/tmp/migration’.Custom types mappingcontentsmappingis the section that allows to convert one portal_type to another before object creation.There is a plugin system based on subscribers that allows plugins to add custom mappings.You need to register a subscriber forIPortalTypeMappinglike this:<subscriber
    factory=".types_mapping.MyCustomMapping"
    provides="redturtle.importer.base.interfaces.IPortalTypeMapping"/>And then you need to create the class:@adapter(IPloneSiteRoot, IBrowserRequest)
@implementer(IPortalTypeMapping)
class MyCustomMapping(object):
    order = 100

    def __init__(self, context, request):
        self.context = context
        self.request = request

    def __call__(self, item, typekey):
        """
        """
        portal_type = item[typekey]
        if portal_type == "Type-A":
            item[typekey] = "Type-B"
            ...
        return itemCustom steps for specific portal typesIf you are migrating a content-type that needs some manual fixes after the creation, you can do it with an adapter.You only need to register an adapter for your content-type like this:<adapter
  for="my.product.interfaces.IMyInterface"
  provides="redturtle.importer.base.interfaces.IMigrationContextSteps"
  factory=".steps.MyTypeSteps"
/>And then you need to provide a “doSteps” method in the class:from redturtle.importer.base.interfaces import IMigrationContextSteps
from zope.interface import implementer

@implementer(IMigrationContextSteps)
class MyTypeSteps(object):

    def __init__(self, context):
        self.context = context

    def doSteps(self):
        """
        do something here
        """Example specific importersThere are some per-project importers that we used to migrate some projects and you can use them as a starting point
to develop new ones.They are basically packages that you need to include in your buildout and provides some custom steps for specific types:redturtle.importer.rerredturtle.importer.voltoImport Users and groupsYou can also import users and groups from source site.You only need to add a section to your migration config file like this:[users_and_groups]
import-users = True
import-groups = TrueThe tool will call two views from source site and will use the settings
(remote-url, remote-root and credentials) from[catalogsource]section.This import is performed after transmogrifier steps.ContributeIssue Tracker:https://github.com/RedTurtle/redturtle.importer.base/issuesSource Code:https://github.com/RedTurtle/redturtle.importer.baseCreditsThis product has been developed with some help fromLicenseThe project is licensed under the GPLv2.ContributorsRedTurtle,sviluppoplone@redturtle.itChangelog2.1.0 (2022-04-10)Better handle datetime and date values.
[cekk]Make transmogrifier more configurable.
[cekk]2.0.2 (2021-02-01)Add geolocation deserializer.
[cekk]2.0.1 (2020-12-18)Handle custom destination path.
[cekk]2.0.0 (2020-07-28)Heavy refactoring for python3 compatibility.
[cekk]Allow to import users and groups.
[cekk]Remove unmaintained dependencies and move needed code here.
[cekk]1.0.5 (2019-03-19)fixed fix_link_noreference function.
[eikichi18]1.0.4 (2019-02-08)Added fix for links without any references. Added dedicated report after migration.
[daniele]Added check in schemaupdater for leave field empty when value is empty.
[eikichi18]Fix broken links generation list.
[cekk]1.0.3 (2018-10-18)Added json item to adapters methods.
[daniele]1.0.2 (2018-10-11)Fixed mapping for link internal/external link.
[eikichi18]1.0.1 (2018-10-09)Fix uudi matcher after migration.
[eikichi18]1.0.0 (2018-10-04)Add check if Plone Site element was indexed.Add support for specific context steps with adapters.
[cekk]1.0a4 (2018-09-03)Handle cases where exlude-type is not set.
[cekk]Generate a list of broken links in tinymce after migration,
and expose them in final report view.
[cekk]1.0a3 (2018-07-19)Added check for element’s father data.
[eikichi18]1.0a2 (2018-07-03)Break migration if doesn’t find a content types.
[eikichi18]1.0a1 (2018-06-19)Initial release.
[eikichi18]