Package: collective.exportimport
collective.exportimportExport and import content, members, relations, translations, localroles and much more.Export and import all kinds of data from and to Plone sites using a intermediate json-format.
The main use-case is migrations since it enables you to for example migrate from Plone 4 with Archetypes and Python 2 to Plone 6 with Dexterity and Python 3 in one step.
Most features useplone.restapito serialize and deserialize data.See also the training on migrating withexportimport:https://training.plone.org/migrations/exportimport.htmlContentsFeaturesInstallationPython 2 compatibilityUsageExportImportExport- and import locationsUse-casesMigrationsOtherDetailsExport contentImport contentUse for migrationsControl creating imported contentExport versioned contentNotes on speed and large migrationsFormat of export and import of contentCustomize export and importExport ExampleImport ExampleAutomate export and importFAQ, Tips and TricksUsing global_obj_hook during exportUsing dict-hooks during exportExport/Import placeful workflow policyUsing dict-hooks during importChange workflowExport/Import AnnotationsExport/Import Marker InterfacesSkip versioning during importDealing with validation errorsUse a simple setterDefer importHandle LinguaPlone contentAlternative ways to handle items without parentExport/Import Zope UsersExport/Import properties, registry-settings and installed add-onsExport/Import installed settings and add-onsExport/Import registry settingsMigrate PloneFormGen to EasyformExport and import collective.cover contentFixing invalid collection queriesMigrate to VoltoMigrate very old Plone Versions with data created by collective.jsonifyExport with collective.jsonifyImport with collective.jsonifyTranslationsContributeSupportLicenseWritten byContributorsChangelog1.11 (2024-02-28)1.10 (2023-10-11)1.9 (2023-05-18)1.8 (2023-04-20)1.7 (2023-01-20)1.6 (2022-10-07)1.5 (2022-04-26)1.4 (2022-01-07)1.3 (2021-12-08)1.2 (2021-10-11)1.1 (2021-08-02)1.0 (2021-04-27)1.0b1 (2021-03-26)1.0a2 (2021-03-11)1.0a1 (2021-03-10)FeaturesExport & Import contentExport & Import members and groups with their rolesExport & Import relationsExport & Import translationsExport & Import local rolesExport & Import order (position in parent)Export & Import discussions/commentsExport & Import versioned contentExport & Import redirectsExport supports:Plone 4, 5 and 6Archetypes and DexterityPython 2 and 3plone.app.multilingual, Products.LinguaPlone, raptus.multilanguagefieldsImport supports:Plone 5.2+, Dexterity, Python 2 and 3, plone.app.multilingualInstallationInstall collective.exportimport as you would install any other Python package.You don’t need to activate the add-on in the Site Setup Add-ons control panel to be able to use the forms@@export_contentand@@import_contentin your site.If you need help, see:
- for Plone 4:https://4.docs.plone.org/adapt-and-extend/install_add_ons.html- for Plone 5:https://5.docs.plone.org/manage/installing/installing_addons.html- for Plone 6:https://6.docs.plone.org/install/manage-add-ons-packages.htmlPython 2 compatibilityThis package is compatible with Python 3 and Python 2.
Depending on the Python version different versions of it’s dependencies will be installed.
If you run into problems, file an issue at:https://github.com/collective/collective.exportimport/issuesUsageExportUse the form with the URL/@@export_content, and select what you want to export:You can export one or more types and a whole site or only a specific path in a site. Since items are exported ordered by path importing them will create the same structure as you had originally.The downloaded json-file will have the name of the path you exported from, e.g.Plone.json.The exports for members, relations, localroles and relations are linked to in this form but can also be called individually:/@@export_members,/@@export_relations,/@@export_localroles,/@@export_translations,/@@export_ordering,/@@export_discussion.ImportUse the form with the URL/@@import_content, and upload a json-file that you want to import:The imports for members, relations, localroles and relations are linked to in this form but can also be called individually:/@@import_members,/@@import_relations,/@@import_localroles,/@@import_translations,/@@import_ordering,/@@import_discussion.As a last step in a migration there is another view@@reset_datesthat resets the modified date on imported content to the date initially contained in the imported json-file. This is necessary since varous changes during a migration will likely result in a updated modified-date. During import the original is stored asobj.modification_date_migratedon each new object and this view sets this date.Export- and import locationsIf you select ‘Save to file on server’, the Export view will save json files in the <var> directory of your Plone instanc in /var/instance.
The import view will look for  files under /var/instance/import.
These directories will normally be different, under different Plone instances and possibly on different servers.You can set the environment variable ‘COLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY’ to add a ‘shared’ directory on one server or maybe network share.
With this variable set, collective.exportimport will both save to and load .json files from the same server directory.
This saves time not having to move .json files around from the export- to the import location.
You should be aware that the Export views will overwrite any existing previous .json file export that have the same name.Use-casesMigrationsWhen a in-place-migration is not required you can choose this add-on to migrate the most important parts of your site to json and then import it into a new Plone instance of your targeted version:Export content from a Plone site (it supports Plone 4 and 5, Archetypes and Dexterity, Python 2 and 3).Import the exported content into a new site (Plone 5.2+, Dexterity, Python 3)Export and import relations, users and groups with their roles, translations, local roles, ordering, default-pages, comments, portlets and redirects.How to migrate additional features like Annotations or Marker Interfaces is discussed in the FAQ section.OtherYou can use this add-on toArchive your content as JSON.Export data to prepare a migration to another system.Combine content from multiple plone-sites into one.Import a plone-site as a subsite into another.Import content from other systems as long as it fits the required format.Update or replace existing data.DetailsExport contentExporting content is basically a wrapper for the serializers of plone.restapi:fromplone.restapi.interfacesimportISerializeToJsonfromzope.componentimportgetMultiAdapterserializer=getMultiAdapter((obj,request),ISerializeToJson)data=serializer(include_items=False)Import contentImporting content is a elaborate wrapper for the deserializers of plone.restapi:fromplone.restapi.interfacesimportIDeserializeFromJsonfromzope.componentimportgetMultiAdaptercontainer.invokeFactory(item['@type'],item['id'])deserializer=getMultiAdapter((new,self.request),IDeserializeFromJson)new=deserializer(validate_all=False,data=item)Use for migrationsA main use-case of this package is migration from one Plone-Version to another.Exporting Archetypes content and importing that as Dexterity content works fine but due to changes in field-names some settings would get lost.
For example the setting to exclude content from the navigation was renamed fromexcludeFromNavtoexclude_from_nav.To fix this you can check the checkbox “Modify exported data for migrations”.
This will modify the data during export:Drop unused data (e.g.next_itemandcomponents)Remove all relation fieldsChange some field names that changed between Archetypes and DexterityexcludeFromNav→exclude_from_navallowDiscussion→allow_discussionsubject→subjectsexpirationDate→expireseffectiveDate→effectivecreation_date→createdmodification_date→modifiedstartDate→startendDate→endopenEnd→open_endwholeDay→whole_daycontactEmail→contact_emailcontactName→contact_namecontactPhone→contact_phoneUpdate view names on Folders and Collection that changed since Plone 4.ExportATTopicand their criteria to Collections with querystrings.Update Collection-criteria.Links and images in Richtext-Fields of content and portlets have changes since Plone 4.
the view/@@fix_htmlallows you to fix these.Control creating imported contentYou can choose between four options how to deal with content that already exists:Skip: Don’t import at allReplace: Delete item and create newUpdate: Reuse and only overwrite imported dataIgnore: Create with a new idImported content is initially created withinvokeFactoryusing portal_type and id of the exported item before deserializing the rest of the data.
You can set additional values by specifying a dictfactory_kwargsthat will be passed to the factory.
Like this you can set values on the imported object that are expected to be there by subscribers to IObjectAddedEvent.Export versioned contentExporting versions of Archetypes content will not work because of a bug in plone.restapi (https://github.com/plone/plone.restapi/issues/1335).
For export to work you need to use a version between 7.7.0 and 8.0.0 (if released) or a source-checkout of the branch 7.x.x.Notes on speed and large migrationsExporting and importing large amounts of content can take a while. Export is pretty fast but import is constrained by some features of Plone, most importantly versioning:Importing 5000 Folders takes ~5 minutesImporting 5000 Documents takes >25 minutes because of versioning.Importing 5000 Documents without versioning takes ~7 minutes.During import you can commit every x number of items which will free up memory and disk-space in your TMPDIR (where blobs are added before each commit).When exporting large numbers of blobs (binary files and images) you will get huge json-files and may run out of memory.
You have various options to deal with this.
The best way depends on how you are going to import the blobs:Export as download urls: small download, butcollective.exportimportcannot import the blobs, so you will need an own import script to download them.Export as base-64 encoded strings: large download, butcollective.exportimportcan handle the import.Export as blob paths: small download andcollective.exportimportcan handle the import, but you need to copyvar/blobstorageto the Plone Site where you do the import or set the environment variableCOLLECTIVE_EXPORTIMPORT_BLOB_HOMEto the old blobstorage path:exportCOLLECTIVE_EXPORTIMPORT_BLOB_HOME=/path-to-old-instance/var/blobstorage.
To export the blob-path you do not need to have access to the blobs!Format of export and import of contentBy default all content is exported to and imported from one large json-file.
To inspect such very large json-files without performance-issues you can use klogg (https://klogg.filimonov.dev).Since version 1.10 collective.exportimport also supports exporting and importing each content item as a separate json-file.
To use that selectSave each item as a separate file on the serverin the form or specifydownload_to_server=2when calling the export in python.
In the import-form you can manually select a directory on the server or specifyserver_directory="/mydir"when calling the import in python.Customize export and importThis add-on is designed to be adapted to your requirements and has multiple hooks to make that easy.To make that easier here are packages you can reuse to override and extend the export and import.
Use these templates and adapt them to your own projects:https://github.com/starzel/contentexporthttps://github.com/starzel/contentimportMany examples for customizing the export and import are collected in the chapter “FAQ, Tips and Tricks” below.NoteAs a rule of thumb you should make changes to the data during import unless you need access to the original object for the required changes.
One reason is that this way the serialized content in the json-file more closely represents the original data.
Another reason is that it allows you to fix issues during the process you are currently developing (i.e. without having to redo the export).Export Examplefromcollective.exportimport.export_contentimportExportContentclassCustomExportContent(ExportContent):QUERY={'Document':{'review_state':['published','pending']},}DROP_PATHS=['/Plone/userportal','/Plone/en/obsolete_content',]DROP_UIDS=['71e3e0a6f06942fea36536fbed0f6c42',]defupdate(self):"""Use this to override stuff before the export starts
        (e.g. force a specific language in the request)."""defstart(self):"""Hook to do something before export."""deffinish(self):"""Hook to do something after export."""defglobal_obj_hook(self,obj):"""Inspect the content item before serialisation data.
        Bad: Changing the content-item is a horrible idea.
        Good: Return None if you want to skip this particular object.
        """returnobjdefglobal_dict_hook(self,item,obj):"""Use this to modify or skip the serialized data.
        Return None if you want to skip this particular object.
        """returnitemdefdict_hook_document(self,item,obj):"""Use this to modify or skip the serialized data by type.
        Return the modified dict (item) or None if you want to skip this particular object.
        """returnitemRegister it with your own browserlayer to override the default.<browser:page
    name="export_content"
    for="zope.interface.Interface"
    class=".custom_export.CustomExportContent"
    layer="My.Custom.IBrowserlayer"
    permission="cmf.ManagePortal"
    />Import Examplefromcollective.exportimport.import_contentimportImportContentclassCustomImportContent(ImportContent):CONTAINER={'Event':'/imported-events'}# These fields will be ignoredDROP_FIELDS=['relatedItems']# Items with these uid will be ignoredDROP_UIDS=['04d1477583c74552a7fcd81a9085c620']# These paths will be ignoredDROP_PATHS=['/Plone/doormat/','/Plone/import_files/']# Default values for some fieldsDEFAULTS={'which_price':'normal'}defstart(self):"""Hook to do something before importing one file."""deffinish(self):"""Hook to do something after importing one file."""defglobal_dict_hook(self,item):ifisinstance(item.get('description',None),dict):item['description']=item['description']['data']ifisinstance(item.get('rights',None),dict):item['rights']=item['rights']['data']returnitemdefdict_hook_customtype(self,item):# change the typeitem['@type']='anothertype'# drop a fielditem.pop('experiences',None)returnitemdefhandle_file_container(self,item):"""Use this to specify the container in which to create the item in.
        Return the container for this particular object.
        """returnself.portal['imported_files']Register it:<browser:page
    name="import_content"
    for="zope.interface.Interface"
    class=".custom_import.CustomImportContent"
    layer="My.Custom.IBrowserlayer"
    permission="cmf.ManagePortal"
    />Automate export and importRun all exports and save all data invar/instance/:fromploneimportapifromProducts.FiveimportBrowserViewclassExportAll(BrowserView):def__call__(self):export_content=api.content.get_view("export_content",self.context,self.request)self.request.form["form.submitted"]=Trueexport_content(portal_type=["Folder","Document","News Item","File","Image"],# only export theseinclude_blobs=2,# Export files and images as blob pathsdownload_to_server=True)other_exports=["export_relations","export_members","export_translations","export_localroles","export_ordering","export_defaultpages","export_discussion","export_portlets","export_redirects",]fornameinother_exports:view=api.content.get_view(name,portal,request)# This saves each export in var/instance/export_xxx.jsonview(download_to_server=True)# Important! Redirect to prevent infinite export loop :)returnself.request.response.redirect(self.context.absolute_url())Run all imports using the data exported in the example above:fromcollective.exportimport.fix_htmlimportfix_html_in_content_fieldsfromcollective.exportimport.fix_htmlimportfix_html_in_portletsfrompathlibimportPathfromploneimportapifromProducts.FiveimportBrowserViewclassImportAll(BrowserView):def__call__(self):portal=api.portal.get()# Import contentview=api.content.get_view("import_content",portal,request)request.form["form.submitted"]=Truerequest.form["commit"]=500view(server_file="Plone.json",return_json=True)transaction.commit()# Run all other importsother_imports=["relations","members","translations","localroles","ordering","defaultpages","discussion","portlets","redirects",]cfg=getConfiguration()directory=Path(cfg.clienthome)/"import"fornameinother_imports:view=api.content.get_view(f"import_{name}",portal,request)path=Path(directory)/f"export_{name}.json"results=view(jsonfile=path.read_text(),return_json=True)logger.info(results)transaction.commit()# Run cleanup stepsresults=fix_html_in_content_fields()logger.info("Fixed html for%scontent items",results)transaction.commit()results=fix_html_in_portlets()logger.info("Fixed html for%sportlets",results)transaction.commit()reset_dates=api.content.get_view("reset_dates",portal,request)reset_dates()transaction.commit()NoteThe views@@export_alland@@import_allare also contained in the helper-packageshttps://github.com/starzel/contentexportandhttps://github.com/starzel/contentimportFAQ, Tips and TricksThis section covers frequent use-cases and examples for features that are not required for all migrations.Using global_obj_hook during exportUsingglobal_obj_hookduring export to inspect content and decide to skip it.defglobal_obj_hook(self,obj):# Drop subtopicsifobj.portal_type=="Topic"andobj.__parent__.portal_type=="Topic":return# Drop files and images from PFG formfoldersifobj.__parent__.portal_type=="FormFolder":returnreturnobjUsing dict-hooks during exportUseglobal_dict_hookduring export to inspect content and modify the serialized json.
You can also usedict_hook_<somecontenttype>to better structure your code for readability.Sometimes you need to handle data that you add inglobal_dict_hookduring export in corresponding code inglobal_object_hookduring import.The following example about placeful workflow policy is a perfect example for that pattern:Export/Import placeful workflow policyExport:defglobal_dict_hook(self,item,obj):ifobj.isPrincipiaFolderishand".wf_policy_config"inobj.keys():wf_policy=obj[".wf_policy_config"]item["exportimport.workflow_policy"]={"workflow_policy_below":wf_policy.workflow_policy_below,"workflow_policy_in":wf_policy.workflow_policy_in,}returnitemImport:defglobal_obj_hook(self,obj,item):wf_policy=item.get("exportimport.workflow_policy")ifwf_policy:obj.manage_addProduct["CMFPlacefulWorkflow"].manage_addWorkflowPolicyConfig()wf_policy_config=obj[".wf_policy_config"]wf_policy_config.setPolicyIn(wf_policy["workflow_policy_in"],update_security=True)wf_policy_config.setPolicyBelow(wf_policy["workflow_policy_below"],update_security=True)Using dict-hooks during importA lot of fixes can be done during import using theglobal_dict_hookordict_hook_<contenttype>.Here we prevent the expire-date to be before the effective date since that would lead to validation-errors during deserializing:defglobal_dict_hook(self,item):effective=item.get('effective',None)expires=item.get('expires',None)ifeffectiveandexpiresandexpires<=effective:item.pop('expires')returnitemHere we drop empty lines from the creators:defglobal_dict_hook(self,item):item["creators"]=[iforiinitem.get("creators",[])ifi]returnitemThis example migrates aPloneHelpCenterto a simple folder/document structure during import.
There are a couple more types to handle (as folder or document) but you get the idea, don’t you?defdict_hook_helpcenter(self,item):item["@type"]="Folder"item["layout"]="listing_view"returnitemdefdict_hook_helpcenterglossary(self,item):item["@type"]="Folder"item["layout"]="listing_view"returnitemdefdict_hook_helpcenterinstructionalvideo(self,item):item["@type"]="File"ifitem.get("video_file"):item["file"]=item["video_file"]returnitemdefdict_hook_helpcenterlink(self,item):item["@type"]="Link"item["remoteUrl"]=item.get("url",None)returnitemdefdict_hook_helpcenterreferencemanualpage(self,item):item["@type"]="Document"returnitemIf you change types during import you need to take care of other cases where types are referenced.Examples are collection-queries (see “Fixing invalid collection queries” below) or constrains (see here):PORTAL_TYPE_MAPPING={"Topic":"Collection","FormFolder":"EasyForm","HelpCenter":"Folder",}defglobal_dict_hook(self,item):ifitem.get("exportimport.constrains"):types_fixed=[]forportal_typeinitem["exportimport.constrains"]["locally_allowed_types"]:ifportal_typeinPORTAL_TYPE_MAPPING:types_fixed.append(PORTAL_TYPE_MAPPING[portal_type])elifportal_typeinALLOWED_TYPES:types_fixed.append(portal_type)item["exportimport.constrains"]["locally_allowed_types"]=list(set(types_fixed))types_fixed=[]forportal_typeinitem["exportimport.constrains"]["immediately_addable_types"]:ifportal_typeinPORTAL_TYPE_MAPPING:types_fixed.append(PORTAL_TYPE_MAPPING[portal_type])elifportal_typeinALLOWED_TYPES:types_fixed.append(portal_type)item["exportimport.constrains"]["immediately_addable_types"]=list(set(types_fixed))returnitemChange workflowREVIEW_STATE_MAPPING={"internal":"published","internally_published":"published","obsolete":"private","hidden":"private",}defglobal_dict_hook(self,item):ifitem.get("review_state")inREVIEW_STATE_MAPPING:item["review_state"]=REVIEW_STATE_MAPPING[item["review_state"]]returnitemExport/Import AnnotationsSome core-features of Plone (e.g. comments) use annotations to store data.
The core features are already covered but your custom code or community add-ons may use annotations as well.
Here is how you can migrate them.Export: Only export those Annotations that your really need.fromzope.annotation.interfacesimportIAnnotationsANNOTATIONS_TO_EXPORT=["syndication_settings",]ANNOTATIONS_KEY='exportimport.annotations'classCustomExportContent(ExportContent):defglobal_dict_hook(self,item,obj):item=self.export_annotations(item,obj)returnitemdefexport_annotations(self,item,obj):results={}annotations=IAnnotations(obj)forkeyinANNOTATIONS_TO_EXPORT:data=annotations.get(key)ifdata:results[key]=IJsonCompatible(data,None)ifresults:item[ANNOTATIONS_KEY]=resultsreturnitemImport:fromzope.annotation.interfacesimportIAnnotationsANNOTATIONS_KEY="exportimport.annotations"classCustomImportContent(ImportContent):defglobal_obj_hook(self,obj,item):item=self.import_annotations(obj,item)returnitemdefimport_annotations(self,obj,item):annotations=IAnnotations(obj)forkeyinitem.get(ANNOTATIONS_KEY,[]):annotations[key]=item[ANNOTATIONS_KEY][key]returnitemSome features also store data in annotations on the portal, e.g.plone.contentrules.localassignments,plone.portlets.categoryblackliststatus,plone.portlets.contextassignments,syndication_settings.
Depending on your requirements you may want to export and import those as well.Export/Import Marker InterfacesExport: You may only want to export the marker-interfaces you need.
It is a good idea to inspect a list of all used marker interfaces in a portal before deciding what to migrate.fromzope.interfaceimportdirectlyProvidedByMARKER_INTERFACES_TO_EXPORT=["collective.easyslider.interfaces.ISliderPage","plone.app.layout.navigation.interfaces.INavigationRoot",]MARKER_INTERFACES_KEY="exportimport.marker_interfaces"classCustomExportContent(ExportContent):defglobal_dict_hook(self,item,obj):item=self.export_marker_interfaces(item,obj)returnitemdefexport_marker_interfaces(self,item,obj):interfaces=[i.__identifier__foriindirectlyProvidedBy(obj)]interfaces=[iforiininterfacesifiinMARKER_INTERFACES_TO_EXPORT]ifinterfaces:item[MARKER_INTERFACES_KEY]=interfacesreturnitemImport:fromplone.dexterity.utilsimportresolveDottedNamefromzope.interfaceimportalsoProvidesMARKER_INTERFACES_KEY="exportimport.marker_interfaces"classCustomImportContent(ImportContent):defglobal_obj_hook_before_deserializing(self,obj,item):"""Apply marker interfaces before deserializing."""foriface_nameinitem.pop(MARKER_INTERFACES_KEY,[]):try:iface=resolveDottedName(iface_name)ifnotiface.providedBy(obj):alsoProvides(obj,iface)logger.info("Applied marker interface%sto%s",iface_name,obj.absolute_url())exceptModuleNotFoundError:passreturnobj,itemSkip versioning during importThe event-handlers of versioning can seriously slow down your imports.
It is a good idea to skip it before the import:VERSIONED_TYPES=["Document","News Item","Event","Link",]defstart(self):self.items_without_parent=[]portal_types=api.portal.get_tool("portal_types")forportal_typeinVERSIONED_TYPES:fti=portal_types.get(portal_type)behaviors=list(fti.behaviors)if'plone.versioning'inbehaviors:logger.info(f"Disable versioning for{portal_type}")behaviors.remove('plone.versioning')fti.behaviors=behaviorsRe-enable versioning and create initial versions after all imports and fixes are done, e.g in the view@@import_all.fromProducts.CMFEditions.interfaces.IModifierimportFileTooLargeToVersionErrorVERSIONED_TYPES=["Document","News Item","Event","Link",]classImportAll(BrowserView):# re-enable versioningportal_types=api.portal.get_tool("portal_types")forportal_typeinVERSIONED_TYPES:fti=portal_types.get(portal_type)behaviors=list(fti.behaviors)if"plone.versioning"notinbehaviors:behaviors.append("plone.versioning")logger.info(f"Enable versioning for{portal_type}")if"plone.locking"notinbehaviors:behaviors.append("plone.locking")logger.info(f"Enable locking for{portal_type}")fti.behaviors=behaviorstransaction.get().note("Re-enabled versioning")transaction.commit()# create initial version for all versioned typeslogger.info("Creating initial versions")portal_repository=api.portal.get_tool("portal_repository")brains=api.content.find(portal_type=VERSIONED_TYPES)total=len(brains)forindex,braininenumerate(brains):obj=brain.getObject()try:portal_repository.save(obj=obj,comment="Imported Version")exceptFileTooLargeToVersionError:passifnotindex%1000:msg=f"Created versions for{index}of{total}items."logger.info(msg)transaction.get().note(msg)transaction.commit()msg="Created initial versions"transaction.get().note(msg)transaction.commit()Dealing with validation errorsSometimes you get validation-errors during import because the data cannot be validated.
That can happen when options in a field are generated from content in the site.
In these cases you cannot be sure that all options already exist in the portal while importing the content.It may also happen, when you have validators that rely on content or configuration that does not exist on import.NoteFor relation fields this is not necessary since relations are imported after content anyway!There are two ways to handle these issues:Use a simple setter bypassing the validation used by the restapiDefer the import until all other imports were runUse a simple setterYou need to specify which content-types and fields you want to handle that way.It is put in a key, that the normal import will ignore and set usingsetattr()before deserializing the rest of the data.SIMPLE_SETTER_FIELDS={"ALL":["some_shared_field"],"CollaborationFolder":["allowedPartnerDocTypes"],"DocType":["automaticTransferTargets"],"DPDocument":["scenarios"],"DPEvent":["Status"],}classCustomImportContent(ImportContent):defglobal_dict_hook(self,item):simple={}forfieldnameinSIMPLE_SETTER_FIELDS.get("ALL",[]):iffieldnameinitem:value=item.pop(fieldname)ifvalue:simple[fieldname]=valueforfieldnameinSIMPLE_SETTER_FIELDS.get(item["@type"],[]):iffieldnameinitem:value=item.pop(fieldname)ifvalue:simple[fieldname]=valueifsimple:item["exportimport.simplesetter"]=simpledefglobal_obj_hook_before_deserializing(self,obj,item):"""Hook to modify the created obj before deserializing the data.
        """# import simplesetter data before the restforfieldname,valueinitem.get("exportimport.simplesetter",{}).items():setattr(obj,fieldname,value)NoteUsingglobal_obj_hook_before_deserializingmakes sure that data is there when the event-handlers are run after import.Defer importYou can also wait until all content is imported before setting the values on these fields.
Again you need to find out which fields for which types you want to handle that way.Here the data is stored in an annotation on the imported object from which it is later read.
This example also supports setting some data withsetattrwithout validating it:fromplone.restapi.interfacesimportIDeserializeFromJsonfromzope.annotation.interfacesimportIAnnotationsfromzope.componentimportgetMultiAdapterDEFERRED_KEY="exportimport.deferred"DEFERRED_FIELD_MAPPING={"talk":["somefield"],"speaker":["custom_field","another_field",]}SIMPLE_SETTER_FIELDS={"custom_type":["another_field"]}classCustomImportContent(ImportContent):defglobal_dict_hook(self,item):# Move deferred values to a different key to not deserialize.# This could also be done during export.item[DEFERRED_KEY]={}forfieldnameinDEFERRED_FIELD_MAPPING.get(item["@type"],[]):ifitem.get(fieldname):item[DEFERRED_KEY][fieldname]=item.pop(fieldname)returnitemdefglobal_obj_hook(self,obj,item):# Store deferred data in an annotation.deferred=item.get(DEFERRED_KEY,{})ifdeferred:annotations=IAnnotations(obj)annotations[DEFERRED_KEY]={}forkey,valueindeferred.items():annotations[DEFERRED_KEY][key]=valueYou then need a new step in the migration to move the deferred values from the annotation to the field:classImportDeferred(BrowserView):def__call__(self):# This example reuses the form export_other.pt from collective.exportimportself.title="Import deferred data"ifnotself.request.form.get("form.submitted",False):returnself.index()portal=api.portal.get()self.results=[]forbraininapi.content.find(DEFERRED_FIELD_MAPPING.keys()):obj=brain.getObject()self.import_deferred(obj)api.portal.show_message(f"Imported deferred data for{len(self.results)}items!",self.request)defimport_deferred(self,obj):annotations=IAnnotations(obj,{})deferred=annotations.get(DEFERRED_KEY,None)ifnotdeferred:return# Shortcut for simple fields (e.g. storing strings, uuids etc.)forfieldnameinSIMPLE_SETTER_FIELDS.get(obj.portal_type,[]):value=deferred.pop(fieldname,None)ifvalue:setattr(obj,fieldname,value)ifnotdeferred:return# This approach validates the values and converts more complex datadeserializer=getMultiAdapter((obj,self.request),IDeserializeFromJson)try:obj=deserializer(validate_all=False,data=deferred)exceptExceptionase:logger.info("Error while importing deferred data for%s",obj.absolute_url(),exc_info=True)logger.info("Data:%s",deferred)else:self.results.append(obj.absolute_url())# cleanupdelannotations[DEFERRED_KEY]This additional view obviously needs to be registered:<browser:page
    name="import_deferred"
    for="zope.interface.Interface"
    class=".import_content.ImportDeferred"
    template="export_other.pt"
    permission="cmf.ManagePortal"
    />Handle LinguaPlone contentExport:defglobal_dict_hook(self,item,obj):# Find language of the nearest parent with a language# Usefull for LinguaPlone sites where some content is languageindependentparent=obj.__parent__forancestorinparent.aq_chain:ifIPloneSiteRoot.providedBy(ancestor):# keep language for root contentnearest_ancestor_lang=item["language"]breakifgetattr(ancestor,"getLanguage",None)andancestor.getLanguage():nearest_ancestor_lang=ancestor.getLanguage()item["parent"]["language"]=nearest_ancestor_langbreak# This forces "wrong" languages to the nearest parents languageif"language"initemanditem["language"]!=nearest_ancestor_lang:logger.info(u"Forcing%s(was%s) for%s%s",nearest_ancestor_lang,item["language"],item["@type"],item["@id"])item["language"]=nearest_ancestor_lang# set missing languageifnotitem.get("language"):item["language"]=nearest_ancestor_lang# add info on translations to help find the right container# usually this idone by export_translations# but when migrating from LP to pam you sometimes want to check the# tranlation info during importifgetattr(obj.aq_base,"getTranslations",None)isnotNone:translations=obj.getTranslations()iftranslations:item["translation"]={}forlangintranslations:uuid=IUUID(translations[lang][0],None)ifuuid==item["UID"]:continuetranslation=translations[lang][0]ifnotlang:lang="no_language"item["translation"][lang]=translation.absolute_url()Import:defglobal_dict_hook(self,item):# Adapt this to your sitelanguages=["en","fr","de"]default_language="en"portal_id="Plone"# No language => lang of parent or defaultifitem.get("language")notinlanguages:ifitem["parent"].get("language"):item["language"]=item["parent"]["language"]else:item["language"]=default_languagelang=item["language"]ifitem["parent"].get("language")!=item["language"]:logger.debug(f"Inconsistent lang: item is{lang}, parent is{item['parent'].get('language')}for{item['@id']}")# Move item to the correct language-root-folder# This is only relevant for items in the site-root.# Most items containers are usually looked up by the uuid of the old parenturl=item["@id"]parent_url=item["parent"]["@id"]url=url.replace(f"/{portal_id}/",f"/{portal_id}/{lang}/",1)parent_url=parent_url.replace(f"/{portal_id}",f"/{portal_id}/{lang}",1)item["@id"]=urlitem["parent"]["@id"]=parent_urlreturnitemAlternative ways to handle items without parentOften it is better to export and log items for which no container could be found instead of re-creating the original structure.defupdate(self):self.items_without_parent=[]defcreate_container(self,item):# Override create_container to never create parentsself.items_without_parent.append(item)deffinish(self):# export content without parentsifself.items_without_parent:data=json.dumps(self.items_without_parent,sort_keys=True,indent=4)number=len(self.items_without_parent)cfg=getConfiguration()filename='content_without_parent.json'filepath=os.path.join(cfg.clienthome,filename)withopen(filepath,'w')asf:f.write(data)msg=u"Saved{}items without parent to{}".format(number,filepath)logger.info(msg)api.portal.show_message(msg,self.request)Export/Import Zope UsersBy default only users and groups stores in Plone are exported/imported.
You can export/import Zope user like this.Exportfromcollective.exportimport.export_otherimportBaseExportfromploneimportapiimportsixclassExportZopeUsers(BaseExport):AUTO_ROLES=["Authenticated"]def__call__(self,download_to_server=False):self.title="Export Zope users"self.download_to_server=download_to_serverportal=api.portal.get()app=portal.__parent__self.acl=app.acl_usersself.pms=api.portal.get_tool("portal_membership")data=self.all_zope_users()self.download(data)defall_zope_users(self):results=[]foruserinself.acl.searchUsers():data=self._getUserData(user["userid"])data['title']=user['title']results.append(data)returnresultsdef_getUserData(self,userId):member=self.pms.getMemberById(userId)roles=[roleforroleinmember.getRoles()ifrolenotinself.AUTO_ROLES]# userid, password, rolesprops={"username":userId,"password":json_compatible(self._getUserPassword(userId)),"roles":json_compatible(roles),}returnpropsdef_getUserPassword(self,userId):users=self.acl.userspasswords=users._user_passwordspassword=passwords.get(userId,"")returnpasswordImport:classImportZopeUsers(BrowserView):def__call__(self,jsonfile=None,return_json=False):ifjsonfile:self.portal=api.portal.get()status="success"try:ifisinstance(jsonfile,str):return_json=Truedata=json.loads(jsonfile)elifisinstance(jsonfile,FileUpload):data=json.loads(jsonfile.read())else:raise("Data is neither text nor upload.")exceptExceptionase:status="error"logger.error(e)api.portal.show_message(u"Failure while uploading:{}".format(e),request=self.request,)else:members=self.import_members(data)msg=u"Imported{}members".format(members)api.portal.show_message(msg,self.request)ifreturn_json:msg={"state":status,"msg":msg}returnjson.dumps(msg)returnself.index()defimport_members(self,data):app=self.portal.__parent__acl=app.acl_userscounter=0foritemindata:username=item["username"]password=item.pop("password")roles=item.pop("roles",[])ifnotusernameornotpasswordornotroles:continuetitle=item.pop("title",None)acl.users.addUser(username,title,password)forroleinroles:acl.roles.assignRoleToPrincipal(role,username)counter+=1returncounterExport/Import properties, registry-settings and installed add-onsWhen you migrate multiple similar sites that are configured manually it can be useful to export and import configuration that was set by hand.Export/Import installed settings and add-onsThis custom export exports and imports some selected settings and add-ons from a Plone 4.3 site.Export:fromcollective.exportimport.export_otherimportBaseExportfromloggingimportgetLoggerfromploneimportapifromplone.restapi.serializer.convertersimportjson_compatiblelogger=getLogger(__name__)classExportSettings(BaseExport):"""Export various settings for haiku sites
    """def__call__(self,download_to_server=False):self.title="Export installed add-ons various settings"self.download_to_server=download_to_serverifnotself.request.form.get("form.submitted",False):returnself.index()data=self.export_settings()self.download(data)defexport_settings(self):results={}addons=[]qi=api.portal.get_tool("portal_quickinstaller")forproductinqi.listInstalledProducts():ifproduct["id"].startswith("myproject."):addons.append(product["id"])results["addons"]=addonsportal=api.portal.get()registry={}registry["plone.email_from_name"]=portal.getProperty('email_from_name','')registry["plone.email_from_address"]=portal.getProperty('email_from_address','')registry["plone.smtp_host"]=getattr(portal.MailHost,'smtp_host','')registry["plone.smtp_port"]=int(getattr(portal.MailHost,'smtp_port',25))registry["plone.smtp_userid"]=portal.MailHost.get('smtp_user_id')registry["plone.smtp_pass"]=portal.MailHost.get('smtp_pass')registry["plone.site_title"]=portal.titleportal_properties=api.portal.get_tool("portal_properties")iprops=portal_properties.imaging_propertiesregistry["plone.allowed_sizes"]=iprops.getProperty('allowed_sizes')registry["plone.quality"]=iprops.getProperty('quality')site_props=portal_properties.site_propertiesifsite_props.hasProperty("webstats_js"):registry["plone.webstats_js"]=site_props.webstats_jsresults["registry"]=json_compatible(registry)returnresultsImport:The import installs the add-ons and load the settings in the registry.
Since Plone 5portal_propertiesis no longer used.fromloggingimportgetLoggerfromploneimportapifromplone.registry.interfacesimportIRegistryfromProducts.CMFPlone.utilsimportget_installerfromProducts.FiveimportBrowserViewfromzope.componentimportgetUtilityfromZPublisher.HTTPRequestimportFileUploadimportjsonlogger=getLogger(__name__)classImportSettings(BrowserView):"""Import various settings"""def__call__(self,jsonfile=None,return_json=False):ifjsonfile:self.portal=api.portal.get()status="success"try:ifisinstance(jsonfile,str):return_json=Truedata=json.loads(jsonfile)elifisinstance(jsonfile,FileUpload):data=json.loads(jsonfile.read())else:raise("Data is neither text nor upload.")exceptExceptionase:status="error"logger.error(e)api.portal.show_message("Failure while uploading:{}".format(e),request=self.request,)else:self.import_settings(data)msg="Imported addons and settings"api.portal.show_message(msg,self.request)ifreturn_json:msg={"state":status,"msg":msg}returnjson.dumps(msg)returnself.index()defimport_settings(self,data):installer=get_installer(self.context)foraddonindata["addons"]:ifnotinstaller.is_product_installed(addon)andinstaller.is_product_installable(addon):installer.install_product(addon)logger.info(f"Installed addon{addon}")registry=getUtility(IRegistry)forkey,valueindata["registry"].items():registry[key]=valuelogger.info(f"Imported record{key}:{value}")Export/Import registry settingsThe pull-requesthttps://github.com/collective/collective.exportimport/pull/130has views@@export_registryand@@import_registry.
These views export and import registry records that do not use the default-setting specified in the schema for that registry record.
The export alone could also be usefull to figure out which settings were modified for a site.That code will probably not be merged but you can use it in your own projects.Migrate PloneFormGen to EasyformTo be able to export PFG as easyform you should use the branchmigration_features_1.xofcollective.easyformin your old site.
Easyform does not need to be installed, we only need the methodsfields_modelandactions_model.Export:defdict_hook_formfolder(self,item,obj):item["@type"]="EasyForm"item["is_folderish"]=Falsefromcollective.easyform.migration.fieldsimportfields_modelfromcollective.easyform.migration.actionsimportactions_model# this does most of the heavy lifting...item["fields_model"]=fields_model(obj)item["actions_model"]=actions_model(obj)# handle thankspagepfg_thankspage=obj.get(obj.getThanksPage(),None)ifpfg_thankspage:item["thankstitle"]=pfg_thankspage.titleitem["thanksdescription"]=pfg_thankspage.Description()item["showAll"]=pfg_thankspage.showAllitem["showFields"]=pfg_thankspage.showFieldsitem["includeEmpties"]=pfg_thankspage.includeEmptiesitem["thanksPrologue"]=json_compatible(pfg_thankspage.thanksPrologue.raw)item["thanksEpilogue"]=json_compatible(pfg_thankspage.thanksEpilogue.raw)# optionalitem["exportimport._inputStorage"]=self.export_saved_data(obj)# Drop some PFG fields no longer neededobsolete_fields=["layout","actionAdapter","checkAuthenticator","constrainTypesMode","location","thanksPage",]forkeyinobsolete_fields:item.pop(key,None)# optional: disable tabs for imported formsitem["form_tabbing"]=False# fix some custom validatorsreplace_mapping={"request.form['":"request.form['form.widgets.","request.form.get('":"request.form.get('form.widgets.","member and member.id or ''":"member and member.getProperty('id', '') or ''",}# fix overrides in actions and fields to use form.widgets.xyz instead of xyzforschemain["actions_model","fields_model"]:forold,newinreplace_mapping.items():ifoldinitem[schema]:item[schema]=item[schema].replace(old,new)# add your own fields if you have these issues...forfieldnamein["email","replyto",]:if"request/form/{}".format(fieldname)initem[schema]:item[schema]=item[schema].replace("request/form/{}".format(fieldname),"python: request.form.get('form.widgets.{}')".format(fieldname))returnitemdefexport_saved_data(self,obj):actions={}fordata_adapterinobj.objectValues("FormSaveDataAdapter"):data_adapter_name=data_adapter.getId()actions[data_adapter_name]={}cols=data_adapter.getColumnNames()column_count_mismatch=Falseforidx,rowinenumerate(data_adapter.getSavedFormInput()):iflen(row)!=len(cols):column_count_mismatch=Truelogger.debug("Column count mismatch at row%s",idx)continuedata={}forkey,valueinzip(cols,row):data[key]=json_compatible(value)id_=int(time()*1000)whileid_inactions[data_adapter_name]:# avoid collisions during exportid_+=1data["id"]=id_actions[data_adapter_name][id_]=dataifcolumn_count_mismatch:logger.info("Number of columns does not match for all rows. Some data were skipped in ""data adapter%s/%s","/".join(obj.getPhysicalPath()),data_adapter_name,)returnactionsImport exportedPloneFormGendata intoEasyform:defobj_hook_easyform(self,obj,item):ifnotitem.get("exportimport._inputStorage"):returnfromcollective.easyform.actionsimportSavedDataBTreefrompersistent.mappingimportPersistentMappingifnothasattr(obj,'_inputStorage'):obj._inputStorage=PersistentMapping()forname,datainitem["exportimport._inputStorage"].items():obj._inputStorage[name]=SavedDataBTree()forkey,rowindata.items():obj._inputStorage[name][int(key)]=rowExport and import collective.cover contentExport:fromcollective.exportimport.serializerimportget_dx_blob_pathfromplone.app.textfield.valueimportRichTextValuefromplone.namedfile.fileimportNamedBlobImagefromplone.restapi.interfacesimportIJsonCompatiblefromz3c.relationfieldimportRelationValuefromzope.annotation.interfacesimportIAnnotationsdefglobal_dict_hook(self,item,obj):item=self.handle_cover(item,obj)returnitemdefhandle_cover(self,item,obj):ifICover.providedBy(obj):item['tiles']={}annotations=IAnnotations(obj)fortileinobj.get_tiles():annotation_key='plone.tiles.data.{}'.format(tile['id'])annotation=annotations.get(annotation_key,None)ifannotationisNone:continuetile_data=self.serialize_tile(annotation)tile_data['type']=tile['type']item['tiles'][tile['id']]=tile_datareturnitemdefserialize_tile(self,annotation):data={}forkey,valueinannotation.items():ifisinstance(value,RichTextValue):value=value.rawelifisinstance(value,RelationValue):value=value.to_object.UID()elifisinstance(value,NamedBlobImage):blobfilepath=get_dx_blob_path(value)ifnotblobfilepath:continuevalue={"filename":value.filename,"content-type":value.contentType,"size":value.getSize(),"blob_path":blobfilepath,}data[key]=IJsonCompatible(value,None)returndataImport:fromcollectionsimportdefaultdictfromcollective.exportimport.import_contentimportget_absolute_blob_pathfromplone.app.textfield.interfacesimportIRichTextfromplone.app.textfield.interfacesimportIRichTextValuefromplone.namedfile.fileimportNamedBlobImagefromplone.namedfile.interfacesimportINamedBlobImageFieldfromplone.tiles.interfacesimportITileTypefromzope.annotation.interfacesimportIAnnotationsfromzope.componentimportgetUtilitiesForfromzope.schemaimportgetFieldsInOrderCOVER_CONTENT=["collective.cover.content",]defglobal_obj_hook(self,obj,item):ifitem["@type"]inCOVER_CONTENTand"tiles"initem:item=self.import_tiles(obj,item)defimport_tiles(self,obj,item):RICHTEXT_TILES=defaultdict(list)IMAGE_TILES=defaultdict(list)fortile_name,tile_typeingetUtilitiesFor(ITileType):forfieldname,fieldingetFieldsInOrder(tile_type.schema):ifIRichText.providedBy(field):RICHTEXT_TILES[tile_name].append(fieldname)ifINamedBlobImageField.providedBy(field):IMAGE_TILES[tile_name].append(fieldname)annotations=IAnnotations(obj)prefix="plone.tiles.data."foruid,tileinitem["tiles"].items():# TODO: Maybe create all tiles that do not need to be defferred?key=prefix+uidtile_name=tile.pop("type",None)# first set raw dataannotations[key]=item["tiles"][uid]forfieldnameinRICHTEXT_TILES.get(tile_name,[]):raw=annotations[key][fieldname]ifrawisnotNoneandnotIRichTextValue.providedBy(raw):annotations[key][fieldname]=RichTextValue(raw,"text/html","text/x-html-safe")forfieldnameinIMAGE_TILES.get(tile_name,[]):data=annotations[key][fieldname]ifdataisnotNone:blob_path=data.get("blob_path")ifnotblob_path:continueabs_blob_path=get_absolute_blob_path(obj,blob_path)ifnotabs_blob_path:logger.info("Blob path%sfor tile%sof%s%sdoes not exist!",blob_path,tile,obj.portal_type,obj.absolute_url())continue# Determine the class to use: file or image.filename=data["filename"]content_type=data["content-type"]# Write the field.withopen(abs_blob_path,"rb")asmyfile:blobdata=myfile.read()image=NamedBlobImage(data=blobdata,contentType=content_type,filename=filename,)annotations[key][fieldname]=imagereturnitemFixing invalid collection queriesSome queries changes between Plone 4 and 5.
This fixes the issues.The actual migration of topics to collections incollective.exportimport.serializer.SerializeTopicToJsondoes not (yet) take care of that.classCustomImportContent(ImportContent):defglobal_dict_hook(self,item):ifitem["@type"]in["Collection","Topic"]:item=self.fix_query(item)deffix_query(self,item):item["@type"]="Collection"query=item.pop("query",[])ifnotquery:logger.info("Drop item without query:%s",item["@id"])returnfixed_query=[]indexes_to_fix=["portal_type","review_state","Creator","Subject",]operator_mapping={# old -> new"plone.app.querystring.operation.selection.is":"plone.app.querystring.operation.selection.any","plone.app.querystring.operation.string.is":"plone.app.querystring.operation.selection.any",}forcritinquery:ifcrit["i"]=="portal_type"andlen(crit["v"])>30:# Criterion is all typescontinueifcrit["o"].endswith("relativePath")andcrit["v"]=="..":# relativePath no longer accepts ..crit["v"]="..::1"ifcrit["i"]inindexes_to_fix:forold_operator,new_operatorinoperator_mapping.items():ifcrit["o"]==old_operator:crit["o"]=new_operatorifcrit["i"]=="portal_type":# Some types may have changed their namesfixed_types=[]forportal_typeincrit["v"]:fixed_type=PORTAL_TYPE_MAPPING.get(portal_type,portal_type)fixed_types.append(fixed_type)crit["v"]=list(set(fixed_types))ifcrit["i"]=="review_state":# Review states may have changed their namesfixed_states=[]forreview_stateincrit["v"]:fixed_state=REVIEW_STATE_MAPPING.get(review_state,review_state)fixed_states.append(fixed_state)crit["v"]=list(set(fixed_states))ifcrit["o"]=="plone.app.querystring.operation.string.currentUser":crit["v"]=""fixed_query.append(crit)item["query"]=fixed_queryifnotitem["query"]:logger.info("Drop collection without query:%s",item["@id"])returnreturnitemMigrate to VoltoYou can reuse the migration-code provided by@@migrate_to_voltoinplone.voltoin a migration.
The following example (used for migratinghttps://plone.orgto Volto) can be used to migrate a site from any older version to Plone 6 with Volto.You need to have the Blocks Conversion Tool (https://github.com/plone/blocks-conversion-tool) running that takes care of migrating richtext-values to Volto-blocks.Seehttps://6.docs.plone.org/backend/upgrading/version-specific-migration/migrate-to-volto.htmlfor more details on the changes the migration to Volto does.fromApp.configimportgetConfigurationfrombs4importBeautifulSoupfromcollective.exportimport.fix_htmlimportfix_html_in_content_fieldsfromcollective.exportimport.fix_htmlimportfix_html_in_portletsfromcontentimport.interfacesimportIContentimportLayerfromloggingimportgetLoggerfrompathlibimportPathfromploneimportapifromplone.volto.browser.migrate_to_voltoimportmigrate_richtext_to_blocksfromplone.volto.setuphandlersimportadd_behaviorfromplone.volto.setuphandlersimportremove_behaviorfromProducts.CMFPlone.utilsimportget_installerfromProducts.FiveimportBrowserViewfromzope.interfaceimportalsoProvidesimportrequestsimporttransactionlogger=getLogger(__name__)DEFAULT_ADDONS=[]classImportAll(BrowserView):def__call__(self):request=self.request# Check if Blocks-conversion-tool is runningheaders={"Accept":"application/json","Content-Type":"application/json",}r=requests.post("http://localhost:5000/html",headers=headers,json={"html":"<p>text</p>"})r.raise_for_status()# Submit a simple form template to trigger the importifnotrequest.form.get("form.submitted",False):returnself.index()portal=api.portal.get()alsoProvides(request,IContentimportLayer)installer=get_installer(portal)ifnotinstaller.is_product_installed("contentimport"):installer.install_product("contentimport")# install required add-onsforaddoninDEFAULT_ADDONS:ifnotinstaller.is_product_installed(addon):installer.install_product(addon)# Fake the target being a classic site even though plone.volto is installed...# 1. Allow Folders and Collections (they are disabled in Volto by default)portal_types=api.portal.get_tool("portal_types")portal_types["Collection"].global_allow=Trueportal_types["Folder"].global_allow=True# 2. Enable richtext behavior (otherwise no text will be imported)fortype_in["Document","News Item","Event"]:add_behavior(type_,"plone.richtext")transaction.commit()cfg=getConfiguration()directory=Path(cfg.clienthome)/"import"# Import contentview=api.content.get_view("import_content",portal,request)request.form["form.submitted"]=Truerequest.form["commit"]=500view(server_file="Plone.json",return_json=True)transaction.commit()# Run all other importsother_imports=["relations","members","translations","localroles","ordering","defaultpages","discussion","portlets",# not really useful in Volto"redirects",]fornameinother_imports:view=api.content.get_view(f"import_{name}",portal,request)path=Path(directory)/f"export_{name}.json"ifpath.exists():results=view(jsonfile=path.read_text(),return_json=True)logger.info(results)transaction.get().note(f"Finished import_{name}")transaction.commit()else:logger.info(f"Missing file:{path}")# Optional: Run html-fixers on richtextfixers=[anchor_fixer]results=fix_html_in_content_fields(fixers=fixers)msg="Fixed html for{}content items".format(results)logger.info(msg)transaction.get().note(msg)transaction.commit()results=fix_html_in_portlets()msg="Fixed html for{}portlets".format(results)logger.info(msg)transaction.get().note(msg)transaction.commit()view=api.content.get_view("updateLinkIntegrityInformation",portal,request)results=view.update()msg=f"Updated linkintegrity for{results}items"logger.info(msg)transaction.get().note(msg)transaction.commit()# Rebuilding the catalog is necessary to prevent issues later oncatalog=api.portal.get_tool("portal_catalog")logger.info("Rebuilding catalog...")catalog.clearFindAndRebuild()msg="Finished rebuilding catalog!"logger.info(msg)transaction.get().note(msg)transaction.commit()# This uses the blocks-conversion-tool to migrate to blockslogger.info("Start migrating richtext to blocks...")migrate_richtext_to_blocks()msg="Finished migrating richtext to blocks"transaction.get().note(msg)transaction.commit()# Reuse the migration-form from plon.volto to do some more tasksview=api.content.get_view("migrate_to_volto",portal,request)# Yes, wen want to migrate default pagesview.migrate_default_pages=Trueview.slate=Truelogger.info("Start migrating Folders to Documents...")view.do_migrate_folders()msg="Finished migrating Folders to Documents!"transaction.get().note(msg)transaction.commit()logger.info("Start migrating Collections to Documents...")view.migrate_collections()msg="Finished migrating Collections to Documents!"transaction.get().note(msg)transaction.commit()reset_dates=api.content.get_view("reset_dates",portal,request)reset_dates()transaction.commit()# Disallow folders and collections againportal_types["Collection"].global_allow=Falseportal_types["Folder"].global_allow=False# Disable richtext behavior againfortype_in["Document","News Item","Event"]:remove_behavior(type_,"plone.richtext")returnrequest.response.redirect(portal.absolute_url())defanchor_fixer(text,obj=None):"""Remove anchors since they are not supported by Volto yet"""soup=BeautifulSoup(text,"html.parser")forlinkinsoup.find_all("a"):ifnotlink.get("href")andnotlink.text:# drop empty links (e.g. anchors)link.decompose()elifnotlink.get("href")andlink.text:# drop links without a href but keep the textlink.unwrap()returnsoup.decode()Migrate very old Plone Versions with data created by collective.jsonifyVersions older than Plone 4 do not supportplone.restapiwhich is required to serialize the content used bycollective.exportimport.To migrate Plone 1, 2 and 3 to Plone 6 you can usecollective.jsonifyfor the export andcollective.exportimportfor the import.Export with collective.jsonifyUsehttps://github.com/collective/collective.jsonifyto export content.You include the methods ofcollective.jsonifyusingExternal Methods.
Seehttps://github.com/collective/collective.jsonify/blob/master/docs/install.rstfor more info.To work better withcollective.exportimportyou could extend the exported data using the featureadditional_wrappers.
Add info on the parent of an item to make it easier forcollective.exportimportto import the data.Here is a full example forjson_methods.pywhich should be inBUILDOUT_ROOT/parts/instance/Extensions/defextend_item(obj,item):"""Extend to work better well with collective.exportimport"""fromAcquisitionimportaq_parentparent=aq_parent(obj)item["parent"]={"@id":parent.absolute_url(),"@type":getattr(parent,"portal_type",None),}ifgetattr(parent.aq_base,"UID",None)isnotNone:item["parent"]["UID"]=parent.UID()returnitemHere is a full example forjson_methods.pywhich should be in<BUILDOUT_ROOT>/parts/instance/Extensions/fromcollective.jsonify.exportimportexport_contentasexport_content_origfromcollective.jsonify.exportimportget_itemEXPORTED_TYPES=["Folder","Document","News Item","Event","Link","Topic","File","Image","RichTopic",]EXTRA_SKIP_PATHS=["/Plone/archiv/","/Plone/do-not-import/",]# Path from which to continue the export.# The export walks the whole site respecting the order.# It will ignore everything untill this path is reached.PREVIOUS=""defexport_content(self):returnexport_content_orig(self,basedir="/var/lib/zope/json",skip_callback=skip_item,extra_skip_classname=[],extra_skip_id=[],extra_skip_paths=EXTRA_SKIP_PATHS,batch_start=0,batch_size=10000,batch_previous_path=PREVIOUSorNone,)defskip_item(item):"""Return True if the item should be skipped"""portal_type=getattr(item,"portal_type",None)ifportal_typenotinEXPORTED_TYPES:returnTruedefextend_item(obj,item):"""Extend to work better well with collective.exportimport"""fromAcquisitionimportaq_parentparent=aq_parent(obj)item["parent"]={"@id":parent.absolute_url(),"@type":getattr(parent,"portal_type",None),}ifgetattr(parent.aq_base,"UID",None)isnotNone:item["parent"]["UID"]=parent.UID()returnitemTo use these create three “External Method” in the ZMI root at the Zope root to use that:id: “export_content”, module name: “json_methods”, function name: “export_content”id: “get_item”, module name: “json_methods”, function name: “get_item”id: “extend_item”, module name: “json_methods”, function name: “extend_item”Then you can pass the extender to the export using a query-string:http://localhost:8080/Plone/export_content?additional_wrappers=extend_itemImport with collective.jsonifyTwo issues need to be dealt with to allowcollective.exportimportto import the data generated bycollective.jsonify.The data is in directories instead of in one large json-file.The json is not in the expected format.Starting with version 1.8 you can pass an iterator to the import.You need to create a directory-walker that sorts the json-files the right way.
By default it would import them in the order1.json,10.json,100.json,101.jsonand so on.frompathlibimportPathdeffilesystem_walker(path=None):root=Path(path)assert(root.is_dir())folders=sorted([iforiinroot.iterdir()ifi.is_dir()andi.name.isdecimal()],key=lambdai:int(i.name))forfolderinfolders:json_files=sorted([iforiinfolder.glob("*.json")ifi.stem.isdecimal()],key=lambdai:int(i.stem))forjson_fileinjson_files:logger.debug("Importing%s",json_file)item=json.loads(json_file.read_text())item["json_file"]=str(json_file)item=prepare_data(item)ifitem:yielditemThe walker takes the path to be the root with one or more directories holding the json-files.
The sorting of the files is done using the number in the filename.The methodprepare_datamodifies the data before passing it to the import.
A very similar task is done bycollective.exportimportduring export.defprepare_data(item):"""modify jsonify data to work with c.exportimport"""# Drop relationfields or defer the importitem.pop("relatedItems",None)mapping={# jsonify => exportimport"_uid":"UID","_type":"@type","_path":"@id","_layout":"layout",# AT fieldnames => DX fieldnames"excludeFromNav":"exclude_from_nav","allowDiscussion":"allow_discussion","subject":"subjects","expirationDate":"expires","effectiveDate":"effective","creation_date":"created","modification_date":"modified","startDate":"start","endDate":"end","openEnd":"open_end","eventUrl":"event_url","wholeDay":"whole_day","contactEmail":"contact_email","contactName":"contact_name","contactPhone":"contact_phone","imageCaption":"image_caption",}forold,newinmapping.items():item=migrate_field(item,old,new)ifitem.get("constrainTypesMode",None)==1:item=migrate_field(item,"constrainTypesMode","constrain_types_mode")else:item.pop("locallyAllowedTypes",None)item.pop("immediatelyAddableTypes",None)item.pop("constrainTypesMode",None)if"id"notinitem:item["id"]=item["_id"]returnitemdefmigrate_field(item,old,new):ifitem.get(old,_marker)isnot_marker:item[new]=item.pop(old)returnitemYou can pass the generatorfilesystem_walkerto the import:classImportAll(BrowserView):def__call__(self):# ...cfg=getConfiguration()directory=Path(cfg.clienthome)/"import"# import contentview=api.content.get_view("import_content",portal,request)request.form["form.submitted"]=Truerequest.form["commit"]=1000view(iterator=filesystem_walker(directory/"mydata"))# import default-pagesimport_deferred=api.content.get_view("import_deferred",portal,request)import_deferred()classImportDeferred(BrowserView):def__call__(self):self.title="Import Deferred Settings (default pages)"ifnotself.request.form.get("form.submitted",False):returnself.index()forbraininapi.content.find(portal_type="Folder"):obj=brain.getObject()annotations=IAnnotations(obj)ifDEFERRED_KEYnotinannotations:continuedefault=annotations[DEFERRED_KEY].pop("_defaultpage",None)ifdefaultanddefaultinobj:logger.info("Setting%sas default page for%s",default,obj.absolute_url())obj.setDefaultPage(default)ifnotannotations[DEFERRED_KEY]:annotations.pop(DEFERRED_KEY)api.portal.show_message("Done",self.request)returnself.index()collective.jsonifyputs the info on relations, translations and default-pages in the export-file.
You can use the approach to defer imports to deal with that data after all items were imported.
The exampleImportDeferredabove uses that approach to set the default pages.Thisglobal_obj_hookbelow stores that data in a annotation:defglobal_obj_hook(self,obj,item):# Store deferred data in an annotation.keys=["_defaultpage"]data={}forkeyinkeys:ifvalue:=item.get(key,None):data[key]=valueifdata:annotations=IAnnotations(obj)annotations[DEFERRED_KEY]=dataTranslationsThis product has been translated intoSpanishContributeIssue Tracker:https://github.com/collective/collective.exportimport/issuesSource Code:https://github.com/collective/collective.exportimportSupportIf you are having issues, please let us know.LicenseThe project is licensed under the GPLv2.Written byContributorsPhilip Bauer,bauer@starzel.deMaurits van Rees,m.van.rees@zestsoftware.nlFred van Dijk,f.van.dijk@zestsoftware.nlLeonardo J. Caballero G.,leonardocaballero@gmail.comChangelog1.11 (2024-02-28)FixAtributeError: 'NamedFile' object has no attribute '_blob'when using setting
“Include blobs as blob paths” and exporting objects with
plone.namedfile.file.NamedFile properties (so not blobs).
[valipod]Add more Python 2 compatible version specifications and update the README.
[thet]FixKeyError: timewhen importing content with a workflow that does not have thetimevariable.
[maurits]Allow to use fix_html_in_content_fields without applying the default html_fixer.
[pbauer]Try to restore broken blobs when exporting content.
[thet]When exporting into separate JSON files write also the error in a separate errors.json file.
This fixes an error at the end of the export and no errors being written.
[thet]Add support for ATTopic export_content
[avoinea]Add principals to groups that already exist during import (#228)
[pbauer]In export_members ignore transitive membership of groups (#240)
[pbauer]1.10 (2023-10-11)Don’t re-usemappingvariable when migrating portlet data.
[witsch]Fix editing revision author - refs #216
[avoinea]Better support for portal import which avoids parsing JSON twice.
[gotcha]Migrate portlets on site root.
[ThibautBorn]Support export & import to have one separate json-file per content item.
[pbauer]1.9 (2023-05-18)Allow passing custom filenames to exports
[pbauer]Support export and import of Plone Site root (using update strategy).
[pbauer]Fix blob export when Connection uses TmpStore
[gotcha, pbauer]Fix portlet richtext field import
[mpeeters]Add portlet location on exported data
[mpeeters]Migrate root of portlets that used a path in plone4 to using a uid (navigation, search, events, collection).
[pbauer]Make export of discussions and portlets contextual
[mpeeters]Fix critical bug when importing groups: Do not import groups that a groups belongs to as members of the new group.
This could have caused groups to have more privileges than they should.
[pbauer]1.8 (2023-04-20)Import: run set_uuid method before we call custom hooks, so the hooks have access to
the item UUID. Fix #185.Document COLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY in README.
[fredvd]Add Spanish translation.
[macagua]Add i18n support.
[macagua]Fix html: improve mapping from scale to picture variant.  [maurits]Allow overriding the fallback variant in img_variant_fixer.
Use ‘medium’ by default.
[maurits]Let fix_html view work on the current context.  [maurits]Fix the way we get a blob path. (#180)
[ale-rt]Create documents as containers for items without parent when documents are folderish.
[JeffersonBledsoe]Add support for passing any iterator as data-source to the import.
[pbauer]Add example for importing collective.jsonify data to documentation.
[pbauer]Better serialization of Topics:
- Use newer criteria added in Plone 5
- Add fallback for some criteria
- Export sort_on and sort_reversed
- Export customView as tabular_view
[pbauer]Always import discussions independent if discussion support is enabled or not
on a particular content object (#182)
[ajung]1.7 (2023-01-20)Filter out ‘Discussion Item’ in content type export list. Comments have their own export and
import views. A normal content type export for comments will raise a KeyError when trying to find
the parent. (#112)
[fredvd]Be more specific in the import_translation endpoint condition to install in a site with p.a.multilingual 1.x
[erral]Fix importing hidden portlets as visible. (#152)
[pbauer]UseLanguage=allwhen querying TranslationGroup items
[erral]Fix members import, by handling members that already exist.
[sunew]Don’t use new_id because a hook can changeitem["id"][pbauer]Support exporting the blob-path without having access to the blobs.
[pbauer]Set image-variants in html-fields when running @@fix_html targeting in Plone 6.
[pbauer]1.6 (2022-10-07)Export and import all group-members (including ldap-users and -groups).
Previously it only exported users and groups created in Plone.
[pbauer]Support importing content without a UUID (e.g. for importing from an external source).
The minimal required data is @id, @type, id, and @parent[“@id”].
[pbauer]Export only value when serializing vocabulary-based fields instead of token/title.
[pbauer]Improve logging of errors during import.
[pbauer]Add INCLUDE_PATHS to specify which paths only should be imported.
[pbauer]Add import_review_state to allow overriding setting the review_state during import.
[pbauer]Export parent UID and use it to find the container to import.
[pbauer]Move the various export-hooks into update_export_data for readability.
[pbauer]Support export to server by passingdownload_to_server=Truefor all exports (#115).
[pbauer]Add support for adding custom html-fixers to fix_html_in_content_fields.
[pbauer]1.5 (2022-04-26)Fix AttributeError for getPhysicalPath when checking parent, issue 123.
[maurits]Export and import redirection tool data.
[gotcha, Michael Penninck]Serialize Products.TALESField fields as raw instead of evaluated expression.
(useful to export PFG overrides)
[sauzher]Make sure we never change a acquired modification_date or creation_date.
[pbauer]Export and import workflow_history.
[pbauer]Fail gracefully on errors during importing portlets.
[pbauer]Ignore containers where content should be imported to that are non-folderish.
[pbauer]Use catalog instead of ZopeFindAndApply and better logging for export_discussion.
[pbauer]Add converter for long ints (py2 only).
[pbauer]By default no not export linkintegrity relations.
[pbauer]Log detailed exception when exporting content fails.
[pbauer]Add start and finish hooks for export of content.
[pbauer]Rewrite export/import of default pages: Use uuid of default-page instead of id.
Rewrite getting default_page to fix various issues with translated content.
[pbauer]Add export and import of versions/revisions of content (#105).
[pbauer]1.4 (2022-01-07)Fixdebugflag inExportRelations[petschki]Deserialize portlet-data using restapi to fix importing RichText.
[pbauer]Fix importing richtext with html-entities. Fixes #99
[pbauer]Preserve links to browser-views by using a custom find_object. Fixes #97
[pbauer]Ignore linkintegrity when importing items with replace-strategy.
[pbauer]Add tests for fix_html.
[pbauer]1.3 (2021-12-08)Handle default page of the site root object.
[fulv]Optionally (checkbox) skip existing content on import instead of generating it new with a randomized id.
[petschki]FixUnboundLocalErrorwhen callingimport_contentwithreturn_jsonandserver_file.
[petschki]Add option to make a commit every x items.
[pbauer]Improve logging during import in vairous cases.
[pbauer]Work around case where api.content.get(path=parent_path) raises NotFound instead of returning None.
[pbauer]Keep value of import_to_current_folder.
[pbauer]Fix html unescape in py3.
[pbauer]Fix serializing ATNewsItem image field content.
[gotcha]Migrate eventUrl to event_url (AT to DX).
[ThibautBorn]Log items that cannot be serialized instead of aborting the export.
[ThibautBorn]Add a item_hook to export_localroles.
[ThibautBorn]Fix handling of checkboxes for skip_existing_content and import_to_current_folder.
[pbauer]Move intermediary commit code into commit_hook method to allow overriding.
[pbauer]Add hook global_obj_hook_before_deserializing to modify the created obj before deserializing the data.
[pbauer]Add support to update and to replace existing content during import (#76)
[pbauer]Reindex permissions after importing local roles.
[pbauer]Add export/import for constrains but import content without checking constrains or permissions (#71).
[pbauer]1.2 (2021-10-11)Prevent creating content in a different Plone Site in the same database (#52).
In general, cleanup parent paths when in development on localhost.
[maurits]Read environment variableCOLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY(#51).
When set, this is used for storing an export file and getting an import file.
This is useful for sharing content between multiple Plone Sites on the same server.
[maurits]Unescape html entities and line-breaks when importing comments (#43).
[pbauer]Export and import complete sites or content trees with configurable types, depth and path (#40).
[pbauer]Added option to export blobs as blob paths (#50).
[pbauer, maurits]Fixed creating missing folder structure (#45).
[maurits]Export and import portlets (#39).
[pbauer]Export content and write to file using a generator/yield. This avoids memory ballooning to the size of the exported file (#41).
[fredvd]1.1 (2021-08-02)Add option to import file from server.
[maurits]Missing</form>closing tag inexport_content.pt[petschki]Support disabled aquisition of local roles during export/import of local roles.
[pbauer]Use unrestrictedSearchResults to actually export all content.
[pbauer]Add commit message after importing one type.
[pbauer]Fix getting container for some cases.
[pbauer]Fix use in Plone 4.3 without dexterity, zc.relation or plone.app.contenttypes.
[pbauer]Fix @id of collections and parents of subcollections. Fix #30
[pbauer]Fix use in Plone 4.3 with dexterity but without z3c.relationfield.
[maurits]Add export and import for discussions/comments.
[pbauer]Add option to fix collection queries after import.
[thomasmassmann]Reset Creation Date. Fix #29
[pbauer]Remove custom serializer for relations beacuse of ConfigurationConflictError with restapi.
Relations are dropped anyway in update_data_for_migration when using the default setting.
[pbauer]Migrate batch size for topics.
[pbauer]Fix issue of reusing the previous container when no container for a item could be found.
[pbauer]Add hook self.finish() to do things after importing one file.
[pbauer]Fix installation with older versions of setuptools (#35)
[pbauer]Fix installation using pip (#36)
[ericof]Do not constrain exportable FTIs to allow export of types as CalendarXFolder or ATTopic Criteria.
[pbauer]Add hook self.start() to do things after importing one file.
[pbauer]1.0 (2021-04-27)Support setting values withfactory_kwargswhen creating instances during import.
This can be used to set values that need to be there during subscribers to IObjectAddedEvent.
[pbauer]1.0b1 (2021-03-26)Add option to save export on server.
[pbauer]Fix issues in import_relations and import_ordering.
[pbauer]Use links to other exports in export_content for easier override.
[pbauer]Add support for exporting LinguaPlone translations.
[pbauer]1.0a2 (2021-03-11)Simplify package structure and remove all unneeded files
[pbauer]Add export/import for position in parent
[pbauer]1.0a1 (2021-03-10)Initial release.
[pbauer]