Package: perfact-zodbsync
PerFact ZODBSyncThis package provides tools to serialize Zope objects and store them in a file
system structure as well as to restore Zope objects from this structure.Some features require that the file system structure is governed bygitas
version control system.Maintainers:Ján Jockuschjan.jockusch@perfact.deViktor Dickviktor.dick@perfact.deRepository:git clone https://github.com/perfact/zodbsync.gitInstallationThe package should be installed usingpipin the same virt-env aszope, p.e.zope/bin/pip install git+https://github.com/perfact/zodbsyncOn PerFact systems, it should automatically be pulled by therequirements.txtof the packageperfact-dbutils-zope4and included there. If installing on an
older system, runsudo -H /opt/zope/zope2.13/bin/pip install git+https://github.com/perfact/zodbsyncOn newer PerFact Zope4 installations, installtestbranch via, e.g. for
development/testing purposessudo -H /usr/share/perfact/zope4/bin/pip install git+https://github.com/perfact/zodbsync@test --upgradeNote that executing the tests requires ODBC C headers to be installed. On
Debian-like systems, install the packageunixodbc-dev.ConfigurationUse theconfig.pyas a starting point for your configuration. At the moment,
it is not automatically installed. The canonical path for the configuration is/etc/perfact/modsync/zodb.py, so if you do not want to supply the path to
the configuration when calling the scripts, copy the configuration file there.The most important settings are:conf_pathorwsgi_conf_pathSetconf_pathorwsgi_conf_pathto thezope.confof your Zope instance.
If it is a standalone instance exclusively accessing its ZODB, it must be
powered down if used by zodbsync. So usually it is advisable that it is
configured to connect to a ZEO server.The two options are present due to a no longer relevant difference betweenZServerandWSGIinstance handling and can now be used interchangeably.base_dirInside this folder (actually, in a subfolder named__root__), the serialized
objects are placed.manager_userThe name of a user that must be defined in the top-levelUserFolder(acl_users) and which has Manager permissions. Transactions that are used to
upload objects to the ZODB are done by this user.default_ownerThis user is considered to be the default owner of objects if no other
information is stored in the object.force_default_ownerCan be combined withdefault_ownerto enforce a specific owner for objects
in the ZODB.datafs_pathThe path to the location of the Data.fs file. This is needed forzodbsync watch.run_after_playbackPath to a script that is executed after a successful (non-recursive) playback,
including indirect calls fromresetorpick. If the script exists, it is
called and fed the list of changed objects in a JSON format.UsageThe executablezodbsyncprovides several subcommandszodbsync recordTherecordsubcommand is used to record objects from the ZODB to the file
system.Each object is mapped to a folder that contains at least the file__meta__which holds the meta data of the object (properties, permissions etc.).
If the object contains other objects (likeFolders), they are represented as
subfolders. If the object has some sort of source (likePage Templates,DTML Methods etc.), it is stored in an additional file. The filename suffix is
taken from the object type and possibly content type, while the base is either__source__or__source-utf8__Only a specific list of object types is supported byzodbsync. Objects whose
type is not yet supported are created with a minimal__meta__file,
containing only thetitle,typeand anunsupportedmarker.An additional option--commitallows to create agitcommit after the recording, sending a
summary mail containing all changed files to an address specified in the
configuration. This can be used as automated reminder fallback if changes are
not commited timely.Ifzodbsync recordis called with the--lasttxnoption, it tries to do an
incremental recording, reading all transactions that occured since the last
call (the transaction ID is stored in a file__last_txn__in the repository).
The paths to be recorded are extracted from the transaction note, which works
well if editing an object in the management interface of Zope, but not
necessarily if an object is changed from within a script, if it is transferred
by the ZSyncer or if objects are cut and pasted (in the latter case, only the
target of the move operation is recognized).zodbsync watchThis subcommand starts a process that aims to bypass the shortcomings ofzodbsync record --lasttxn. The process stays alive and builds an object tree
of all objects in the ZODB. Each time it wakes up, it scans for new
transactions, opens the Data.FS directly (in read-only mode) to obtain all
affected object IDs, updates its object tree and uses it to obtain the physical
paths of all affected objects. After finishing, it sleeps for 10 seconds before
waking again. This should provide an almost live recording that does not miss
any changes.zodbsync playbackThe opposite operation torecordisplayback, which is able to create and
modify objects in the ZODB from a file system structure as it is created byrecord.By default,playbackrecurses into the subtree below a given
path, removing any superfluous objects and updating existing objects so they
match their file system representation. An exception are objects that are
marked asunsupported, which are ignored if found in the ZODB. If only a
given object itself should be updated (properties, security settings etc.),--no-recursecan be used.zodbsync execThis command requires the base directory to be a git repository and provides a
wrapper for several git commands. It executes the given command, reads the
objects changed between the old and newHEADand plays them back. Any
unstaged changes are stashed away and restored afterwards. The operation is
aborted and rolled back if it results in a broken state (an interruptedmerge,rebase,cherry-picketc.) or if there is an overlap between the
unstaged and the changed files.This allows commands like the following:zodbsync exec "git cherry-pick COMMIT"
zodbsync exec "git checkout otherbranch"
zodbsync exec "git reset --hard COMMIT"
zodbsync exec "git revert COMMIT"zodbsync resetShorthand forzodbsync exec "git reset --hard COMMIT"zodbsync checkoutWrapper forgit checkoutwith some of its functionality.zodbsync pickAs a special case ofexec, this wrapsgit cherry-pickand takes git commits
to be applied as arguments.
This is useful if some development has been done in a branch or on a remote
system that has to be deployed to the current system. It then becomes possible
to do something likegit fetch origin
zodbsync pick origin/masterto pull the latest commit, apply it to the current repository and upload the
affected paths to the Data.FS. It can also be used to pick multiple commits.
Its argument--grepallows, for example, to pull all commits where the commit
message starts with T12345:zodbsync pick --grep="^T12345" source/masterCommit ranges in the form ofCOMMIT1..COMMIT2can also be picked, but be
aware that there is no check that the commit range is actually a straight
forward succession - internally,git logis used and therefore any commits
that are reachable fromCOMMIT2but not fromCOMMIT1are picked. In
practice, choosing commits that are not directly connected will result in some
commit not being able to be picked due to conflicts.zodbsync upload(DEPRECATED)uploadexpects the base directory to be a git repository and provides a tool
to upload JS and CSS libraries into theData.fs. This is achieved by converting
these files into files and directories understood byplaybackand placing them
in the specified directory inside ofbase_dir.Example to upload bootstrap:zodbsync upload /tmp/bootstrap lib/bootstrapzodbsync with-lockIf some combination ofgitandzodbsyncoperations is not yet covered by a
wrapper subcommand, it is possible to usezodbsync with-lockto execute a
series of commands while still making sure that no other similar operation
interferes. Anyzodbsynccommand used as part of this must then use the
option--no-lock. For example:zodbsync with-lock "git rebase origin/main && zodbsync --no-lock playback /"Although this particular example can now be better achieved withzodbsync exec.zodbsync reformatWith version 4.3.2, the formatting of meta files was changed to become more
diff-friendly, placing, for example, lists of roles for a specific permission
onto one line each. When transferring commits from a system that used the old
recording to one that uses the new one,zodbsync reformatcan be used to
rewrite commits of the old to the new version.Use a separate repository clone, check out the starting point and pick the
commits that used the old formatting on top of it. Executingreformatwill
add a commit that reformats the complete repository after the starting point,
followed by rewritten commits that correspond to the original ones, but using
the new formatting. Finally, pick these commits onto the target system.
Detailed steps:find the commit ID of the first commit you want to reformat,
this ID will be referred to asSTARTfrom the source branch or system, check out the commit beforeSTARTand create a work branchgit checkout START~git checkout -b <work-branch>pick the commits to be reformatted into the work branchgit cherry-pick -Xno-renames <commit-ids ...>runreformat: this will create a commit betweenSTART~andSTARTcontaining the reformatting of the entire repo from old to new format and
applies the following commits as if they had been committed using the new
formatzodbsync reformat START~if the project also contains commits made after the format change,cherry-pickthem into the work branch nowpush the work branch to the target system andzodbsync pickthe commits
(except thezodbsync reformatcommit)Hint: This requiresgitin version 2.22 or above.CompatibilityThis package replaces similar functionality that was previously found inpython-perfactandperfact-dbutils-zope2. For backwards compatibility,
those packages were changed so the corresponding calls try to importperfact.zodbsyncand use the functionality there, falling back to the
previous implementation if that fails. Corresponding deprecation warnings are
included.Previous versions contained the scriptsperfact-zoperecordandperfact-zopeplaybackinstead ofzodbsync. For compatibility with systems
automatically callingperfact-zoperecord, it is still included but only
providing the bare functionality:perfact-zoperecord(corresponds tozodbsync record --commit /)perfact-zoperecord --lasttxn(corresponds tozodbsync record --lasttxn,
but including a call toperfact-dbrecordif adatabaseskey is defined
in the configuration)CaveatsZope allowsExternal Methods to be present in the ZODB even if the
corresponding modules are no longer present as extensions. It does not,
however, allow saving such an object. This gives errors if object trees
containing brokenExternal Methods are recorded and played back. The same
holds forZ SQL Methods which haveclass_nameandclass_fileset to a no
longer existing extension.To Do / RoadmapSpecifying a path to a python script that is executed afterplaybackinside the same transaction will allow to store database changes in a
connected relational (SQL) database inside ZODB using Z SQL Methods and
have them executed as part of the same deployment step.