Package: omero-cli-transfer
omero-cli-transferAn OMERO CLI plugin for creating and using transfer packets between OMERO servers.Transfer packets contain objects and annotations. This project creates a zip file from an object
(Project, Dataset, Image, Screen, Plate) containing all original files necessary to create the images
in that object, plus an XML file detailing the links between entities, annotations and ROIs thereof.The CLI plugin add the subcommandtransfer, which in its turn has two further subcommandsomero transfer packandomero transfer unpack. Both subcommands (pack and unpack) will use an existing OMERO session created via CLI or prompt the user for parameters to create one.Installationtl;dr: if you havepython>=3.8, a simplepip install omero-cli-transfermightdo. We recommend conda, though.omero-cli-transferrequires at least Python 3.8. This is due toome-typesrequiring that as well;
this package relies heavily on it, and it is not feasible without it.Of course, this CAN be an issue, especially givenomero-pyofficiallyonly supports Python 3.6. However,
it is possible to runomero-pyin Python 3.8 or newer as well. Our recommended way to do so it usingconda.
With conda installed, you can doconda create -n myenv -c conda-forge python=3.8 zeroc-ice=3.6.5
conda activate myenv
pip install omero-cli-transferIt is possible to do the same thing withoutcondaas long as your python/pip version is at least 3.8,
but that will require locally building a wheel forzeroc-ice(which pip does automatically) - it is a
process that can be anything from "completely seamless and without issues" to "I need to install every
dependency ever imagined". Try at your own risk.If you want optional RO-Crate exports, you can dopip install omero-cli-transfer[rocrate]instead.Usageomero transfer packCreates a transfer packet for moving objects between OMERO server instances.The syntax for specifying objects is:<object>:<id>where<object>can beImage,Project,Dataset,PlateorScreen.Projectis assumed if<object>:is omitted. A file path needs to be provided; a tar file with the contents of the packet will be created at the specified path.Currently, only MapAnnotations, Tags, FileAnnotations and CommentAnnotations are packaged into the transfer pack. All kinds of ROI (except Masks) should work.Note that, if you are packing aPlateorScreen, default OMERO settings prevent you from downloading Plates and you will generate an empty pack file if you do so. If you want to generate a pack file from these entities, you will need to setomero.policy.binary_accessappropriately.--zippacks the object into a compressed zip file rather than a tarball.--barchivecreates a package compliant with Bioimage Archive submission standards - see below for more detail.--rocrategenerates a RO-Crate compliant package with flat structure (all image
files in a single folder). A JSON metadata file is added with basic information
about the files (name, mimetype).--simplecreates a "human-readable" package; one folder per project or dataset is created and image files are placed according to where they came from in the OMERO server. Note that a package generated with this option is not guaranteed to work withunpack, though it often will.--metadataallows you to specify which transfer metadata will be saved intransfer.xmlas possible MapAnnotation values to the images. Defaults to image ID, timestamp, software version, source hostname, md5, source username, source group.Examples:omero transfer pack Image:123 transfer_pack.tar
omero transfer pack --zip Image:123 transfer_pack.zip
omero transfer pack Dataset:1111 /home/user/new_folder/new_pack.tar
omero transfer pack 999 tarfile.tar  # equivalent to Project:999omero transfer unpackUnpacks an existing transfer packet, imports images/plates as orphans and uses the XML contained in the transfer packet to re-create links, annotations and ROIs.Note that unpack needs to be able to identify the images it imports inequivocally; this can be a problem in case you have other images with the sameclientPath(i.e. that were imported from the exact same location, including filename) and no annotations created by omero-cli-transfer. The most common case to generate this issue is an unpack that fails after the import step - the lingering images are not annotated correctly and a retry of the same unpack will use the sameclientPathand cause issues. The best solution is cleaning up after failed unpacks.--ln_sforces imports to use the transfer=ln_s option, in-place importing files. Same restrictions of regular in-place imports apply.--outputallows for specifying an optional output folder where the packet will be unzipped.--folderallows the user to point to a previously-unpacked folder rather than a single file.--mergewill use existing Projects, Datasets and Screens if the current user
already owns entities with the same name as ones defined intransfer.xml,
effectively merging the "new" unpacked entities with existing ones.--metadataallows you to specify which transfer metadata will be used fromtransfer.xmlas MapAnnotation values to the images. Fields that do not exist ontransfer.xmlwill be ignored. Defaults to image ID, timestamp, software version, source hostname, md5, source username, source group.Examples:omero transfer unpack transfer_pack.zip
omero transfer unpack --output /home/user/optional_folder --ln_s
omero transfer unpack --folder /home/user/unpacked_folder/omero transfer prepareCreates an XML from a folder with images.Creates an XML file appropriate for usage withomero transfer unpackfrom
a folder that contains image files, rather than a source OMERO server. This
is intended as a first step on a bulk-import workflow, followed by usingomero transfer unpackto complete an import.Examples:omero transfer prepare /home/user/folder_with_filesNOTE: please refer to optional requirement instructions below and consider that this feature is experimental!Bioimage Archive submission contentsFolder structure in the generated zip/tar follows project/dataset structure rather than original ManagedRepository folder structure, and instead of atransfer.xmlfile, asubmission.tsvfile is generated.submission.tsvfile has:one line per file being submitted, betweenImagefiles andFileAnnotationfiles;a column indicating whether that file was originally anImageorFileAnnotation;a "comment" column if any Image has aCommentAnnotation;a column per key in aMapAnnotationinside the pack, with an empty value for all images but the ones with aMapAnnotationwith that key; for those images, it has the value for that annotation;a finaloriginal_omero_idscolumn listing all OMERO IDs associated to that file in the origin server: for images, that is allImageIDs that use that file, and for file annotations that is allImageIDs that had thatFileAnnotationattached to them.RO-Crate export formatThis requires an optional dependency onro-crate-pythat can be installed withpip install omero-cli-transfer[rocrate].Largely due to library limitations, current exports create a flat structure inside a zip file. For each image,nameandmimetypeare recorded. Aro-crate-metadata.jsonis added to the zip file.omero transfer prepareoptional requirementspreparerequiresbftoolsto work (in particular, we need to be able to runshowinf). The easiest way to install this is by using conda; A simpleconda install -c bioconda bftoolson your conda environment should suffice.Note that this is a Java application. The conda package will install a JDK for you if necessary, but if you're installing it for yourself you'll need to make sure Java is available.This tool runsshowinfin a subprocess and needs to be able to parse the output. This can be problematic if yourstdoutis not set to UTF-8; it can mess up special characters is e.g. measurement units and lead to XML validation errors. In addition to that, Java itself might output data in non-UTF-8 encodings, in which case it might be necessary to setJAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8.