Package: scingestor
Scicat Dataset IngestorThescingestorpython package provides a support for scripts which ingest Datasets and OrigDatablocks into the SciCat metadata server.scicat_dataset_ingestorSciCat Dataset ingestor server ingests scan metadata just after a scan is finished. It can be executed byscicat_dataset_ingestor -c ~/.scingestor.yamlConfiguration variablesThe configuration written in YAML can contain the following variablesscicat_url(str), default:"http://localhost:3000/api/v3"ingestor_credential_file(str), default:Nonebeamtime_dirs(list<str>), default:[]beamtime_base_dir(str), default:""ingestor_var_dir(str), default:""ingestor_username(str), default:"ingestor"dataset_pid_prefix(str), default:""dataset_update_strategy("no","patch","create","mixed") , default:"patch"relative_path_in_datablock(bool), default:Falsescandir_blacklist(list<str>), default:["/gpfs/current/scratch_bl", "/gpfs/current/processed", "/gpfs/current/shared"]beamtimeid_blacklist_file(str), default:Nonebeamtime_type_blacklist(list<str>), default:["P"]chmod_json_files(str), default:Nonemax_scandir_depth(int), default:-1oned_in_metadata(bool), default:Falseforce_measurement_keyword(bool), default:Truemax_oned_size(int), default:Nonescan_metadata_postfix(str), default:".scan.json"datablock_metadata_postfix(str), default:".origdatablock.json"attachment_metadata_postfix(str), default:".attachment.json"metadata_in_var_dir(bool), default:Falseuse_corepath_as_scandir(bool), default:Falsebeamtime_filename_postfix(str), default:"beamtime-metadata-"beamtime_filename_prefix(str), default:".json"datasets_filename_pattern(str), default:"scicat-datasets-{beamtimeid}.lst"ingested_datasets_filename_pattern(str), default:"scicat-ingested-datasets-{beamtimeid}.lst"file_dataset_metadata_generator(str), default:"nxsfileinfo metadata -k4  -o {metapath}/{scanname}{scanpostfix}  -b {beamtimefile} -p {beamtimeid}/{scanname}  -w {ownergroup} -c {accessgroups} {mastefile}"dataset_metadata_generator(str), default:"nxsfileinfo metadata -k4  -o {metapath}/{scanname}{scanpostfix}  -c {accessgroups} -w {ownergroup} -b {beamtimefile} -p {beamtimeid}/{scanname}"datablock_metadata_generator(str), default:"nxsfileinfo origdatablock  -s *.pyc,*{datablockpostfix},*{scanpostfix},*~  -p {pidprefix}{beamtimeid}/{scanname}  -w {ownergroup} -c {accessgroups} -o {metapath}/{scanname}{datablockpostfix} "datablock_metadata_stream_generator(str), default:"nxsfileinfo origdatablock  -s *.pyc,*{datablockpostfix},*{scanpostfix},*~  -w {ownergroup} -c {accessgroups} -p {pidprefix}{beamtimeid}/{scanname} "datablock_metadata_generator_scanpath_postfix(str), default:" {scanpath}/{scanname} "attachment_metadata_generator(str), default:"nxsfileinfo attachment  -w {ownergroup} -c {accessgroups} -o {metapath}/{scanname}{attachmentpostfix} {plotfile} "metadata_generated_callback(str), default:"nxsfileinfo groupmetadata  {lastmeasurement} -m {metapath}/{scanname}{scanpostfix} -d {metapath}/{scanname}{datablockpostfix} -a {metapath}/{scanname}{attachmentpostfix} -p {beamtimeid}/{lastmeasurement} -f -k4 "metadata_group_map_file(str), default:""raw_metadata_callback(bool), default:Falseskip_multi_datablock_ingestion(bool), default:Falseskip_multi_attachment_ingestion(bool), default:Falseskip_scan_dataset_ingestion(bool), default:Falsecall_metadata_generated_callback(bool), default:Falsemetadata_group_map_file_generator_switch(str), default:" --group-map-file {groupmapfile} "raw_metadata_callback_switch(str), default:" --raw "execute_commands(bool), default:Falseplot_file_extension_list(list<str>), default:["png", "nxs", "h5", "ndf", "nx", "fio"]master_file_extension_list(list<str>), default:["nxs", "h5", "ndf", "nx", "fio"]chmod_generator_switch(str), default:" -x {chmod} "relative_path_generator_switch(str), default:" -r {relpath} "oned_dataset_generator_switch(str), default:" --oned "max_oned_dataset_generator_switch(str), default:" --max-oned-size {maxonedsize} "override_attachment_signals_generator_switch(bool), default:" --override "hidden_attributes_generator_switch(str), default:" -n {hiddenattributes} "hidden_attributes(str), default:"nexdatas_source,nexdatas_strategy,units"attachment_signals_generator_switch(str), default:" -s {signals} "attachment_axes_generator_switch(str), default:" -e {axes} "attachment_frame_generator_switch(str), default:" -m {frame} "attachment_signal_names(str), default:""attachment_axes_names(str), default:""attachment_image_frame_number(str), default:""ingest_dataset_attachment(bool), default:Falseoverride_attachment_signals(bool), default:Falseretry_failed_dataset_ingestion(bool), default:Falseretry_failed_attachement_ingestion(bool), default:Falselog_generator_commands(bool), default:Falseadd_empty_units_generator_switch(str), default:" --add-empty-units "add_empty_units(bool), default:Truemetadata_copy_map_file(str), default:Nonemetadata_copy_map_file_generator_switch(str), default:" --copy-map-file {copymapfile} "inotify_timeout(float), default:0.1get_event_timeout(float), default:0.01ingestion_delay_time(float), default:5.0max_request_tries_number(int), default:100recheck_dataset_list_interval(int), default:1000recheck_beamtime_file_interval(int), default:1000request_headers(dict<str,str>), default:{"Content-Type": "application/json", "Accept": "application/json"}scicat_datasets_path(str), default:"Datasets"scicat_proposals_path(str), default:"Proposals"scicat_datablocks_path(str), default:"OrigDatablocks"scicat_attachments_path(str), default:"Datasets/{pid}/Attachments"scicat_users_login_path(str), default:"Users/login"owner_access_groups_from_proposal(bool), default:Falsemetadata_fields_without_checks(list<str>), default:["techniques", "classification", "createdBy", "updatedBy", "datasetlifecycle", "numberOfFiles", "size", "createdAt", "updatedAt", "history", "creationTime", "version", "scientificMetadata", "endTime"]e.g.beamtime_dirs:
  - "{homepath}/gpfs/current"
  - "{homepath}/gpfs/commissioning"
scicat_url: http://localhost:3000/api/v3
ingestor_credential_file: "{homepath}/gpfs/pwd"Pattern keywords for configuration variablesThedatasets_filename_pattern,ingested_datasets_filename_patternandingestor_var_dircan contain the{beamtimeid}and{hostname}keywords,  e.g."scicat-ingested-datasets-{beamtimeid}.lst"or"scicat-ingested-datasets-{hostname}-{beamtimeid}.lst"which is instantiated during the ingestor execution.Thebeamtime_dirs,beamtime_base_dir,ingestor_var_dir,ingestor_credential_file,scandir_blacklistcan contain the{homepath}keyword.Similarly,file_dataset_metadata_generator,dataset_metadata_generator,datablock_metadata_generator,datablock_metadata_stream_generator,datablock_metadata_generator_scanpath_postfix,attachment_metadata_generator,chmod_generator_switch,relative_path_generator_switchcan contain the following keywords:{beamtimeid},{scanname},{chmod},{scanpath},{metapath},{relpath},{beamtimeid},{beamline},{pidprefix},{beamtimefile},{scanpostfix},{datablockpostfix},{ownergroup},{accessgroups},{hostname},{homepath},{hiddenattributes},{ext}, "{masterfile}", "{plotfile}", "{masterscanname}", "{entryname}"The "{masterfile}" is either equal to   "{scanpath}/{scanname}.{ext}" or "{scanpath}/{scanname}/{scanname}.{ext}". Also
the "{plotfile}" is either equal to  "{scanpath}/{scanname}.{plotext}" or "{scanpath}/{scanname}/{scanname}.{plotext}".scicat_dataset_ingestRe-ingestion script for SciCat Datasets and OrigDatablocks is usually launched at the end of the beamtime.scicat_dataset_ingest -c ~/.scingestor.yamlIts configuration written YAML like forscicat_dataset_ingestorscicat_ingestGeneral ingestion script for SciCat Models could be used for manual scicat model ingestion, e.g. Sample, Instrument or DerivedDataset.scicat_ingest  -m Samples  -c ~/.scingestor.yaml  ./metadata.jsonIts configuration written YAML like forscicat_dataset_ingestorInstallationRequired packagespython3 >= 3.7nxstools >= 3.38.0inotifyx (python3 version)requestssetuptoolspyyamlpytest (to run tests)sphinx (to build the documentation)Install from sourcesThe code fromhttps://github.com/jkotan/scingestorcan be built withpython3 setup.py installTo build the documentation usepython3 setup.py build_sphinxThe resulting documentation can be found belowbuild/sphinx/htmlin the root
directory of the source distribution.Finally, the package can be tested usingpython3 -m pytest testInstall in conda or pip environmentThe code can be installed in your conda environment byconda create -n myenv python=3.9
conda activate myenv

pip install inotifyx-py3
pip install scingestoror in your pip environment bypython3 -m venv myvenv
. myvenv/bin/activate

pip install inotifyx-py3
pip install scingestorDebian and Ubuntu packagesDebianbookworm,bullseye,busteror Ubuntulunar,jammy,focalpackages can be found in the HDRI repository.To install the debian packages, add the PGP repository keysudo su
curl -s http://repos.pni-hdri.de/debian_repo.pub.gpg  | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/debian-hdri-repo.gpg --import
chmod 644 /etc/apt/trusted.gpg.d/debian-hdri-repo.gpgand then download the corresponding source list, e.g.
forbookwormcd /etc/apt/sources.list.d
wget http://repos.pni-hdri.de/bookworm-pni-hdri.listorjammycd /etc/apt/sources.list.d
wget http://repos.pni-hdri.de/jammy-pni-hdri.listrespectively.Finally,apt-get update
apt-get install python3-scingestorDataset list file contentThe scicat ingestor triggers its actions on append a new line in the dataset list file.
The dataset list file is located in the scan directory and its filename is defined
bydatasets_filename_patternvariable, i.e. by default "scicat-datasets-{beamtimeid}.lst".By default the scan dataset metadata are fetched from the corresponding the master file with its filename given by <scanname>.<ext> where usually <ext> isnxsorfio. The detector files related to the particular scan are placed in the <scanname> subdirectory and they are added to the scan origindatablock.A separete line in the dataset list file may containscanname to ingest e.g.myscan_00012scanname to re-ingest with a unique identifier (timestamp), e.g.myscan_00012:1702988846.0770347scanname and additional detector subdirectories to ingest, e.g.myscan_00012 pilatus1 lambdastring with a  base master filename, a NXentry NeXus path and a scanname representing scan metadata from the multi-scan nexus file to ingest, e.g.myscan::/scan12;myscan_00012command to start a measurement with a given name which groups related scans,  e.g.__command__ start mycalib6command to stop a measurement which groups related scans,  e.g.__command__ stopMeasurment Datasets which group scan metadataThe__command__ start \<measurement\>and__command__ stopallow to pass information to scicat ingestor which scan datasets should be grouped into the measurement dataset, i.e. by default of scan datasets between start and stop commands are grouped to the one measurement.Sardana Measurement macrosThe config/scmacros.py module provides sardana macros which help to start/stop the measurementstart_measurement <measurement>starts a new measurment with the given namemake_measurement <measurement>starts a new measurment with the given name and adds to the measurement the last scanupdate_measurementupdates the current measurement dataset in the SciCat databasestop_measurementupdates the current measurement dataset in the SciCat database and stops the  measurementshow_measurementshows the current measurement nameSardana Measurement with SciCatAutoGroupingSetting theSciCatAutoGroupingsardana environment variable toFalsewe can switch on the autogrouping mode. In this mode scan metadata is grouped automatically into the measurement dataset and the measurement dataset updated after each scan. The name of measurement is taken from the base scanname after removing ScanID, e.g. for<scanname>= "mycalib2_00012" the measurement name is "mycalib2"