Package: irods-capability-automated-ingest
iRODS Automated Ingest FrameworkThe automated ingest framework gives iRODS an enterprise solution that solves two major use cases: getting existing data under management and ingesting incoming data hitting a landing zone.Based on the Python iRODS Client and Celery, this framework can scale up to match the demands of data coming off instruments, satellites, or parallel filesystems.The example diagrams below show a filesystem scanner and a landing zone.Supported/Tested Python versions3.73.83.93.10Usage optionsRedis optionsoptioneffectdefaultredis_hostDomain or IP address of Redis hostlocalhostredis_portPort number for Redis6379redis_dbRedis DB number to use for ingest0S3 optionsTo scan S3 bucket, minimally requires--s3_keypairand source path of the form/bucket_name/path/to/root/folder.optioneffectdefaults3_keypairpath to S3 keypair fileNones3_endpoint_domainS3 endpoint domains3.amazonaws.coms3_region_nameS3 region nameus-east-1s3_proxy_urlURL to proxy for S3 accessNones3_insecure_connectionDo not use SSL when connecting to S3 endpointFalseLogging/Profiling optionsoptioneffectdefaultlog_filenamePath to output file for logsNonelog_levelMinimum level of message to logNonelog_intervalTime interval with which to rollover ingest log fileNonelog_whenType/units of log_interval (see TimedRotatingFileHandler)None--profileallows you to use vis to visualize a profile of Celery workers over time of ingest job.optioneffectdefaultprofile_filenameSpecify name of profile filename (JSON output)Noneprofile_levelMinimum level of message to log for profilingNoneprofile_intervalTime interval with which to rollover ingest profile fileNoneprofile_whenType/units of profile_interval (see TimedRotatingFileHandler)NoneIngest start optionsThese options are used at the "start" of an ingest job.optioneffectdefaultjob_nameReference name for ingest joba generated uuidintervalRestart interval (in seconds). If absent, will only sync once.Nonefile_queueName for the file queue.filepath_queueName for the path queue.pathrestart_queueName for the restart queue.restartevent_handlerPath to event handler fileNone (see "event_handler methods" below)synchronousBlock until sync job is completedFalseprogressShow progress bar and task counts (must have --synchronous flag)Falseignore_cacheIgnore last sync time in cache - like starting a new syncFalseOptimization optionsoptioneffectdefaultexclude_file_typetypes of files to exclude: regular, directory, character, block, socket, pipe, linkNoneexclude_file_namea list of space-separated python regular expressions defining the file names to exclude such as "(\S+)exclude" "(\S+).hidden"Noneexclude_directory_namea list of space-separated python regular expressions defining the directory names to exclude such as "(\S+)exclude" "(\S+).hidden"Nonefiles_per_taskNumber of paths to process in a given task on the queue50initial_ingestUse this flag on initial ingest to avoid check for data object paths already in iRODSFalseirods_idle_disconnect_secondsSeconds to hold open iRODS connection while idle60available--event_handlermethodsmethodeffectdefaultpre_data_obj_createuser-defined pythonnonepost_data_obj_createuser-defined pythonnonepre_data_obj_modifyuser-defined pythonnonepost_data_obj_modifyuser-defined pythonnonepre_coll_createuser-defined pythonnonepost_coll_createuser-defined pythonnonepre_coll_modifyuser-defined pythonnonepost_coll_modifyuser-defined pythonnonecharacter_mapuser-defined pythonnoneas_usertakes action as this iRODS userauthenticated usertarget_pathset mount path on the irods server which can be different from client mount pathclient mount pathto_resourcedefines  target resource request of operationas provided by client environmentoperationdefines the mode of operationOperation.REGISTER_SYNCmax_retriesdefines max number of retries on failure0timeoutdefines seconds until job times out3600delaydefines seconds between retries0Event handlers can useloggerto write logs. Seestructlogfor available logging methods and signatures.Character Mapping optionIf an application should require that iRODS logical paths produced by the ingest process exclude subsets of the
range of possible Unicode characters, we can add a character_map method that returns a dict object. For example:class event_handler(Core):
        @staticmethod
        def character_map():
            return {
                re.compile('[^a-zA-Z0-9]'):'_'
            }
        # ...The returned dictionary, in this case, indicates that the ingest process should replace all non-alphanumeric (as
well as non-ASCII) characters with an underscore wherever they may occur in an otherwise normally generated logical path.
The substition process also applies to the intermediate (ie collection name) elements in a logical path, and a suffix is
appended to affected path elements to avoid potential collisions with other remapped object names.Each key of the returned dictionary indicates a character or set of characters needing substitution.
Possible key types include:character# substitute backslashes with underscores
    '\\': '_'tuple of characters# any character of the tuple is replaced by a Unicode small script x
    ('\\','#','-'): '\u2093'regular expression# any character outside of range 0-256 becomes an underscore
    re.compile('[\u0100-\U0010ffff]'): '_'callable accepting a character argument and returning a boolean# ASCII codes above 'z' become ':'
    (lambda c: ord(c) in range(ord('z')+1,128)): ':'In the event that the order-of-substitution is significant, the method may instead return a list of key-value tuples.UnicodeEncodeErrorAny file whose path in the filesystem whose ingest results in a UnicodeEncodeError exception being raised (e.g. by the
inclusion of an unencodable UTF8 sequence) will be automatically renamed using a base-64 sequence to represent the original,
unmodified vault path.Additionally, data objects that have had their names remapped, whether pro forma or via a UnicodeEncodeError, will be
annotated with an AVU of the formAttribute:	"irods::automated_ingest::" + ANNOTATION_REASON
Value:	A PREFIX plus the base64-converted "bad filepath"
Units:	"python3.base64.b64encode(full_path_of_source_file)"Where :ANNOTATION_REASON is either "UnicodeEncodeError" or "character_map" depending on why the remapping occurred.PREFIX is either "irods_UnicodeEncodeError_" or blank(""), again depending on the re-mapping cause.Note that the UnicodeEncodeError type of remapping is unconditional, whereas the character remapping is contingent on
an event handler's character_map method being defined.  Also, if a UnicodeEncodeError-style ingest is performed on a
given object, this precludes character mapping being done for the object.operation modeoperationnew filesupdated filesOperation.REGISTER_SYNC(default)registers in catalogupdates size in catalogOperation.REGISTER_AS_REPLICA_SYNCregisters first or additional replicaupdates size in catalogOperation.PUTcopies file to target vault, and registers in catalogno actionOperation.PUT_SYNCcopies file to target vault, and registers in catalogcopies entire file again, and updates catalogOperation.PUT_APPENDcopies file to target vault, and registers in catalogcopies only appended part of file, and updates catalogOperation.NO_OPno actionno action--event_handlerusage examples can be foundin the examples directory.DeploymentBasic: manual redis, Celery, pipRunning the sync job and Celery workers requires a valid iRODS environment file for an authenticated iRODS user on each node.Starting Redis ServerInstall redis-server:sudo yum install redis-serversudo apt-get install redis-serverOr, build it yourself:https://redis.io/topics/quickstartStart redis:redis-serverOr, dameonized:sudo service redis-server startsudo systemctl start redisTheRedis documentationalso recommends an additional step:Make sure to set the Linux kernel overcommit memory setting to 1. Add vm.overcommit_memory = 1 to /etc/sysctl.conf and then reboot or run the command sysctl vm.overcommit_memory=1 for this to take effect immediately.This allows the Linux kernel to overcommit virtual memory even if this exceeds the physical memory on the host machine. Seekernel.org documentationfor more information.Note:If running in a distributed environment, make sure Redis server accepts connections by editing thebindline in /etc/redis/redis.conf or /etc/redis.conf.Setting up virtual environmentYou may need to upgrade pip:pip install --upgrade pipInstall virtualenv:pip install virtualenvCreate a virtualenv with python3:virtualenv -p python3 rodssyncActivate virtual environment:source rodssync/bin/activateInstall this packagepip install irods_capability_automated_ingestSet up environment for Celery:export CELERY_BROKER_URL=redis://<redis host>:<redis port>/<redis db> # e.g. redis://127.0.0.1:6379/0
export PYTHONPATH=`pwd`Start celery worker(s):celery -A irods_capability_automated_ingest.sync_task worker -l error -Q restart,path,file -c <num workers>Note:Make sure queue names match those of the ingest job (default queue names shown here).Run testsNote:The test suite requires Python version >=3.7.Note:The tests should be run without running Celery workers or with an unused redis database.python -m irods_capability_automated_ingest.test.test_irods_syncSeedocker/test/README.mdfor how to run in a dockerized environment.Start sync jobpython -m irods_capability_automated_ingest.irods_sync start <source dir> <destination collection>List jobspython -m irods_capability_automated_ingest.irods_sync listStop jobspython -m irods_capability_automated_ingest.irods_sync stop <job name>Watch jobs (same as using--progress)python -m irods_capability_automated_ingest.irods_sync watch <job name>Intermediate: dockerize, manually config (needs to be updated for Celery)Seedocker/README.mdAdvanced: kubernetes (needs to be updated for Celery)This does not assume that your iRODS installation is in kubernetes.kubeadmsetupGlusterfsandHeketicreate storage classcreate a persistent volume claimdatainstall minikube and helmset memory to at least 8g and cpu to at least 4minikube start --memory 8192 --cpus 4enable ingress on minikubeminikube addons enable ingressmount host dirsThis is where you data and event handler. In this setup, we assume that your event handler is under/tmp/host/event_handlerand you data is under/tmp/host/data. We will mount/tmp/host/datainto/host/datain minikube which will mount/host/datainto/datain containers,/tmp/host/data-> minikube/host/data-> container/data.and similarly,/tmp/host/event_handler-> minikube/host/event_handler-> container/event_handler. Your setup may differ.mkdir /tmp/host/event_handler
mkdir /tmp/host/data/tmp/host/event_handler/event_handler.pyfromirods_capability_automated_ingest.coreimportCorefromirods_capability_automated_ingest.utilsimportOperationclassevent_handler(Core):@staticmethoddeftarget_path(session,meta,**options):returnpathminikube mount /tmp/host:/host --gid 998 --uid 998 --9p-version=9p2000.Lenable incubatorhelm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/build local docker images (optional)If you want to use local docker images, you can build the docker images into minikube as follows:fisheval (minikube docker-env)basheval $(minikube docker-env)cd <repo>/docker/irods-postgresql
docker build . -t irods-provider-postgresql:4.2.2cd <repo>/docker/irods-cockroachdb
docker build . -t irods-provider-cockroachdb:4.3.0cd <repo>/docker
docker build . -t irods_capability_automated_ingest:0.1.0cd <repo>/docker/rq
docker build . -t irods_rq:0.1.0cd <repo>/docker/rq-scheduler
docker build . -t irods_rq-scheduler:0.1.0install irodspostgresqlcd <repo>/kubernetes/irods-provider-postgresql
helm dependency updatecd <repo>/kubernetes
helm install ./irods-provider-postgresql --name irodscockroachdbcd <repo>/kubernetes/irods-provider-cockroachdb
helm dependency updatecd <repo>/kubernetes
helm install ./irods-provider-cockroachdb --name irodswhen reinstalling, runkubectl delete --all pv
kubectl delete --all pvcupdate irods configurationsSet configurations in<repo>/kubernetes/chart/values.yamlor--setcommand line argument.install chartcd <repo>/kubernetes/chartWe call our releaseicai.cd <repo>/kubernetes
helm install ./chart --name icaiscale rq workerskubectl scale deployment.apps/icai-rq-deployment --replicas=<n>access by REST API (recommended)submit jobsubmit.yamlroot:/datatarget:/tempZone/home/rods/datainterval:<interval>append_json:<yaml>timeout:<timeout>all:<all>event_handler:<event_handler>event_handler_data:|from irods_capability_automated_ingest.core import Corefrom irods_capability_automated_ingest.utils import Operationclass event_handler(Core):@staticmethoddef target_path(session, meta, **options):return pathfishcurl -XPUT "http://"(minikube ip)"/job/<job name> -H "Content-Type: application/x-yaml" --data-binary=`submit.yaml`bashcurl -XPUT "http://$(minikube ip)/job/<job name>" -H "Content-Type: application/x-yaml" --data-binary "@submit.yaml"fishcurl -XPUT "http://"(minikube ip)"/job" -H "Content-Type: application/x-yaml" --data-binary "@submit.yaml"bashcurl -XPUT "http://$(minikube ip)/job" -H "Content-Type: application/x-yaml" --data-binary "@submit.yaml"list jobfishcurl -XGET "http://"(minikube ip)"/job"bashcurl -XGET "http://$(minikube ip)/job"delete jobfishcurl -XDELETE "http://"(minikube ip)"/job/<job name>"bashcurl -XDELETE "http://$(minikube ip)/job/<job name>"access by command line (not recommended)submit jobkubectl run --rm -i icai --image=irods_capability_automated_ingest:0.1.0 --restart=Never -- start /data /tempZone/home/rods/data -i <interval> --event_handler=event_handler --job_name=<job name> --redis_host icai-redis-masterlist jobkubectl run --rm -i icai --image=irods_capability_automated_ingest:0.1.0 --restart=Never -- list --redis_host icai-redis-masterdelete jobkubectl run --rm -i icai --image=irods_capability_automated_ingest:0.1.0 --restart=Never -- stop <job name> --redis_host icai-redis-masterinstall logging toolInstall chart with setlog_leveltoINFO.helm del --purge icaicd <repo>/kubernetes
helm install ./chart --set log_level=INFO --name icaiset parameters for elasticsearchminikube ssh 'echo "sysctl -w vm.max_map_count=262144" | sudo tee -a /var/lib/boot2docker/bootlocal.sh'
minikube stop
minikube startcd <repo>/kubernetes
helm install ./elk --name icai-elkGrafanalook for service portkubectl get svc icai-elk-grafanaforward portkubectl port-forward svc/icai-elk-grafana 8000:80If--set grafana.adminPassword=""system generates a random password, lookup admin passwordkubectl get secret --namespace default icai-elk-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echoopen browser urllocalhost:8000login with usernameadminand passwordadminclick onicai dashboardKibanaUncomment kibana sections in the yaml files under the<repo>/kubernetes/elkdirectorylook for service portkubectl get svc icai-elk-kibanaforward portkubectl port-forward svc/icai-elk-kibana 8000:443open browser urllocalhost:8000