Package: plantit
plantitCLICLI, Python library and science gateway for high-throughput phenotyping on clusters and the CyVerse cloudgenerate job scripts and launch containers to a SLURM cluster with a single commandadd theplantit-actionto a GitHub Actions workflow for continuous analysisautomatically transfer data and results to and from the CyVerse Data Storediscover or publish workflows and monitor your submissions in theweb UIIn development, not stable.ContentsRequirementsInstallationQuickstartUsageCommandsTokenUserListStatPullPushExistsCreateShareUnshareTagTagsScriptsSubmitDevelopmentTestsMarkersRequirementsPython 3.8+InstallationTo install with pip:pipinstallplantitQuickstartTheplantitCLI parses YAML configuration files, generates job scripts, and submitsSingularitycontainer workflows to a SLURM scheduler in the local environment or to a remote cluster via SSH.At minimum, a configuration file must contain:imageentrypointworkdiremailqueueFor instance, inhello.yaml:image:alpineentrypoint:echo "hello world"workdir:/path/to/your/scratch/directoryemail:you@institution.eduqueue:batchTo generate a job script and submit this job to a scheduler running on the host machine:plantithello.yamlIf the job was submitted successfully, the job ID will be printed.UsageTo show CLI docs runplantit -h. Besides the mainplantit <config file>.ymlcommand, a number of subcommands can be invoked withplantit <command>.CommandsThe following commands are available:token: Retrieve a CyVerse authentication token.compat: Check if the current system is compatible.user: Retrieve the user's profile information.list: List files in a collection.stat: Get information about a file or collection.pull: Download one or more files from a collection.push: Upload one or more files to a collection.exists: Check if a path exists in the data store.create: Create a collection in the data store.share: Share a file or collection with another user.unshare: Revoke another user's access to your file or collection.tag: Set metadata for a given file or collection.tags: Get metadata for a given file or collection.scripts: Generate job scripts for a container workflow.submit: Submit jobs for a container workflow to a cluster.TokenTo request a CyVerse CAS authentication token, use thetokencommand:plantittoken--username<yourCyVerseusername>--password<yourCyVersepassword>The token can be passed via--token (-t)argument to authenticate subsequent commands.CompatTheplantit compatcommand determines whether jobs can be submitted to the host system, affirming tostdoutif the following conditions are met:singularityis installed and available on the paththe CyVerse data store is reachable via iRODS or science APIsfor the former, the user must have runiinitto configure iCommandsSLURM is up and standard commandssbatch,squeue,sacct, etc are availableOtherwise the command terminates with an error signal and information on the missing or misconfigured dependencies is printed tostderr.UserTheusercommand can be used to retrieve public profile information for CyVerse users. For instance, to get my profile info:plantituser-t<token>wbonelliListTo list the contents of a collection in the data store, use thelistcommand. For instance:plantitlist-t<token>/iplant/home/shared/iplantcollaborative/testing_tools/StatTo view metadata for a particular collection or object in the data store, use thestatcommand. For instance:plantitstat-t<token>/iplant/home/shared/iplantcollaborative/testing_tools/PullTo download a single file from the data store to the current working directory, simply provide its full path:plantitpull-t<token>/iplant/home/shared/iplantcollaborative/testing_tools/cowsay/cowsay.txtTo download all files from the/iplant/home/shared/iplantcollaborative/testing_tools/cowsay/collection to the current working directory, just provide the collection path instead:plantitpull-t<token>/iplant/home/shared/iplantcollaborative/testing_tools/cowsay/Optional arguments are:--local_path (-p): Local path to download files to--include_pattern (-ip): File patterns to include (0+)--force (-f): Whether to overwrite already-existing filesPushTo upload all files in the current working directory to the/iplant/home/<my>/<directory/in the CyVerse Data Store, use:plantitpush-t<token>/iplant/home/<username>/<collection>/Optional arguments include:--local_path (-p): Local path to download files to--include_pattern (-ip): File patterns to include (0+)--include_name (-in): File names to include (0+)--exclude_pattern (-ep): File patterns to exclude (0+)--exclude_name (-en): File names to exclude (0+)To upload a single file to the data store, provide the--local_path (-p)argument. For instance:plantitpush-t<token>/iplant/home/<username>/<collection/-p/my/local/file.txtIf onlyinclude_...s are provided, only the file patterns and names specified will be included. If onlyexclude_...s section are present, all files except the patterns and names specified will be included. If you provide bothinclude_...andexclude_...sections, theinclude_...rules will first be applied to generate a subset of files, which will then be filtered by theexclude_...rules.ExistsTo determine whether a particular path exists in the data store, use theexistscommand. For instance, to check if a collection exists:plantitexists-t<token>/iplant/home/<username>/<collectionThe--typeoption can be provided with valuedirorfileto verify that the given path is of the specified type.CreateTo create a new collection, use thecreatecommand:plantitcreate-t<token>/iplant/home/<username>/<newcollectionname>ShareTo share a file or collection with another user, use thesharecommand:plantitshare-t<token>/iplant/home/<username>/<collection>--username<usertosharewith>--permission<'read'or'write'>Note that you must provide both the--usernameand--permissionflags.UnshareTo revoke another user's access to your file or collection, use theunsharecommand:plantitunshare-t<token>/iplant/home/<username>/<collection>--username<username>This applies to bothreadandwritepermissions for the specified user.TagTo set metadata for a given file object or collection in your data store, use thetagcommand:plantittag<dataobjectID>-t<token>-ak1=v1-ak2=v2This applies the two given attributes to the data object (attributes must be formattedkey=value).Warning:this command is an overwrite, not an append. We do not support appending tags as there is no Terrain endpoint to add/remove individual metadata attributes. Note also that by default, key/value pairs are passed on theavusattribute of the request body rather thanirods-avus, e.g.:POSThttps://de.cyverse.org/terrain/secured/filesystem/<ID>/metadata{"irods-avus":[],"avus":[{"attr":"some key""value":"some value","unit":""}]}To configureirods-avusattributes as well as or in place of standard attributes, use the--irods_attribute (-ia)option. Both standard and iRODS attributes can be used in the same invocation.TagsTo retrieve the metadata describing a particular file object or collection, use thetagscommand:plantittags<dataobjectID>-t<token>This will retrieve standard attributes by default. To retrieve iRODS attributes instead, use the--irods (-i)option.ScriptsTo generate SLURM job scripts for a container workflow, use thescriptscommand. For instance:plantitscripts...SubmitTo submit a container workflow as a job script on a cluster, use thesubmitcommand. For instance, to copy the contents of the current working directory to a cluster and submit the job defined injob.sh:plantitsubmit-p.-jjob.sh\--cluster_host<hostnameorIP>\--cluster_user<useraccountname>\--cluster_key<privateSSHkey>\--cluster_target<locationtocopyjobscriptsandinputfiles>DevelopmentFirst, clone the repo withgit clone https://github.com/Computational-Plant-Science/plantit-cli.git.Create a Python3 virtual environment, e.g.python3 -m venv venv, then installplantitand core dependencies withpip install .. Install testing and linting dependencies as well with `pip install ".[test]".TestsThe tests can be run from the project root withpytest(orpython3 -m pytest). Use-vfor verbose mode and-n autoto run them in parallel on as many cores as your machine will spare.Note:some tests required theCYVERSE_USERNAMEandCYVERSE_PASSWORDenvironment variables. You can set these manually or put them in a.envfile in the project root —pytest-dotenvwill detect them in the latter case. Test cases will use this CyVerse account and its associated data store as a test environment. Each test case isolates its workspace to a folder named by GUID.MarkersThe full test suite should take 5-10 minutes to run, depending on the delay configured to allow the CyVerse Data Store to become consistent. This is 10 seconds per write operation, by default.Note:The CyVerse data store is not immediately consistent and write operations may take some time to be reflected in subsequent reads. Tests must wait some unknown amount of time to allow the Data Store to update its internal state. If tests begin to fail intermittently, theDEFAULT_SLEEPvariable inplantit/terrain/tests/conftest.pymay need to be increased.A fast subset of the tests can be run withpytest -S(short for--smoke). The smoke tests should complete in under a minute.