Package: seqerakit
seqerakitseqerakitis a Python wrapper for theSeqera Platform CLI. It can be leveraged to automate the creation of all of the entities in Seqera Platform via a simple configuration file in YAML format.The key features are:Simple configuration: All of the command-line options available when using the Seqera Platform CLI can be defined in simple YAML format.Infrastructure as Code: Enable users to manage and provision their infrastructure specifications.Automation: End-to-end creation of entities within Seqera Platform, all the way from adding an Organization to launching pipeline(s) within that Organization.PrerequisitesYou will need to have an account on Seqera Platform (seePlans and pricing).Installationseqerakitrequires the following dependencies:Seqera Platform CLIPython (>=3.8)PyYAMLCondaYou can installseqerakitand its dependencies via Conda. Ensure that you have the correct channels configured:conda config --add channels defaultsconda config --add channels biocondaconda config --add channels conda-forgeconda config --set channel_priority strictYou can then create a conda environment withseqerakitinstalled using the following:conda env create -n seqerakit seqerakit
conda activate seqerakitPipIf you already haveSeqera Platform CLIand Python installed on your system, you can installseqerakitdirectly fromPyPI:pip install seqerakitYou can force overwrite the installation to use the latest changes with the command below:pip install --upgrade --force-reinstall seqerakitLocal development installationYou can install the development branch ofseqerakiton your local machine to test feature updates of the tool. Before proceeding, ensure that you havePythonandGitinstalled on your system.To install directly from pip:pipinstallgit+https://github.com/seqeralabs/seqera-kit.git@devAlternatively, you may clone the repository locally and install manually:gitclonehttps://github.com/seqeralabs/seqera-kit.gitcdseqera-kit
gitcheckoutdev
pipinstall.You can verify your installation with:pipshowseqerakitConfigurationCreate a Seqera Platform access token using theSeqera Platformweb interface via theYour Tokenspage in your profile.seqerakitreads this token from the environment variableTOWER_ACCESS_TOKEN. Please export it into your terminal as shown below:exportTOWER_ACCESS_TOKEN=<youraccesstoken>UsageTo confirm the installation ofseqerakit, configuration of the Seqera Platform CLI and connection is working as expected:seqerakit--infoUse the-hor--helpparameter to list the available commands and their associated options:seqerakit--helpDryrunTo print the commands that would executed withtwwhen using a YAML file, you can runseqerakitwith the--dryrunflag:seqerakitfile.yaml--dryrunRecursively deleteInstead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the--deleteflag:seqerakitfile.yaml--deleteFor example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the--deleteflag,seqerakitwill recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization.Usingtwspecific CLI optionstwspecific CLI options can be specified with the--cli=flag:seqerakitfile.yaml--cli="--arg1 --arg2"You can find the full list of options by runningtw -h.The Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run yourtwcommands with the--insecureflag.To usetwspecific CLI options such as--insecure, use the--cli=flag, followed by the options you would like to use enclosed in double quotes.For example:seqerakitfile.yaml--cli="--insecure"For Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a customcacertsstore as accepted by thetwCLI, you can specify the-Djavax.net.ssl.trustStore=/absolute/path/to/cacertsoption enclosed in double quotes toseqerakitas you would totw, preceded by--cli=.For example:seqerakithello-world-config.yml--cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts"Note: Use of--verboseoption for thetwCLI is currently not supported byseqerakit. Supplying--cli="--verbose"will raise an error.YAML Configuration OptionsThere are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed astwCLI options.1. Pipeline parameters usingparamsandparams-fileTo specify pipeline parameters, you may either useparams:to specify a list of parameters, or useparams-file:to point to a parameters file.For example, to specify pipeline parameters within your YAML:params:outdir:'s3://path/to/outdir'fasta:'s3://path/to/reference.fasta'Alternatively, to specify a file containing pipeline parameters:params-file:'/path/to/my/parameters.yaml'Optionally, you may provide both:params-file:'/path/to/my/parameters.yaml'params:outdir:'s3://path/to/outdir'fasta:'s3://path/to/reference.fasta'Note: If duplicate parameters are provided, the parameters provided as key-value pairs inside theparamsnested dictionary of the YAML file will take precedenceovervalues in the providedparams-file.2.overwriteFunctionalityFor every entity defined in your YAML file, you can specifyoverwrite: Trueto overwrite any existing entities in Seqera Platform of the same name.seqerakitwill first check to see if the name of the entity exists, if so, it will invoke atw <subcommand> deletecommand before attempting to create it based on the options defined in the YAML file.DEBUG:root: Overwrite is set to 'True' for organizationsDEBUG:root: Running command: tw -o json organizations listDEBUG:root: The attempted organizations resource already exists. Overwriting.DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAMEDEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization'3. Specifying JSON configuration files withfile-pathThe Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use thefile-pathkey to specify a path to a JSON configuration file.An example of thefile-pathoption is provided in thecompute-envs.ymltemplate:compute-envs:-name:'my_aws_compute_environment'# requiredworkspace:'my_organization/my_workspace'# requiredcredentials:'my_aws_credentials'# requiredwait:'AVAILABLE'# optionalfile-path:'./compute-envs/my_aws_compute_environment.json'# requiredoverwrite:TrueQuick startYou must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform.You will need to have an account on Seqera Platform (seePlans and pricing). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.Launch via YAMLCreate a YAML file calledhello-world-config.ymlwith the contents below, and customise the<YOUR_WORKSPACE>and<YOUR_COMPUTE_ENVIRONMENT>entries as required:launch:-name:'hello-world'# Workflow nameworkspace:'<YOUR_WORKSPACE>'# Workspace namecompute-env:'<YOUR_COMPUTE_ENVIRONMENT>'# Compute environmentrevision:'master'# Pipeline revisionpipeline:'https://github.com/nextflow-io/hello'# Pipeline URLLaunch the pipeline withseqerakit:seqerakithello-world-config.ymlLogin to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!Launch via a Python scriptYou can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python.Download thelaunch_hello_world.pyPython script and customise the<YOUR_WORKSPACE>and<YOUR_COMPUTE_ENVIRONMENT>entries as required.Launch the pipeline withseqerakit:pythonlaunch_hello_world.pyLogin to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!Defining your YAML file using CLI optionsAll available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity.Retrieve CLI OptionsTo obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows:$twpipelinesadd-hUsage: tw pipelines add [OPTIONS] PIPELINE_URLAdd a workspace pipeline.Parameters:*     PIPELINE_URL                         Nextflow pipeline URL.Options:* -n, --name=<name>                        Pipeline name.-w, --workspace=<workspace>              Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName-d, --description=<description>          Pipeline description.--labels=<labels>[,<labels>...]      List of labels seperated by coma.-c, --compute-env=<computeEnv>           Compute environment name.--work-dir=<workDir>                 Path where the pipeline scratch data is stored.-p, --profile=<profile>[,<profile>...]   Comma-separated list of one or more configuration profile names you want to use for this pipeline execution.--params-file=<paramsFile>           Pipeline parameters in either JSON or YML format.--revision=<revision>                A valid repository commit Id, tag or branch name....Define Key-Value Pairs in YAMLTranslate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance:pipelines:-name:'my_first_pipeline'url:'https://github.com/username/my_pipeline'workspace:'my_organization/my_workspace'description:'Mytestpipeline'labels:'yeast,test_data'compute-env:'my_compute_environment'work-dir:'s3://my_bucket'profile:'test'params-file:'/path/to/params.yaml'revision:'1.0'In this example:name,url,workspace, etc., are the keys derived from the CLI options.The corresponding values are user-definedBest Practices:Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting.Use quotes around strings that contain special characters or spaces.When listing multiple values (labels,instance-types,allow-buckets, etc), separate them with commas as shown above.For complex configurations, refer to theTemplatesprovided in this repository.TemplatesWe have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in thetemplates/directory and should form a good starting point for you to add your own customization:organizations.ymlteams.ymlworkspaces.ymlparticipants.ymlcredentials.ymlsecrets.ymlcompute-envs.ymlactions.ymldatasets.ymlpipelines.ymllaunch.ymlReal world examplePlease seeseqerakit-e2e.ymlfor an end-to-end example that highlights how you can useseqerakitto create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline.You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it withseqerakit examples/yaml/seqerakit-e2e.yml, you will have to set the following environment variables:$TOWER_GITHUB_PASSWORD$DOCKERHUB_PASSWORD$AWS_ACCESS_KEY_ID$AWS_SECRET_ACCESS_KEY$AWS_ASSUME_ROLE_ARN$AZURE_BATCH_KEY$AZURE_STORAGE_KEY$GOOGLE_KEY$SENTIEON_LICENSE_BASE64Contributions and SupportIf you would like to contribute toseqerakit, please see thecontributing guidelines.For further information or help, please don't hesitate to create anissuein this repository.Creditsseqerakitwas written byEsha Joshi,Adam TalbotandHarshil Patelfrom the Scientific Development Team atSeqera Labs.