Package: twkit
twkittwkitis a Python wrapper for theNextflow Tower CLI. It can be leveraged to automate the creation of all of the entities in Nextflow Tower via a simple configuration file in YAML format.The key features are:Simple configuration: All of the command-line options available when using the Nextflow Tower CLI can be defined in simple YAML format.Infrastructure as Code: Enable users to manage and provision their infrastructure specifications.Automation: End-to-end creation of entities within Nextflow Tower, all the way from adding an Organization to launching pipeline(s) within that Organization.PrerequisitesYou will need to have an account on Nextflow Tower (seePlans and pricing).1. Dependenciestwkitrequires the following dependencies:Nextflow Tower CLIPython (>=3.8)PyYAMLAlternatively, you can install the dependencies via Conda by downloading and using theConda environment filethat has been supplied in this repository:conda env create -f environment.ymlconda activate twkit2. InstallationThe scripts in this repository are packaged and available onPyPI. They can be installed viapip:pip install twkitYou can force overwrite the installation to use the latest changes with the command below:pip install --upgrade --force-reinstall twkit3. ConfigurationCreate a Tower access token using theNextflow Towerweb interface via theYour Tokenspage in your profile.twkitreads this token from the environment variableTOWER_ACCESS_TOKEN. Please export it into your terminal as shown below:exportTOWER_ACCESS_TOKEN=<youraccesstoken>UsageUse the -h or --help parameter to list the available commands and their associated options:twkit -hDryrunTo print the commands that would executed withtwwhen using a YAML file, you can runtwkitwith the--dryrunflag:twkit file.yaml --dryrunRecursively deleteInstead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the--deleteflag:twkit file.yaml --deleteFor example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the--deleteflag,twkitwill recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization.Usingtwspecific CLI optionstwspecific CLI options can be specified with the--cli=flag:twkit file.yaml --cli="--arg1 --arg2"You can find the full list of options by runningtw -h.The Tower CLI expects to connect to a Tower instance that is secured by a TLS certificate. If your Tower instance does not present a certificate, you will need to qualify and run yourtwcommands with the--insecureflag.To usetwspecific CLI options such as--insecure, use the--cli=flag, followed by the options you would like to use enclosed in double quotes.For example:twkit file.yaml --cli="--insecure"To use an SSL certificate that is not accepted by the default Java certificate authorities and specify a customcacertsstore as accepted by thetwCLI, you can specify the-Djavax.net.ssl.trustStore=/absolute/path/to/cacertsoption enclosed in double quotes totwkitas you would totw, preceded by--cli=.For example:twkit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts"Note: Use of--verboseoption for thetwCLI is currently not supported bytwkit. Supplying--cli="--verbose"will raise an error.Quick startYou must provide a YAML file that defines the options for each of the entities you would like to create in Nextflow Tower.You will need to have an account on Nextflow Tower (seePlans and pricing). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.Launch via YAMLCreate a YAML file calledhello-world-config.ymlwith the contents below, and customise the<YOUR_WORKSPACE>and<YOUR_COMPUTE_ENVIRONMENT>entries as required:launch:
  - name: 'hello-world'                               # Workflow name
    workspace: '<YOUR_WORKSPACE>'                     # Workspace name
    compute-env: '<YOUR_COMPUTE_ENVIRONMENT>'         # Compute environment
    revision: 'master'                                # Pipeline revision
    pipeline: 'https://github.com/nextflow-io/hello'  # Pipeline URLLaunch the pipeline withtwkit:twkit hello-world-config.ymlLogin to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!Launch via a Python scriptYou can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Tower CLI by leveraging the flexibility and customisation options available in Python.Download thelaunch_hello_world.pyPython script and customise the<YOUR_WORKSPACE>and<YOUR_COMPUTE_ENVIRONMENT>entries as required.Launch the pipeline withtwkit:python launch_hello_world.pyLogin to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!Real world examplePlease seetwkit-e2e.ymlfor an end-to-end example that highlights how you can usetwkitto create everything sequentially in Nextflow Tower all the way from creating a new Organization to launching a pipeline.You can modify this YAML to similarly create Nextflow Tower resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it withtwkit examples/yaml/twkit-e2e.yml, you will have to set the following environment variables:$TOWER_GITHUB_PASSWORD
$DOCKERHUB_PASSWORD
$AWS_ACCESS_KEY_ID
$AWS_SECRET_ACCESS_KEY
$AWS_ASSUME_ROLE_ARN
$AZURE_BATCH_KEY
$AZURE_STORAGE_KEY
$GOOGLE_KEY
$SENTIEON_LICENSE_BASE64TemplatesWe have provided template YAML files for each of the entities that can be created on Tower. These can be found in thetemplates/directory and should form a good starting point for you to add your own customization:organizations.ymlteams.ymlworkspaces.ymlparticipants.ymlcredentials.ymlsecrets.ymlcompute-envs.ymlactions.ymldatasets.ymlpipelines.ymllaunch.ymlContributions and SupportIf you would like to contribute totwkit, please see thecontributing guidelines.For further information or help, please don't hesitate to create anissuein this repository.Creditstwkitwas written byEsha Joshi,Adam TalbotandHarshil Patelfrom the Scientific Development Team atSeqera Labs.