Repository: seqeralabs/seqera-kit (language: python)

Issue:
feat: add hello world quick start script/example and e2e yaml
- Add an example of importing `tw-pywrap` package into python script to launch pipeline in `community/showcase`
- Add e2e yaml as an example for eventual import of CI pipeline from tw-pywrap-ci


Discussion / hints:

Fix patch:
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index 2dc5f05..5783248 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -26,7 +26,7 @@ Install the dependencies locally before prior to development. You can use a virt
 
 ```console
 conda env create -f environment.yml
-conda activate tw_pywrap
+conda activate tw-pywrap
 ```
 
 You can then install the local repository for development using the `pip -e` command which will install it in place without copying it to your `PYTHONPATH`. Using `--no-deps` will ignore dependencies which have already been installed via Conda. This assumes the current working directory is a clone of this repository.
@@ -55,7 +55,7 @@ You can debug in VSCode for stepping through code and monitoring errors. To do s
       "name": "Python: Current File",
       "type": "python",
       "request": "launch",
-      "program": "${workspaceFolder}/tw_pywrap/cli.py",
+      "program": "${workspaceFolder}/tw-pywrap/cli.py",
       "args": ["-l", "debug", "example.yml"], // Change to your specific commands you wish to run
       "cwd": "${workspaceFolder}"
     }
@@ -73,7 +73,7 @@ You may need additional settings based on your set-up. A more complete example m
       "name": "Python: Current File",
       "type": "python",
       "request": "launch",
-      "program": "${workspaceFolder}/tw_py/cli.py",
+      "program": "${workspaceFolder}/tw-pywrap/cli.py",
       "console": "integratedTerminal",
       "purpose": ["debugging"],
       "justMyCode": true,
diff --git a/.gitignore b/.gitignore
index df26afb..2f2cbd6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,5 +13,5 @@ build/lib/*
 node_modules/
 package-lock.json
 package.json
-tw_pywrap.egg-info/*
+tw-pywrap.egg-info/*
 build/*
diff --git a/README.md b/README.md
index 2bd713c..46a25f2 100644
--- a/README.md
+++ b/README.md
@@ -16,17 +16,17 @@ You will need to have an account on Nextflow Tower (see [Plans and pricing](http
 
 `tw-pywrap` requires the following dependencies:
 
-  1. [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli#1-installation)
+1. [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli#1-installation)
 
-  2. [Python (`>=3.8`)](https://www.python.org/downloads/)
-    
-  3. [PyYAML](https://pypi.org/project/PyYAML/)
+2. [Python (`>=3.8`)](https://www.python.org/downloads/)
+
+3. [PyYAML](https://pypi.org/project/PyYAML/)
 
 Alternatively, you can install the dependencies via Conda by downloading and using the [Conda environment file](environment.yml) that has been supplied in this repository:
 
 ```console
 conda env create -f environment.yml
-conda activate tw_pywrap
+conda activate tw-pywrap
 ```
 
 ### 2. Installation
@@ -55,79 +55,68 @@ export TOWER_ACCESS_TOKEN=<your access token>
 
 ## Quick start
 
-You must provide a YAML file that defines the options for each of the entities you would like to create in Nextflow Tower. 
-
-You will need to have an account on Nextflow Tower (see [Plans and pricing](https://cloud.tower.nf/pricing/)).
-
-- Launching on Tower Cloud
+You must provide a YAML file that defines the options for each of the entities you would like to create in Nextflow Tower.
 
-  If you have an account on Tower Cloud you should have access to the [`community/showcase`](https://seqera.io/blog/introducing-the-tower-cloud-community-workspace/) Workspace. Let's launch a pipeline there!
+You will need to have an account on Nextflow Tower (see [Plans and pricing](https://cloud.tower.nf/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.
 
-  1. Create a file called `hello_world_config.yml` with the contents below:
+### Launch via YAML
 
-      ```
-      launch:
-        - name: 'hello-world'                               # Workflow name
-          workspace: 'community/showcase'                   # Workspace name
-          compute-env: 'AWS_Batch_Ireland_FusionV2_NVMe'    # Compute environment
-          revision: 'master'                                # Pipeline revision
-          pipeline: 'https://github.com/nextflow-io/hello'  # Pipeline URL
-      ```
+1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>'` entries as required:
 
-  2. Launch the pipeline with `tw-pywrap`:
+   ```
+   launch:
+     - name: 'hello-world'                               # Workflow name
+       workspace: '<YOUR_WORKSPACE>'                     # Workspace name
+       compute-env: '<YOUR_COMPUTE_ENVIRONMENT>'         # Compute environment
+       revision: 'master'                                # Pipeline revision
+       pipeline: 'https://github.com/nextflow-io/hello'  # Pipeline URL
+   ```
 
-      ```
-      tw-pywrap hello_world_config.yml
-      ```
+2. Launch the pipeline with `tw-pywrap`:
 
-  3. Login to Tower Cloud and check the [Runs page](https://tower.nf/orgs/community/workspaces/showcase/watch]) in the `community/showcase` for the pipeline you just launched!
+   ```
+   tw-pywrap hello-world-config.yml
+   ```
 
-- Launching on Tower Enterprise
+3. Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
 
-  If you are a customer with Seqera Labs you will need access to a Workspace with Launch permissions, as well as a pre-defined Compute Environment where you can launch a pipeline. Please customise the entries in the config below as required.
+### Launch via a Python script
 
-  1. Create a file called `hello_world_config.yml` with the contents below:
+You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Tower CLI by leveraging the flexibility and customisation options available in Python.
 
-      ```
-      launch:
-        - name: 'hello-world'                               # Workflow name
-          workspace: '<YOUR_WORKSPACE>'                     # Workspace name
-          compute-env: '<YOUR_COMPUTE_ENVIRONMENT>'         # Compute environment
-          revision: 'master'                                # Pipeline revision
-          pipeline: 'https://github.com/nextflow-io/hello'  # Pipeline URL
-      ```
+1. Download the [`launch_hello_world.py`](examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>'` entries as required.
 
-  2. Launch the pipeline with `tw-pywrap`:
+2. Launch the pipeline with `tw-pywrap`:
 
-      ```
-      tw-pywrap hello_world_config.yml
-      ```
+   ```
+   python launch_hello_world.py
+   ```
 
-  3. Login to your Tower Enterprise instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
+3. Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
 
 ## Real world example
 
-We have created an end-to-end example that highlights how you can use `tw_pywrap` to create everything sequentially in Nextflow Tower all the way from adding a new Organization to launching a pipeline.
+Please see [`tw-pywrap-e2e.yml`](examples/yaml/tw-pywrap-e2e.yml) for an end-to-end example that highlights how you can use `tw-pywrap` to create everything sequentially in Nextflow Tower all the way from creating a new Organization to launching a pipeline.
 
 ## Templates
 
 We have provided template YAML files for each of the entities that can be created on Tower. These can be found in the [`templates/`](templates) directory and should form a good starting point for you to add your own customization:
 
-  - [organizations.yml](templates/organizations.yml)
-  - [teams.yml](templates/teams.yml)
-  - [workspaces.yml](templates/workspaces.yml)
-  - [participants.yml](templates/participants.yml)
-  - [credentials.yml](templates/credentials.yml)
-  - [secrets.yml](templates/secrets.yml)
-  - [compute-envs.yml](templates/compute-envs.yml)
-  - [actions.yml](templates/actions.yml)
-  - [datasets.yml](templates/datasets.yml)
-  - [pipelines.yml](templates/pipelines.yml)
-  - [launch.yml](templates/launch.yml)
+- [organizations.yml](templates/organizations.yml)
+- [teams.yml](templates/teams.yml)
+- [workspaces.yml](templates/workspaces.yml)
+- [participants.yml](templates/participants.yml)
+- [credentials.yml](templates/credentials.yml)
+- [secrets.yml](templates/secrets.yml)
+- [compute-envs.yml](templates/compute-envs.yml)
+- [actions.yml](templates/actions.yml)
+- [datasets.yml](templates/datasets.yml)
+- [pipelines.yml](templates/pipelines.yml)
+- [launch.yml](templates/launch.yml)
 
 ## Contributions and Support
 
-If you would like to contribute to `tw_pywrap`, please see the [contributing guidelines](.github/CONTRIBUTING.md).
+If you would like to contribute to `tw-pywrap`, please see the [contributing guidelines](.github/CONTRIBUTING.md).
 
 For further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/tw-pywrap/issues) in this repository.
 
diff --git a/environment.yml b/environment.yml
index 51ea761..f927c18 100644
--- a/environment.yml
+++ b/environment.yml
@@ -1,4 +1,4 @@
-name: tw_pywrap
+name: tw-pywrap
 channels:
   - bioconda
   - conda-forge
diff --git a/examples/python/launch_hello_world.py b/examples/python/launch_hello_world.py
new file mode 100755
index 0000000..3f3ba42
--- /dev/null
+++ b/examples/python/launch_hello_world.py
@@ -0,0 +1,32 @@
+import logging
+
+# Import the tw-pywrap package
+from tw_pywrap import tower
+
+logging.basicConfig(level=logging.DEBUG)
+
+# Construct a new tw-pywrap Tower instance
+tw = tower.Tower()
+
+# Customise the entries below as required
+workspace = "<YOUR_WORKSPACE>"  # Name of your Workspace
+compute_env = "<YOUR_COMPUTE_ENVIRONMENT>"  # Name of your Compute Environment
+
+# Specify a human-readable run name
+run_name = "hello-world-tw-pywrap"
+
+# Launch the 'hello-world' pipeline using the 'launch' method
+pipeline_run = tw.launch(
+    "--workspace",
+    workspace,
+    "--compute-env",
+    compute_env,
+    "--name",
+    run_name,
+    "--revision",
+    "master",
+    "--wait",
+    "SUBMITTED",
+    "https://github.com/nextflow-io/hello",
+    to_json=True,
+)
diff --git a/examples/yaml/compute-envs/seqera_aws_ireland_fusionv2_nvme.json b/examples/yaml/compute-envs/seqera_aws_ireland_fusionv2_nvme.json
new file mode 100644
index 0000000..042d281
--- /dev/null
+++ b/examples/yaml/compute-envs/seqera_aws_ireland_fusionv2_nvme.json
@@ -0,0 +1,23 @@
+{
+  "region" : "eu-west-1",
+  "workDir" : "s3://seqeralabs-showcase",
+  "environment" : [ ],
+  "waveEnabled" : true,
+  "fusion2Enabled" : true,
+  "nvnmeStorageEnabled" : true,
+  "forge" : {
+    "type" : "SPOT",
+    "minCpus" : 0,
+    "maxCpus" : 500,
+    "gpuEnabled" : false,
+    "instanceTypes" : [ "c6id", "m6id", "r6id" ],
+    "subnets" : [ ],
+    "securityGroups" : [ ],
+    "disposeOnDeletion" : true,
+    "allowBuckets" : [ "s3://ngi-igenomes", "s3://nf-core-awsmegatests", "s3://seqeralabs-showcase" ],
+    "efsCreate" : false,
+    "dragenEnabled" : false,
+    "ebsBootSize" : 50
+  },
+  "discriminator" : "aws-batch"
+}
diff --git a/examples/yaml/compute-envs/seqera_azure_virginia.json b/examples/yaml/compute-envs/seqera_azure_virginia.json
new file mode 100644
index 0000000..ea7eb64
--- /dev/null
+++ b/examples/yaml/compute-envs/seqera_azure_virginia.json
@@ -0,0 +1,14 @@
+{
+  "workDir" : "az://seqeralabs-showcase",
+  "region" : "eastus",
+  "forge" : {
+    "vmType" : "Standard_D16s_v3",
+    "vmCount" : 16,
+    "autoScale" : true,
+    "disposeOnDeletion" : true
+  },
+  "deleteJobsOnCompletion" : "on_success",
+  "deletePoolsOnCompletion" : false,
+  "environment" : [ ],
+  "discriminator" : "azure-batch"
+}
diff --git a/examples/yaml/compute-envs/seqera_gcp_finland.json b/examples/yaml/compute-envs/seqera_gcp_finland.json
new file mode 100644
index 0000000..7861760
--- /dev/null
+++ b/examples/yaml/compute-envs/seqera_gcp_finland.json
@@ -0,0 +1,10 @@
+{
+  "location" : "europe-north1",
+  "workDir" : "gs://seqeralabs-showcase-eu-north-1",
+  "spot" : true,
+  "projectId" : "tower-cloud-testing",
+  "sshDaemon" : false,
+  "usePrivateAddress" : false,
+  "environment" : [ ],
+  "discriminator" : "google-batch"
+}
diff --git a/examples/yaml/datasets/rnaseq_samples.csv b/examples/yaml/datasets/rnaseq_samples.csv
new file mode 100644
index 0000000..6a6fcbd
--- /dev/null
+++ b/examples/yaml/datasets/rnaseq_samples.csv
@@ -0,0 +1,8 @@
+sample,fastq_1,fastq_2,strandedness
+WT_REP1,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357070_1.fastq.gz,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357070_2.fastq.gz,auto
+WT_REP1,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357071_1.fastq.gz,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357071_2.fastq.gz,auto
+WT_REP2,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357072_1.fastq.gz,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357072_2.fastq.gz,reverse
+RAP1_UNINDUCED_REP1,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357073_1.fastq.gz,,auto
+RAP1_UNINDUCED_REP2,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357074_1.fastq.gz,,reverse
+RAP1_UNINDUCED_REP2,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357075_1.fastq.gz,,reverse
+RAP1_IAA_30M_REP1,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357076_1.fastq.gz,s3://ngi-igenomes/test-data/rnaseq/minimal/GSE110004/SRR6357076_2.fastq.gz,reverse
\ No newline at end of file
diff --git a/examples/yaml/datasets/sarek_samples.csv b/examples/yaml/datasets/sarek_samples.csv
new file mode 100644
index 0000000..2c9c7cb
--- /dev/null
+++ b/examples/yaml/datasets/sarek_samples.csv
@@ -0,0 +1,2 @@
+patient,sex,status,sample,lane,fastq_1,fastq_2
+test,XX,0,test,test_L1,https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/homo_sapiens/illumina/fastq/test_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/homo_sapiens/illumina/fastq/test_2.fastq.gz
diff --git a/examples/yaml/datasets/sentieon_samples.csv b/examples/yaml/datasets/sentieon_samples.csv
new file mode 100644
index 0000000..0ef2c2c
--- /dev/null
+++ b/examples/yaml/datasets/sentieon_samples.csv
@@ -0,0 +1,6 @@
+sample,rg_id,rg_lb,rg_pl,fastq_1,fastq_2
+NA12878,NA12878.H88WKADXX.1.CGATGT-1,NA12878.U0a,ILLUMINA,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140744_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140744_R2.fastq.gz
+NA12878,NA12878.H88WKADXX.1.CGATGT-2,NA12878.U0a,ILLUMINA,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140746_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140746_R2.fastq.gz
+NA13000,NA13000.H90WKADXX.1.GCCAAT-1,NA13000.U5a,ILLUMINA,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140748_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140748_R2.fastq.gz
+NA14000,NA14000.H91WKADXX.1.GTGAAA-1,NA14000.U1b,ILLUMINA,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140750_R1.fastq.gz,
+NA14000,NA14000.H91WKADXX.1.GTGAAA-2,NA14000.U1b,ILLUMINA,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/sispa/SRR11140748_R1.fastq.gz,
diff --git a/examples/yaml/datasets/viralrecon_illumina_samples.csv b/examples/yaml/datasets/viralrecon_illumina_samples.csv
new file mode 100644
index 0000000..6163ceb
--- /dev/null
+++ b/examples/yaml/datasets/viralrecon_illumina_samples.csv
@@ -0,0 +1,5 @@
+sample,fastq_1,fastq_2
+SAMPLE1_PE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample1_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample1_R2.fastq.gz
+SAMPLE2_PE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample2_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample2_R2.fastq.gz
+SAMPLE3_SE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample1_R1.fastq.gz,
+SAMPLE3_SE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample2_R1.fastq.gz,
diff --git a/examples/yaml/datasets/viralrecon_nanopore_samples.csv b/examples/yaml/datasets/viralrecon_nanopore_samples.csv
new file mode 100644
index 0000000..38e64b3
--- /dev/null
+++ b/examples/yaml/datasets/viralrecon_nanopore_samples.csv
@@ -0,0 +1,12 @@
+sample,barcode
+SAMPLE_01,1
+SAMPLE_02,2
+SAMPLE_03,3
+SAMPLE_04,4
+SAMPLE_05,5
+SAMPLE_06,6
+SAMPLE_07,7
+SAMPLE_08,8
+SAMPLE_09,9
+SAMPLE_10,10
+SAMPLE_11,11
diff --git a/examples/yaml/pipelines/nextflow.config b/examples/yaml/pipelines/nextflow.config
new file mode 100644
index 0000000..612d322
--- /dev/null
+++ b/examples/yaml/pipelines/nextflow.config
@@ -0,0 +1,1 @@
+process.maxRetries = 1
\ No newline at end of file
diff --git a/examples/yaml/pipelines/nf-core-sarek_pipeline.json b/examples/yaml/pipelines/nf-core-sarek_pipeline.json
new file mode 100644
index 0000000..bb6fdb3
--- /dev/null
+++ b/examples/yaml/pipelines/nf-core-sarek_pipeline.json
@@ -0,0 +1,15 @@
+{
+  "description" : "Analysis pipeline to detect germline or somatic variants (pre-processing, variant calling and annotation) from WGS / targeted sequencing.",
+  "launch" : {
+    "pipeline" : "https://github.com/nf-core/sarek",
+    "workDir" : "az://seqeralabs-showcase",
+    "revision" : "3.1.2",
+    "configProfiles" : [ "test" ],
+    "configText" : "",
+    "paramsText" : "outdir: 'az://seqeralabs-showcase/nf-core-sarek/results'",
+    "preRunScript" : "",
+    "resume" : false,
+    "pullLatest" : false,
+    "stubRun" : false
+  }
+}
diff --git a/examples/yaml/pipelines/nf_core_rnaseq_params.yml b/examples/yaml/pipelines/nf_core_rnaseq_params.yml
new file mode 100644
index 0000000..ce0e8c9
--- /dev/null
+++ b/examples/yaml/pipelines/nf_core_rnaseq_params.yml
@@ -0,0 +1,1 @@
+outdir: 'az://seqeralabs-showcase/nf-core-rnaseq/results'
\ No newline at end of file
diff --git a/examples/yaml/pipelines/nf_core_rnaseq_pipeline.json b/examples/yaml/pipelines/nf_core_rnaseq_pipeline.json
new file mode 100644
index 0000000..d54eb8f
--- /dev/null
+++ b/examples/yaml/pipelines/nf_core_rnaseq_pipeline.json
@@ -0,0 +1,17 @@
+{
+  "description" : "RNA sequencing analysis pipeline using STAR, RSEM, HISAT2 or Salmon with gene/isoform counts and extensive quality control.",
+  "launch" : {
+    "pipeline" : "https://github.com/nf-core/rnaseq",
+    "workDir" : "s3://seqeralabs-showcase",
+    "revision" : "3.12.0",
+    "configProfiles" : [ "test" ],
+    "userSecrets" : [ ],
+    "workspaceSecrets" : [ ],
+    "configText" : "",
+    "paramsText" : "outdir: 's3://seqeralabs-showcase/nf-core-rnaseq/results'",
+    "preRunScript" : "",
+    "resume" : false,
+    "pullLatest" : false,
+    "stubRun" : false
+  }
+}
diff --git a/examples/yaml/pipelines/nf_core_viralrecon_illumina_params.yml b/examples/yaml/pipelines/nf_core_viralrecon_illumina_params.yml
new file mode 100644
index 0000000..e369cf8
--- /dev/null
+++ b/examples/yaml/pipelines/nf_core_viralrecon_illumina_params.yml
@@ -0,0 +1,1 @@
+outdir: 'gs://seqeralabs-showcase-eu-north-1/nf-core-viralrecon/illumina/results'
\ No newline at end of file
diff --git a/examples/yaml/pipelines/nf_core_viralrecon_illumina_pipeline.json b/examples/yaml/pipelines/nf_core_viralrecon_illumina_pipeline.json
new file mode 100644
index 0000000..0493cb7
--- /dev/null
+++ b/examples/yaml/pipelines/nf_core_viralrecon_illumina_pipeline.json
@@ -0,0 +1,15 @@
+{
+  "description" : "Pipeline for assembly and intrahost/low-frequency variant calling for viral samples.",
+  "launch" : {
+    "pipeline" : "https://github.com/nf-core/viralrecon",
+    "workDir" : "s3://seqeralabs-showcase",
+    "revision" : "2.6.0",
+    "configProfiles" : [ "test" ],
+    "configText" : "",
+    "paramsText" : "outdir: 's3://seqeralabs-showcase/nf-core-viralrecon/illumina/results'",
+    "preRunScript" : "",
+    "resume" : false,
+    "pullLatest" : false,
+    "stubRun" : false
+  }
+}
diff --git a/examples/yaml/pipelines/nf_core_viralrecon_nanopore_pipeline.json b/examples/yaml/pipelines/nf_core_viralrecon_nanopore_pipeline.json
new file mode 100644
index 0000000..829d37a
--- /dev/null
+++ b/examples/yaml/pipelines/nf_core_viralrecon_nanopore_pipeline.json
@@ -0,0 +1,15 @@
+{
+  "description" : "Pipeline for assembly and intrahost/low-frequency variant calling for viral samples.",
+  "launch" : {
+    "pipeline" : "https://github.com/nf-core/viralrecon",
+    "workDir" : "s3://seqeralabs-showcase",
+    "revision" : "2.6.0",
+    "configProfiles" : [ "test_nanopore" ],
+    "configText" : "",
+    "paramsText" : "outdir: 's3://seqeralabs-showcase/nf-core-viralrecon/nanopore/results'",
+    "preRunScript" : "",
+    "resume" : false,
+    "pullLatest" : false,
+    "stubRun" : false
+  }
+}
diff --git a/examples/yaml/pipelines/nf_sentieon_pipeline.json b/examples/yaml/pipelines/nf_sentieon_pipeline.json
new file mode 100644
index 0000000..2423e5b
--- /dev/null
+++ b/examples/yaml/pipelines/nf_sentieon_pipeline.json
@@ -0,0 +1,17 @@
+{
+  "description" : "Proof-of-concept Nextflow pipeline to run Sentieon software.\nhttps://www.sentieon.com/",
+  "launch" : {
+    "pipeline" : "https://github.com/seqeralabs/nf-sentieon",
+    "workDir" : "s3://seqeralabs-showcase",
+    "revision" : "master",
+    "configProfiles" : [ "test" ],
+    "userSecrets" : [ ],
+    "workspaceSecrets" : [ "SENTIEON_LICENSE_BASE64" ],
+    "configText" : "",
+    "paramsText" : "outdir: 's3://seqeralabs-showcase/nf-sentieon/results'",
+    "preRunScript" : "",
+    "resume" : false,
+    "pullLatest" : false,
+    "stubRun" : false
+  }
+}
diff --git a/examples/yaml/pipelines/pre_run.txt b/examples/yaml/pipelines/pre_run.txt
new file mode 100644
index 0000000..3e565b1
--- /dev/null
+++ b/examples/yaml/pipelines/pre_run.txt
@@ -0,0 +1,1 @@
+export NXF_VER=23.06.0-edge
\ No newline at end of file


Test patch:
diff --git a/.github/workflows/e2e-testing.yml b/.github/workflows/e2e-testing.yml
new file mode 100644
index 0000000..a0d4c57
--- /dev/null
+++ b/.github/workflows/e2e-testing.yml
@@ -0,0 +1,55 @@
+name: e2e-testing
+run-name: Create all entities in Tower from end-to-end with tw-pywrap
+# This workflow can be triggered manually with GitHub actions workflow dispatch button.
+# It will automate the end-to-end creation all of the following entities in Nextflow Tower.
+
+on:
+  workflow_dispatch:
+
+jobs:
+  e2e-testing:
+    name:
+    if: github.repository == 'seqeralabs/tw-pywrap'
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -el {0}
+    env:
+      TOWER_ACCESS_TOKEN: ${{ secrets.TOWER_ACCESS_TOKEN }}
+      TOWER_GITHUB_PASSWORD: ${{ secrets.TOWER_GITHUB_PASSWORD }}
+      DOCKERHUB_PASSWORD: ${{ secrets.DOCKERHUB_PASSWORD }}
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      AWS_ASSUME_ROLE_ARN: ${{ secrets.AWS_ASSUME_ROLE_ARN }}
+      AZURE_BATCH_KEY: ${{ secrets.AZURE_BATCH_KEY }}
+      AZURE_STORAGE_KEY: ${{ secrets.AZURE_STORAGE_KEY }}
+      GOOGLE_KEY: ${{ secrets.GOOGLE_KEY }}
+      SENTIEON_LICENSE_BASE64: ${{ secrets.SENTIEON_LICENSE_BASE64 }}
+    steps:
+      - name: Check out source-code repository
+        uses: actions/checkout@v3
+
+      - name: Setup conda
+        uses: conda-incubator/setup-miniconda@v2.2.0
+        with:
+          auto-update-conda: true
+          environment-file: environment.yml
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          python-version: "3.10"
+          activate-environment: tw-pywrap
+          use-mamba: true
+
+      - name: Install pip & tw-pywrap
+        run: |
+          mamba install -y -c conda-forge pip
+          pip install -e .
+
+      - name: Create all entities on Tower from end-to-end
+        run: |
+          mamba init
+          temp_file=$(mktemp -q)
+          echo $GOOGLE_KEY | base64 -d > $temp_file
+          export GOOGLE_KEY=$temp_file
+
+          tw-pywrap examples/yaml/tw-pywrap-e2e.yml
diff --git a/examples/yaml/tw-pywrap-e2e.yml b/examples/yaml/tw-pywrap-e2e.yml
new file mode 100644
index 0000000..0fc4976
--- /dev/null
+++ b/examples/yaml/tw-pywrap-e2e.yml
@@ -0,0 +1,215 @@
+organizations:
+  - name: 'tw-pywrap-e2e'
+    full-name: 'tw-pywrap-e2e'
+    description: 'Organization created E2E with tw CLI scripting'
+    location: 'Global'
+    website: 'https://seqera.io/'
+    overwrite: True
+teams:
+  - name: 'scidev_team'
+    organization: 'tw-pywrap-e2e'
+    description: 'Scientific Development team @ Seqera Labs'
+    members:
+      - 'esha.joshi@seqera.io'
+      - 'adam.talbot@seqera.io'
+      - 'drpatelhh@gmail.com'
+    overwrite: True
+workspaces:
+  - name: 'showcase'
+    full-name: 'showcase'
+    organization: 'tw-pywrap-e2e'
+    description: 'Workspace created E2E with tw CLI scripting'
+    visibility: 'PRIVATE'
+    overwrite: True
+participants:
+  - name: 'scidev_team'
+    type: 'TEAM'
+    workspace: 'tw-pywrap-e2e/showcase'
+    role: 'ADMIN'
+    overwrite: True
+  - name: 'adam.talbot@seqera.io'
+    type: 'MEMBER'
+    workspace: 'tw-pywrap-e2e/showcase'
+    role: 'LAUNCH'
+    overwrite: True
+credentials:
+  - type: 'github'
+    name: 'github_credentials'
+    workspace: 'tw-pywrap-e2e/showcase'
+    username: 'ejseqera'
+    password: '$TOWER_GITHUB_PASSWORD'
+    overwrite: True
+  - type: 'container-reg' 
+    name: 'dockerhub_credentials'
+    workspace: 'tw-pywrap-e2e/showcase'
+    username: 'ejseqera'
+    password: '$DOCKERHUB_PASSWORD'
+    registry: 'docker.io'
+    overwrite: True
+  - type: 'aws'
+    name: 'aws_credentials'
+    workspace: 'tw-pywrap-e2e/showcase'
+    access-key: '$AWS_ACCESS_KEY_ID'
+    secret-key: '$AWS_SECRET_ACCESS_KEY'
+    assume-role-arn: '$AWS_ASSUME_ROLE_ARN'
+    overwrite: True
+  - type: 'azure'
+    name: 'azure_credentials'
+    workspace: 'tw-pywrap-e2e/showcase'
+    batch-key: '$AZURE_BATCH_KEY'
+    batch-name: 'seqeralabs'
+    storage-key: '$AZURE_STORAGE_KEY'
+    storage-name: 'seqeralabs'
+    overwrite: True
+  - type: 'google'
+    name: 'google_credentials'
+    workspace: 'tw-pywrap-e2e/showcase'
+    key: '$GOOGLE_KEY'
+    overwrite: True
+secrets:
+  - name: 'SENTIEON_LICENSE_BASE64'
+    workspace: 'tw-pywrap-e2e/showcase'
+    value: '$SENTIEON_LICENSE_BASE64'
+    overwrite: True
+compute-envs:
+  - name: 'seqera_aws_ireland_fusionv2_nvme'
+    workspace: 'tw-pywrap-e2e/showcase'
+    credentials: 'aws_credentials'
+    wait: 'AVAILABLE'
+    file-path: './compute-envs/seqera_aws_ireland_fusionv2_nvme.json'
+    overwrite: True
+  - name: 'seqera_azure_virginia'
+    workspace: 'tw-pywrap-e2e/showcase'
+    credentials: 'azure_credentials'
+    wait: 'AVAILABLE'
+    file-path: './compute-envs/seqera_azure_virginia.json'
+    overwrite: True
+  - name: 'seqera_gcp_finland'
+    workspace: 'tw-pywrap-e2e/showcase'
+    credentials: 'google_credentials'
+    wait: 'AVAILABLE'
+    file-path: './compute-envs/seqera_gcp_finland.json'
+    overwrite: True
+actions:
+  - type: 'github'
+    name: 'nf_core_rnaseq_launch_hook'
+    pipeline: 'https://github.com/nf-core/rnaseq'
+    workspace: 'tw-pywrap-e2e/showcase'
+    compute-env: 'seqera_aws_ireland_fusionv2_nvme'
+    work-dir: 's3://seqeralabs-showcase'
+    profile: 'test'
+    params:
+      outdir: 's3://seqeralabs-showcase/nf-core-rnaseq/results'
+    revision: '3.12.0'
+    overwrite: True
+  - type: 'tower'
+    name: 'nf_core_rnaseq_web_hook'
+    pipeline: 'https://github.com/nf-core/rnaseq'
+    workspace: 'tw-pywrap-e2e/showcase'
+    compute-env: 'seqera_aws_ireland_fusionv2_nvme'
+    work-dir: 's3://seqeralabs-showcase'
+    profile: 'test'
+    params:
+      outdir: 's3://seqeralabs-showcase/nf-core-rnaseq/results'
+    revision: '3.12.0'
+    overwrite: True
+datasets:
+  - name: 'rnaseq_samples'
+    description: 'Samplesheet to run the nf-core/rnaseq pipeline from end-to-end'
+    header: true
+    workspace: 'tw-pywrap-e2e/showcase'
+    file-path: './datasets/rnaseq_samples.csv'
+    overwrite: True
+  - name: 'sarek_samples'
+    description: 'Samplesheet to run the nf-core/sarek pipeline from end-to-end'
+    header: true
+    workspace: 'tw-pywrap-e2e/showcase'
+    file-path: './datasets/sarek_samples.csv'
+    overwrite: True
+  - name: 'viralrecon_illumina_samples'
+    description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Illumina data'
+    header: true
+    workspace: 'tw-pywrap-e2e/showcase'
+    file-path: './datasets/viralrecon_illumina_samples.csv'
+    overwrite: True
+  - name: 'viralrecon_nanopore_samples'
+    description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Nanopore data'
+    header: true
+    workspace: 'tw-pywrap-e2e/showcase'
+    file-path: './datasets/viralrecon_nanopore_samples.csv'
+    overwrite: True
+  - name: 'sentieon_samples'
+    description: 'Samplesheet to run the seqeralabs/nf-sentieon pipeline from end-to-end'
+    header: true
+    workspace: 'tw-pywrap-e2e/showcase'
+    file-path: './datasets/sentieon_samples.csv'
+    overwrite: True
+pipelines:
+  - name: 'nf-core-rnaseq'
+    url: 'https://github.com/nf-core/rnaseq'
+    workspace: 'tw-pywrap-e2e/showcase'
+    description: 'RNA sequencing analysis pipeline using STAR, RSEM, HISAT2 or Salmon with gene/isoform counts and extensive quality control.'
+    compute-env: 'seqera_aws_ireland_fusionv2_nvme'
+    work-dir: 's3://seqeralabs-showcase'
+    profile: 'test'
+    revision: '3.12.0'
+    params:
+      outdir: 's3://seqeralabs-showcase/nf-core-rnaseq/results'
+    config: './pipelines/nextflow.config'
+    pre-run: './pipelines/pre_run.txt'
+    overwrite: True
+  - name: 'nf-core-sarek'
+    workspace: 'tw-pywrap-e2e/showcase'
+    compute-env: 'seqera_azure_virginia'
+    file-path: './pipelines/nf-core-sarek_pipeline.json'
+    overwrite: True
+  - name: 'nf-core-viralrecon-nanopore'
+    workspace: 'tw-pywrap-e2e/showcase'
+    compute-env: 'seqera_aws_ireland_fusionv2_nvme'
+    file-path: './pipelines/nf_core_viralrecon_nanopore_pipeline.json'
+    overwrite: True
+  - name: 'nf-core-viralrecon-illumina'
+    url: 'https://github.com/nf-core/viralrecon'
+    workspace: 'tw-pywrap-e2e/showcase'
+    description: 'Pipeline for assembly and intrahost/low-frequency variant calling for viral samples.'
+    compute-env: 'seqera_gcp_finland'
+    work-dir: 'gs://seqeralabs-showcase-eu-north-1'
+    profile: 'test'
+    revision: '2.6.0'
+    params-file: './pipelines/nf_core_viralrecon_illumina_params.yml'
+    config: './pipelines/nextflow.config'
+    pre-run: './pipelines/pre_run.txt'
+    overwrite: True
+  - name: 'nf-sentieon'
+    workspace: 'tw-pywrap-e2e/showcase'
+    compute-env: 'seqera_aws_ireland_fusionv2_nvme'
+    file-path: './pipelines/nf_sentieon_pipeline.json'
+    overwrite: True
+launch:
+  - name: 'nf-core-rnaseq-launchpad'
+    workspace: 'tw-pywrap-e2e/showcase'
+    pipeline: 'nf-core-rnaseq'
+  - name: 'nf-core-rnaseq-remote'
+    workspace: 'tw-pywrap-e2e/showcase'
+    pipeline: 'https://github.com/nf-core/rnaseq'
+    compute-env: 'seqera_azure_virginia'
+    work-dir: 'az://seqeralabs-showcase'
+    profile: 'test'
+    revision: '3.12.0'
+    params-file: './pipelines/nf_core_rnaseq_params.yml'
+    config: './pipelines/nextflow.config'
+    pre-run: './pipelines/pre_run.txt'
+  - name: 'nf-core-sarek'
+    workspace: 'tw-pywrap-e2e/showcase'
+    pipeline: 'nf-core-sarek'
+    config: './pipelines/nextflow.config'
+    pre-run: './pipelines/pre_run.txt'
+  - name: 'nf-core-viralrecon-nanopore'
+    workspace: 'tw-pywrap-e2e/showcase'
+    pipeline: 'nf-core-viralrecon-nanopore'
+  - name: 'nf-core-viralrecon-illumina'
+    workspace: 'tw-pywrap-e2e/showcase'
+    pipeline: 'nf-core-viralrecon-illumina'
+  - name: 'nf-sentieon'
+    workspace: 'tw-pywrap-e2e/showcase'
+    pipeline: 'nf-sentieon'