Package: ngs-pipeline-lib
IntroductionProvide :A base set of tools and classes to implement BioInfo algorithmsExecution context, using thengs-runscriptInstallationAdd the CodeArtifact repository to your pyproject.toml[[tool.poetry.source]]
name = "codeartifact"
url = "https://pdx-platform-224016688692.d.codeartifact.eu-west-1.amazonaws.com/pypi/pdx-python-libs/simple/"
secondary = trueThen authenticate your local environment to CodeArtifactexportCODEARTIFACT_AUTH_TOKEN=$(awscodeartifactget-authorization-token--domainpdx-platform--queryauthorizationToken--outputtext--profileADX_DEV)poetryconfighttp-basic.codeartifactaws$CODEARTIFACT_AUTH_TOKENNote: The token acquired with AWS is a temporary one. Each time you want to download new packages from the CodeArtifact repository, you may have to re-do the authentication process.Then, simply add the library to your poetry dependencies.poetryaddngs-pipeline-lib--sourcecodeartifactUpdateTo update to a newer version of the library :poetryupdatengs-pipeline-libYou may need to update your version constraint in the pyproject.toml fileGetting startedOnce the library has been installed in your project, you can implement your algorithms by extending theAlgorithmclass.If you want to add specific inputs to your Algorithm, extendBaseInputs(which is a Pydantic Model) and use it as the inputs Type.In order to adds outputs, extend BaseOutputs and set theoutputs_classclass attribute of your algorithm as this class.If you have specific inputs or outputs classes, you should also provide them toAlgorithmwhen subclassing it. You place them between brackets as shown below, this will help your IDE undestand what kind of object it is dealing with, thus improving the autocompletion and the tooltips.frompydanticimportFieldfromngs_pipeline_lib.base.algorithmimportAlgorithmfromngs_pipeline_lib.base.inputsimportBaseInputsfromngs_pipeline_lib.base.fileimportJsonFilefromngs_pipeline_lib.base.outputsimportBaseOutputsclassYourInputs(BaseInputs):your_input:str=Field(description="Description")classYourOutputs(BaseOutputs):def__init__(self):super().__init__()self.my_own_output=JsonFile(name="my_json_file")classYourAlgorithm(Algorithm[YourInputs,YourOutputs]):outputs_class=YourOutputsdefexecute_stub(self):...defexecute_implementation(self):print(self.inputs.your_input)...Then, you can call your Algorithm with the following commandpoetry run ngs-run --sample-id 1If you want to only create the stub output file, add the--stubparameter.ExampleInexample/you'll find the implementation of a dummy algorithmDemoAlgorithm.This algorithm takes 3 parameters:value: an integerkb: a path to a knowledge base (a folder that containsinfo.jsonwhich references other local inputs and/or contains values), here it holdsvalue: an optional float (you can safely remove it frominfo.json)json_file: a path to a json filetext_file: a path to a text fileTheexample/datafolder contains some dummy data to run the algorithm.You can call it (from withinexample/) using :poetryrunngs-run--sample-idsome_id--publish-dir/tmp--text-filedata/some_text_file.txt--kb.pathdata/demo_kbNote that for aKnowledgeBasethe argument's name passed should be--<name of the knowledge base>.path.Add the--stubflag to run the stub instead of the implementation.Docker build & pushThis library also includes two utilitary scripts to build & push Docker image :ngs-buildngs-pushBuildThis script accepts the following arguments :Short ArgLong ArgDescriptionMandatory ?Default value-e--env-filePath to env file to useNo.envThis script accepts the following environment variables as parametersENV VARDescriptionMandatory ?Default valuePROCESS_NAMEName of the processYes--IMAGE_PREFIXPrefix used with the process name to create Docker repo nameNongs-pipeline-process-TAGTag of the image to createNolatestDOCKERFILERelative path to DockerfileNoDockerfilePIP_REGISTRY_USERNAMEIf needed, username to use for pip authNo--PIP_REGISTRY_PASSWORDIf needed, password to use for pip authNo--Note : the docker context used to build is.PushThis script accepts the following arguments :Short ArgLong ArgDescriptionMandatory ?Default value-e--env-filePath to env file to useNo.envThis script accepts the following environment variables as parametersENV VARDescriptionMandatory ?Defaut valueEXTERNAL_REGISTRY_URLURL of Destination RegistryYes--PROCESS_NAMEName of the process to pushYes--IMAGE_PREFIXPrefix used in the process Docker repo nameNongs-pipeline-process-TAGTag of the image to createNolatestDOCKER_USERNAMEIf needed, username to use for docker authNo--DOCKER_PASSWORDIf needed, password to use for docker authNo--Test toolsThis library comes with two tools :Integration test, to verify the behaviour of one processE2E test, to verify the workflow of a complete pipeline with one/multiple samplesTo run the integration test :"""bash
poetry run ngs-test integration
"""To run the E2E test :"""bash
poetry run ngs-test e2e --output-path <<YOUR_PIPELINE_OUTPUT_DIR>>  --scenario-file <<SCENARIO_PATH>>
"""End-To-End TestThis tool will do the following :Load specified test scenarioCan be local or S3 pathLoad specified pipeline run :Import the trace fileImport the hashed_id mapping fileImport the execution.json fileExplore all published files per sample & processCompare scenario and runCheck samples consistency (missing or extra)Check task consistency for each sample (missing or extra, but also status)Check published files for each task on each samples (missing or extra)Note : no validation is done on the file's content, only its presence. Please use integration tests for this purpose.Execution paramsArgDescriptionMandatory ?Default value--output-pathThe pipeline run to verifyYes--scenario-fileThe scenario to use, containing expected samples, tasks & filesYesSettingsAll these settings are primarily loaded from the.envfile. Ensure to have that file before running E2E tests.
They can be overridden by manually defining environment variables before launchingngs-test integration.Environment VariableDescriptionMandatory ?Default valuePROFILEAWS Profile to use when connecting to S3 through boto3YesADX_DEVTEST_OUTPUT_FOLDERLocal path where to store results and expected output of pipelineNotests/e2e/outputsNEXTFLOW_TRACE_FILETrace file path to look for in the output_folderYestrace.txtNEXTFLOW_HASHED_ID_FILEsample_to_hashed_id file path to look for in the output_folderNosample_to_hash_map.tsvNEXTFLOW_EXECUTION_FILEExecution file path to look for in the output_folderNoexecution.jsonIntegration TestThis tool must be used within a process project, containing a.envfile with standard envvar (image_prefix, process_name etc...)
When used, this tool will do the following :Load all scenariosSearch for scenario in a dedicated folder (by default : tests/integration/scenarios)Can be filtered with the paramname_filterFor each scenarioRun the process image with the specified inputsExtract output from containerDownload expected outputCompare outputsExecution paramsArgDescriptionMandatory ?Default value--name-filterSpecify some filter on scenario name. Can be used multiple times (logicalANDapplied between filters)No--post-clean / --no-post-cleanFlag to enable/disable cleaning of input and output files after test completion.NoTrueSettingsAll these settings are primarly loaded from.envfile. Ensure to have that file before running integration tests.
They can be overriden by defining manually environment variables before launchingngs-test integration.Environment VariableDescriptionMandatory ?Default valueREMOTE_DOCKER_REPODocker repository to use when launching process containerYesIMAGE_PREFIXCombined with PROCESS_NAME to set the docker image to use when launching process containerYesPROCESS_NAMECombined with IMAGE_PREFIX to set the docker image to use when launching process containerYesTAGDocker tag to use when launching process containerYesPROFILEAWS Profile to use when connecting to S3/ECR through boto3YesTEST_SCENARIOS_FOLDERLocal path where to look for scenariosNotests/integration/scenariosTEST_OUTPUT_FOLDERLocal path where to store results and expected output of processNotests/integration/outputsTEST_LOCAL_INPUT_FOLDERLocal path where to put input files (downloaded from S3)Notests/integration/inputsTEST_CONFIGURATION_FILENAMEFilename to look for when loading a scenarioNotest.jsonBest PracticesWhen implementing your process, please refer to theguidelinesdocumentation.