Package: django-nextflow
django-nextflowdjango-nextflow is Django app for running Nextflow pipelines and storing their
results in a database within a Django web app.Installationnextflow.py is available through PyPI:pipinstalldjango-nextflowYou must install the Nextflow executable itself separately: see theNextflow Documentationfor help with this.SetupTo use the app within Django, adddjango-nextflowto your list ofINSTALLED_APPS.You must define four values in yoursettings.py:NEXTFLOW_PIPELINE_ROOT- the location on disk where the Nextflow pipelines
are stored. All references to pipeline files will use this as the root.NEXTFLOW_DATA_ROOT- the location on disk to store execution records.NEXTFLOW_UPLOADS_ROOT- the location on disk to store uploaded data.NEXTFLOW_PUBLISH_DIR- the name of the folder published files will be saved
to. Within an execution directory, django-nextflow will look in
NEXTFLOW_PUBLISH_DIR/process_name for output files for that process. These filesmustbe published as symlinks, not copies, otherwise django-nextflow will not
recognise them.UsageBegin by defining one or more Pipelines. These are .nf files somewhere within
theNEXTFLOW_PIPELINE_ROOTyou defined:fromdjango_nextflow.modelsimportPipelinepipeline=Pipeline.objects.create(path="workflows/main.nf")You can also provide paths to a JSON input schema file (structured using the
nf-core style) and a config file to use when running it:pipeline=Pipeline.objects.create(path="workflows/main.nf",description="Some useful pipeline.",schema_path="main.json",config_path="nextflow.config")print(pipeline.input_schema)# Returns inputs as dictThese can be assigned toPipelineCategoryobjects for organisation.To run the pipeline:execution=pipeline.run(params={"param1":"xxx"})This will run the pipeline using Nextflow, and save database entries for three
different models:TheExecutionthat is returned represents the running of this pipeline on
this occasion. It stores the stdout and stderr of the command, and has aget_log_text()method for reading the full log file from disk. A directory
will be created inNEXTFLOW_DATA_ROOTfor the execution to take place in.ProcessExecutionrecords for each process that execution within the running
of the pipeline. These also have their own stdout and stderr, as well as status
information etc.Datarecords for each file published by the processes in the pipeline. Note
that this is not every file produced - but specifically those output by the
process via its output channel. For this to work the processes must be
configured to publish these files to a particular directory name (the one thatNEXTFLOW_PUBLISH_DIRis set to), and to a subdirectory within that directory
with the process's name.If you want to supply a file for which there is aDataobject as the input to
a pipeline, you can do so as follows:execution=pipeline.run(params={"param1":"xxx"},data_params={"param2":23,"param3":[24,25]})...where 23, 24 and 25 are the IDs ofDataobjects.You can also supply entire executions as inputs, in which case they will be
provided to the pipeline as a directory of symlinked files:execution=pipeline.run(params={"param1":"xxx"},execution_params={"genome1":23,"genome2":24})The aboverunmethod will run the entire pipeline and create the database
objects at the end. To create the Execution object straight away and update it
as execution proceeds, userun_and_update. This can take apost_pollfunction which will execute every time the Execution updates.TheDataobjects above were created by running some pipeline, but you might
want to create one from scratch without running a pipeline. You can do so either
from a path string, or from a DjangoUploadedFileobject:data1=Data.create_from_path("/path/to/file.txt")data2=Data.create_from_upload(django_upload_object)The file will be copied toNEXTFLOW_UPLOADS_ROOTin this case.You can also create a Data object in chunks using:data=Data.create_from_partial_upload(django_upload_object1,filename="large-file.txt")data=Data.create_from_partial_upload(django_upload_object2,data=data)data=Data.create_from_partial_upload(django_upload_object3,data=data,final=True)You can determine all the downstream data of a data object within its generating
execution using thedownstream_within_executionmethod. Likewise theupstream_within_executionmethod will return all upstream data within the
execution.Changelog0.12.14th June, 2022Added compatability with nextflow.py v0.30.122nd May, 2022Executions now save their input params.Execution inputs to executions now organise their inputs by param.0.1122nd April, 2022Data is now marked as binary or non-binary.Data now hascontents()method for returning their plain text contents.0.1019th April, 2022Pipelines can now be organised into categories with a newPipelineCatgeorymodel.Pipelines now have anorderfield for ordering within categories.0.9.323rd March, 2022Fixed data post-delete hook handling of missing upstream objects.0.9.221st March, 2022Added data post-delete behavior.Fixed up/downstream within execution lookups.0.9.17th March, 2022Partial data upload will now optionally verify filesize matches expected.0.96th March, 2022Data objects can now be created from upload objects blob by blob.is_readyfield added to Data to denote those in process of being created.Models now have default ordering.Executions can now be manually created with very minimal information.Fixed missing MD5 hash for pipeline output files.Fixed issue with generating finished time when values were missing.0.827th February, 2022Execution objects now have label and notes fields.Data objects now have label and notes fields.Added method to 'remove' data objects rather than delete them entirely.Data objects now have MD5 hash upon creation.0.719th February, 2022Executions can now generate a graph representation of themselves.Data objects can now detect all their up/downstream data within an execution.0.614th February, 2022Pipelines now have arun_and_updatemethod, for constant Execution updating.Better Execution ID generation.0.53rd February, 2022Pipelines can now take execution inputs.Fixed method for detecting downstream data products.0.412th January, 2022Better support for multiple data objects.Data objects can now be directories, which will be automatically zipped.When creating upstream data connections, data objects will be created if needed.0.3.226th December, 2021Allow IDs to be big ints.0.3.124th December, 2021Data file sizes can now be more than 232.Data file names can now be 1000 characters long.0.321st December, 2021Pipelines can now take multiple data inputs per param.Profiles can now be specified when running a pipeline.Compression extension .gz now ignored when detecting filetype.Process executions start and end times are now recorded.Improved system for identifying upstream data inputs.Improved publish_dir identification.Improved log file reading.0.214th November, 2021Pipelines now have description fields.Data objects now have creation time fields.Added upstream data objects as well as downstream to process executions.0.1.13rd November, 2021Fixed duration string parsing.0.129th October, 2021Initial models for pipelines, execution, process executions and data.