Package: gaia
gaia python clientrunningTo connect to a running Gaia server, find the host (open an ssh tunnel to it if needed) and do the following
(optionally passing in aconfiglike{'gaia_host': 'localhost:24442'}):import gaia
flow = gaia.Gaia()Now that we have a client objectflow, we can call these methods:workflows - list current workflows with summary information about each oneupload - upload a new workflow (properties, Commands, and Steps)command - see what Commands are available and add new Commands to a new or existing workflowmerge - update or add new Steps to a new or existing workflowhalt - stop a running workflowrun - resume running a workflowstatus - get information about a workflowexpire - run the Steps needed to (re)-output the given files and/or Steps as well as their dependent StepsworkflowsTo list current workflows with summary info on each one:flow.workflows()uploadTo get something running, upload a demo workflow:import json
commands = json.load('../../resources/test/demo-commands.json')
steps = json.load('../../resources/test/demo-processes.json')
flow.upload('crick_demo_20191130.121500', {'owner': 'crick'}, commands, steps)Each workflow needs a unique name. The standard practice is to construct a name in the
formowner_program_datetime, e.g.crick_DemoWorkflow_20191209.133734.
This aids sorting and filtering workflows.You will also need to launch some sisyphus workers. To do that
[NOTE: This part is in flux]:flow.launch(['a', 'b'])Launch more if you want : ) Give each a unique name.
They will deallocate 5 minutes after finishing their last Steps.commandCommands are the base level operations that can be run, specifically: command line programs in a given docker container image. Once defined, a Command can be invoked any number of times with a new set of vars, inputs, and outputs.If you call this method with an empty or absent array argument, it will return all Commands in the named workflow.flow.command('biostream')
# [{'name': 'ls', 'image': 'ubuntu', ...}, ...]A Command is expressed as a dictionary with the following keys:name - the Command nameimage - docker image to run incommand - array of shell tokens to executeinputs - map of storage keys to internal paths inside the docker container where the Command's input files will be placedoutputs - map of storage keys to internal paths inside the docker container where the Command's output files will be retrieved after the Command has runvars - map of var keys to string values to insert into Command tokensThey may also have an optionalstdoutkey which specifies what path to place stdout output (so that stdout can be used as one of the outputs of the command).flow.command('biostream', [...])Ifflow.command()is called with an array of Command entries it will merge the given Commands into the workflow, thus adding and/or replacing Commands and triggering the recomputation of any Steps that refer to these Commands.mergeOnce some Commands exist in the workflow you can start merging in Steps to run. Every Step names a Command and sets the Command's vars, inputs, and outputs. Inputs and outputs refer to paths in the data store. Vars are strings that can be spliced into various parts of the Command's shell tokens.Commands and Steps are kept inworkflowswhich are entirely encapsulated from one another. Each workflow has its own data space with its own set of names and values.To call themergemethod, provide a workflow name and an array of Steps:flow.merge('biostream', [{'name': 'ls-home', 'command': 'ls', 'inputs': {...}, ...}, ...])Each Step is a dictionary with the following keys:name - the Step namecommand - name of the Command to invokeinputs - map of input keys defined by the Command to keys in the data store to read the input filesoutputs - map of output keys from the Command to keys in the data store to write the output files after successfully invoking the Commandvars - map of var keys to values. If this is an array it will create a Step for each element in the array with the given valuetimeout - number of seconds to allow the Step to runIf this is a Step with a name that hasn't been seen before, it will create the Step entry and trigger the computation of outputs if the required inputs are available in the data store.  If thenameof the Step being merged already exists in the workflow, that Step will be updated and recomputed, along with all Steps that depend on outputs from the updated Step in that workflow.runTherunmethod simply triggers the computation in the provided workflow if it is not already running:flow.run('biostream')haltThehaltmethod will immediately cancel all running tasks and stop the computation in the given workflow:flow.halt('biostream')statusThestatusmethod provides information about a workflow, formatted as a
dictionary with these keys:state - a string representing the state of the overall workflow. Possible values are 'initialized', 'running', 'complete', 'halted', and 'error'.commands - a list of the workflow's Commandswaiting-inputs - a list of the data inputs (file and directory paths) that Steps are waiting onflow.status('biostream')Or to include internal debugging details from the Gaia server:flow.status('biostream', debug=True)expireTheexpiremethod accepts a workflow and a list of Steps names and storage paths (storage keys). It makes those Steps and their dependent Steps have to run again.flow.expire('biostream', ['ls-home', 'genomes', …])