Package: dplutils
SSEC-JHU dplutilsDistributed Data Pipeline UtilitiesUsage:SetupGet (or build, see below) docker imagedocker pull {TBD}Start clusterTo start a cluster, start one ray head node and any number of worker nodes on network-connected hosts. To start the head
node, running the container using the docker engine, this can be used:docker run -d -n rayhead -v /path/to/data:/data --net host \
  dplutils /opt/startray.sh --head --blockwhich will start the head node (blocking in order that the container stay up). The--net hostoption is given to
expose all open ports on the host as ray requires several bi-directional connections to workers. The-v ...option is
an example of mounted a local path into the container so it can access files (for example directory containing source
data, and output directory). It also exposes a dashboard at 8265 that can be viewed using a web browser.On the workers, similarly start using the command:docker run -d -v /path/to/data:/data --net host \
  dplutils /opt/startray.sh --block --address={head-node}:6379For hosts with custome resources (e.g. other than those that get auto-detected such as CPUs and GPUs), you can pass
resources to the start command:docker run -d --net host -v /path/to/data:/data \
  dplutils /opt/startray.sh --block --address={head-node}:6379 \
  --resources '{"mycustomresource": 1}'In the dashboard you should see the workers listed in the clusters tab.Start pipelinePipelines can be run via interactive python sessions or asynchronously. In an interactive session one would import or
define a pipeline within the session and then callrunorwritetomethod to kick off execution. For longer
running or production jobs it is generally advisable to submit a job to ray. dplutils contains helpers for making it
easy to run configurable pipelines via the command line. For example, assuming a script like:from dplutils.pipeline import PipelineTask
from dplutils.pipeline.ray import RayDataPipelineExecutor
from dplutils.cli import cli_run

if __name__ == '__main__':
   pl = RayDataPipelineExecutor([PipelineTask('task1', lambda x: x.assign(newcol=1))])
   cli_run(pl)We can submit the job in the following way, assuming a container is already running and has had the ray head node
started (here namedrayhead; see above):docker exec -it rayhead ray job submit -- python /path/to/script.py -o outdirNote that as this is run within the container environment, the paths are what is exposed within and not necessarily the
same as in the host environmentThe progress and log files can be viewed on the ray dashboard, and generated data will be available as a parquet table
written to/outdir(one file per batch, as they are completed)Installation, Build, & Run instructionsAn "official" docker image is provided based on the latest release, but for development or those needing a custom build
or running outside of a containerized environment, below are instructions for installing the code from the source
repository.SetupInstall dependencies:pip install -r requirements/dev.txtTestsRun tox:tox -e test, to run just the teststox, to run linting, tests and build. This should be run without errors prior to commitDockerFrom the repo directory, rundocker build -f docker/Dockerfile --tag dplutils .