Package: borealis-fireworks
BorealisRunsFireWorks workflowsonGoogle Cloud Platform computing services(GCP).See the repoBorealisand the
PyPI pageborealis-fireworks.What is it?FireWorksis an open-source
workflow management system, that is, it drives interdependent computing tasks.
Among the many workflow systems available,
FireWorks is exceptionally straightforward, lightweight, and
adaptable. It's well tested and supported. The only shared services it needs are
a MongoDB server (acting as the workflow "LaunchPad") and a file store.Borealislets you spin up as many temporary worker machines as you want in
theGoogle Cloud Platformto run your
workflow. That means pay-per-use and no contention between workflows.How does Borealis support workflows on Google Cloud Platform?TL;DR:Spin up worker machines when you need them, deploy your task code to
the workers in Docker Images, and store the data in Google Cloud Storage instead
of NFS.Figure 1. Blue hexagons represent Google Cloud Platform services.
The LaunchPad is shown in a blue hexagon because it's implemented as a
MongoDB server that you can run on Compute Engine.Worker VMs:As acloud computingplatform,Google Compute
Engine(GCE) has a vast number of machines
available. You can spin up lots of GCE "instances" (also called Virtual Machines
or VMs) to run your workflow, change your code, re-run some tasks, then let the
workers time out and shut down. Google will charge you based on usage and
there's no resource contention with your teammates.Borealis provides theComputeEngineclass and its command line wrappergceto create, tweak, and delete groups of worker VMs.Borealis provides thefireworkerPython script to run as the top level program
of each worker. It calls FireWorks'rlaunchfeature.You can run these Fireworkers on and off GCE as long as they can connect to
your MongoDB server and to the data store for their input and output files.Docker:You need to deploy your payload task code to those GCE VMs. It
might be Python source code and its runtime environment, e.g. Python 3.8,
Python pip packages, Linux apt packages, compiled
Cython code, data files, and environment variable settings. A GCE VM starts up
from aGCE Disk Imagewhichcouldhave all that preinstalled (with or
without the Python source code) but it'd be hard to keep it up to date and
hard to keep track of how to reproduce it.This is what Docker Images are designed for. You maintain aDockerfilecontaining
instructions to build the Docker Image, then use theGoogle Cloud Buildservice to build the Image and store it in theGoogle Container Registry.Borealis provides theDockerTaskFiretask to run a task in Docker. It
pulls a named Docker Image, starts up a Docker Container, runs a given shell
command in that Container, and shuts down the Container. Running in a Container
also isolates the task's runtime environment and side effects from the
Fireworker and other tasks.Google Cloud Storage:Although youcanset up an NFS shared file service
for the workers' files,Google Cloud Storage(GCS) is the native storage
service. GCS costs literally 1/10th as much as NFS service and it scales up
better. GCS lets you archive your files in yet lower cost tiers intended for
infrequent access. Pretty much all of Google's cloud services revolve around GCS,
e.g., Pub/Sub can trigger an action on a particular upload to GCS.But Cloud Storage is not a file system. It's anobject storewith a light
weight protocol to fetch/store/list whole files, called "blobs." It does not
support simultaneous writers. Instead, the last "store" of a blob wins. Blob
pathnames can contain/characters but GCS doesn't have actual directory objects,
so e.g. there's no way to atomically rename a directory.DockerTasksupports Cloud Storage by fetching the task's input files from GCS
and storing its output files to GCS.Logging:DockerTasklogs the Container's stdout and stderr, andfireworkersets up Python logging to write to Google'sStackDrivercloud logging service so you can watch your workers running
in real time.Projects:With Google Cloud Platform, you set up aprojectfor your team
to use. All services, VMs, data, and access controls are scoped by the project.How to run a workflowAfter doing one-time setup, the steps to run a workflow are:Build a Docker container Image containing your payload tasks to run in
the workflow. Thegcloud builds submitcommand will upload your code and aDockerfile, then trigger a Google Cloud Build server server to build the
Docker Image and store it in the Google Container Registry.SeeBuilding Your Docker Imagefor how to build
the Docker Image, starting with writing theDockerfile.Build your workflow and upload it to MongoDB.
You can do this manually by writing a.yamlfile and running thelpadcommand line tool to upload it, or automate it by implementing aworkflow builderprogram which calls FireWorks APIs to construct and upload aWorkflowobject.The workflow will run instances of theDockerTaskFiretask. Of course
it can run other Firetasks as well; they just won't get deployment,
storage, logging, and timeout features fromDockerTask.If you need to open a secure ssh tunnel to the MongoDB server running in
a Google Compute Engine VM, use theborealis/setup/example_mongo_ssh.shshell script.Start one or morefireworkerprocesses to run the workflow tasks.There are three ways to do this:Run thefireworkerprocess locally (this is handy for
debugging) by running thefireworkerPython script.Launch a group of fireworkers on Compute Engine VMs (this is handy for
getting a lot of work done in parallel) by running the Python scriptgce.Automate the launching of a group of fireworkers on GCE by making your
workflow builder code call on the BorealisComputeEngineclass after it
builds and uploads the workflow.While it runs and afterwards, you can:watch themain GCP dashboardTip:Add VM load charts to this page.watch theCompute Engine VM Instancesconsole to see your list of workerswatch theGoogle Cloud Logs Explorerof your running operations.(Set a Logs query likeresource.labels.instance_id:"fireworker" severity>=INFOto
see just the big picture of Fireworkers and Firetasks starting and stopping.
The:means "starts with."Expand the log level toseverity>=DEBUGto include task stdout and stderr lines.If coworkers are also running workflows, you can filter to just your own
Fireworkers using a query likeresource.labels.instance_id:"fireworker-USER-",
inserting your $USER name.)open the Fireworks Web GUIlpadwebguiBug:If thelpad webguicommand prints an errorTypeError: cannot pickle '_thread.lock' objectthat means you hit a compatibility problem with FireWorks on Python 3.Fix:Upgrade tofireworks>=1.9.7.Workaround:If you can't update fireworks, then runlpadwebgui-swhich starts the web server without the web browser, and manually
open a browser to the web server URL that it prints,
such ashttp://127.0.0.1:5000.
(In iTerm, just CMD-click the URL.)query your Fireworks tasks and Workflowslpadget_fws
lpadget_wflows
lpadreportrerun Fireworks tasks or Workflowslpadrerun_fws-i<FW_IDS>
lpadrerun_fws-sFIZZLEDpause/restart Fireworks tasks or Workflowslpadpause_fws-i<FW_IDS>
lpadresume_fws-i<FW_IDS>
lpaddefuse_fws-i<FW_IDS>
lpadreignite_fws-i<FW_IDS>

lpadpause_wflows-i<FW_IDS>
lpaddefuse_wflows-i<FW_IDS>
lpadrerun_fws-i<FW_IDS>
lpadreignite_wflows-i<FW_IDS>build a new Docker Image containing bug-fixes, then re-run Fireworks tasks.If all the fireworkers have exited, run thegcecommand to launch new
fireworkers.launch additional fireworkers:gce--base<BASE>--count<COUNT><NAME-PREFIX>Set the BASE number argument or NAME-PREFIX argument so the new VM names won't
conflict with your already-running VMs.Archive or delete a workflowor reset the entire Launchpad database:lpadarchive_wflows-i<FW_IDS>

lpaddelete_wflows-i<FW_IDS>

lpadresetAccess the output files in GCS via thegsutilcommand line tool, thegcsfusefile mounting tool, and theStorage Browserin theGoogle Cloud Platform web console.Tip:Usegsutil -mto transfer multiple files in parallel.
(If it printsTypeError: cannot pickle...,
configuregcloudandgsutilto use Python 3.7 per the note inInstall the development tools.)More detail on the Borealis componentsThe Borealis components can be used separately. Together they enable running
FireWorks workflows in Google Cloud.gce:TheComputeEngineclass and its command line wrappergcecreate, tweak, and delete a group of worker VMs.After you generate a workflow, call FireWorks'LaunchPad.add_wf()(or run FireWorks'lpad addcommand line tool) to upload it to the
LaunchPad. Then callComputeEngine.create()(or thegcecommand line)
to spin up a group of worker VMs to run that workflow and pass in the
parameters such as the LaunchPad db name and username
(via GCE metadata fields).ComputeEngineandgcecan also immediately delete a group of worker
VMs or ask them to quit cleanly between Firetasks, although we usually let
fireworkers shut down on their own after an idle timeout.ComputeEngineandgcecan also set GCE metadata fields on a group of
workers. This is used to implement the--quit-soonfeature.After installing theborealis-fireworkspip:$gce-h
usage:gce[-h][--delete|--set-metadata|--quit-soon][-d][-bBASE][-cCOUNT][-fFAMILY][-lLAUNCHPAD_FILENAME][-mMETADATA_KEY=VALUE]...[-oOPTION_KEY=VALUE]...NAME-PREFIX

Create,delete,orsetmetadataonagroupofGoogleComputeEngineVMs,e.g.
workflowworkersthatstartupfromadiskimage-family.(Thiscodealsohas
anAPIforprogramstocall.)...Or to run it within theborealis-fireworkssource code repository:$python-mborealis.gce-hfireworker:ThefireworkerPython script runs as as the top level program of each worker
node.fireworkerreads the worker launch parameters and calls the FireWorks library
to "rapidfire" launch your FireWorks "rockets." It also handles server shutdown.fireworkerconnects Python logging to Google Cloud's
StackDriver logging so you can watch all your worker machines in real time.To runfireworkeron GCE VMs, you'll need to create a GCE Disk Image that
contains Python, the borealis-fireworks pip, and such. See the instructions inhow-to-install-gce-server.txt.Thefireworkercommand can also run on your local computer for easier
debugging. For that, you'll need to install theborealis-fireworkspip and set
up your computer to access the right Google Cloud Project.$fireworker-h
usage:fireworker.py[-h][-lLAUNCHPAD_FILENAME][-s][--development]RunasaFireWorksworkernode,launchingrocketsrapidfire.Getsconfiguration
settingsfromGoogleComputeEngine(GCE)metadataattributeswhenrunningon
GCE,elsefromtheLaunchpadfile(seethe`-l`option),withfallbacks.The
setupsourcefilesare"/Users/jerry/dev/borealis/borealis/setup/*"...DockerTask:TheDockerTaskFiretask pulls a named Docker Image, starts up a Docker
Container, runs a given shell command in that Container, and stops the container.Required params:tasknamefor loggingDockerimageto load up into a Containershellcommandto run in the Containerinternal_prefixinput/output base pathname in the Containerstorage_prefixinput/output base pathname in GCSOptional params:inputsinput files and directory treesoutputsoutput files and directory treestimeoutfor stopping the shell command, in secondsDocker always runs a shell command in the Container. If you want to run aFiretaskin the Container, include a little Python script to bridge the gap:
Take a Firetask name and a JSON dictionary as command line arguments,
instantiate the Firetask with those arguments, and call the Firetask'srun_task()method.DockerTasksupports Google Cloud Storage (GCS) by fetching the task's input
files from GCS, mapping them into the Docker Container, running the task, and
storing its output files to GCS. This means you'll need to specify the input
and output paths asDockerTaskarguments.
(Your workflow builder code could use this path information to compute the
task-to-task interdependencies for FireWorks.)Each path you specify in DockerTask'sinputsandoutputsdenotes a directory tree of files iff it ends with a/.When storing task output files,DockerTaskcreates entries with names ending in/to act asdirectory placeholdersthat speed up tree-oriented traversal.
This means you can rungcsfusewithout using the--implicit-dirsflag, resulting in mounted directories that run 10x faster.DockerTaskimposes a given timeout on the task running in the Docker
container.DockerTasklogs the Container's stdout and stderr to a file and to Python
logging (whichfireworkerconnects toStackDriver).What's next?See theTeam Setupsteps and
theDeveloper Setupsteps.SeeBuilding Your Docker Image.SeeChange Log (aka Release Notes).Also seeHandy Links.