Package: bigchem
BigChemA distributed system for scaling and parallelizing quantum chemistry calculations.⚠️ A Note About x86 and ARM ArchitecturesMost Quantum Chemistry packages (including those used by default in BigChem's worker--psi4, xtb, and rdkit) are only compiled and released for the x86 architecture, not the ARM architecture. This meansBigChem's worker will not work or build on ARM machines like Apple's M1 chip. If you want to run Quantum Chemistry programs on your ARM machine, please reach out to your favorite QC developer and ask for distributions compiled for ARM. When they exist, I'll add them to BigChem's ARM builds.If you'd like to play with BigChem without executing QC programs on your ARM machine, comment out theworkerin thedocker-compose.yamlfile, then run the following commands to run a local version of a BigChem worker that can execute theaddandcsumTasksto explore how BigChem works. Note Docker has updated thedocker-composecommand to be a subcommand of the Docker CLIdocker compose(no-). If you are running an older version of Docker Desktop you may still need to use thedocker-composecommand instead.dockercomposeup-d--build
poetryinstall
poetryruncelery-Abigchem.tasksworker--without-heartbeat--without-mingle--without-gossip--loglevel=INFO🐰 Quickstart✅ Make sure you are on an x86 machine (not an ARM machine like Apple's M chip series)✅ Installpoetryto install BigChem💻 Run BigChem on a Single Node (like your laptop)Install project dependenciespoetryinstallCheck that your installation is working correctly by running the tests. (Requiresdocker compose).bashscripts/test.shYou can review test coverage in the now-generatedhtmlcovfolder; openindex.htmlin a browser.Run the following commands to start a broker, backend, and worker.dockercomposeup-d--buildYou can stop these services by running:dockercomposedownWith BigChem running (docker compose up -d --build) execute scripts in theexamplesdirectory to see how to perform computations using BigChem. Add the-iflag when running python to drop into an interactive terminal after a script executes to interact with the returned objects.poetryrunpython-iexamples/example.pyIf you would like to increase the number of BigChem worker processes running, uncomment and edit thebigchem_worker_concurrencyvalue in thedocker-compose.yamlfile.💻💻💻 Run BigChem on Multiple Nodes (Swarm Mode)You do not need to install BigChem or any Quantum Chemistry package on each node. You only need to haveDockerinstalled on each node. We will useDocker Swarmto distribute BigChem to each node. We assume the nodes are networked together and reachable viaTCP port 2377for cluster management communications,TCP and UDP port 7946for communication among nodes, andUDP port 4789for overlay network traffic.Execute the following commands on the node you wish to be your "Manager" node, the node from which you will control the system.Start a SwarmdockerswarminitCollect the join command for adding worker nodes to this swarmdockerswarmjoin-tokenworkerRun the printed command on all the nodes you wish to join your swarm and then return to the Manager node.dockerswarmjoin--tokensuper-secret-token-xxxip-addr-of-manager:2377From your manager node deploy BigChem. This will start a broker and backend running on the manager node and one BigChem worker on each node.dockerstackdeploy-cdocker-compose.yaml--prunebigchemBigChem will now be running on all nodes. It may take a few moments for the services to start since the worker image is large and will need to be downloaded and decompressed on all nodes.You can increase the number of BigChem worker processes running on each node by uncommenting and editing thebigchem_worker_concurrencyvalue in thedocker-compose.yamlfile and then runningdocker stack deploy ...again.Send work to BigChem by running any of the scripts in theexampledirectory. Tasks will be distributed across all worker nodes.poetryrunpython-iexamples/energy.pyNOTE: If the example scripts are hanging without printing out any status or completing it may be alocalhostnetworking issue. Depending on your system configurationlocalhostmay be resolving to an ipv4 or ipv6 address. If you deployed your swarm using an ipv4 address yetlocalhostrefers to anipv6address on your machine, you won't be able to talk to the broker/backend atlocalhost. Usually just the backend has issues. Try the following to explicitly connect to the backend at127.0.0.1. Add thebigchem_broker_url=amqp://127.0.0.1as well, if needed.bigchem_backend_url=redis://127.0.0.1/0pythonexamples/energy.pyShutdown BigChemdockerstackrmbigchem💻 + 💪 Run BigChem on a Single Node with TeraChem (GPU Support)Same as above but use the following instead ofdocker compose up -d --builddockercompose-fdocker-compose.yaml-fdocker/terachem.yamlup-d--buildThis will run TeraChem in unlicensed mode. If you'd like to use a license uncomment the${TERACHEM_LICENSE_PATH}:/terachem/license.keyline indocker/terachem.yamland addTERACHEM_LICENSE_PATH=/path/to/your/license.keyto a.envfile in the root of the project using the path on your local machine containing the license.Send work to TeraChem by modifying any of theexamplesscripts to useterachem_feinstead ofpsi4.Run theexamples/terachem.pyscript to see how to request files back from TeraChempoetryrunpython-iexamples/terachem.py💻💻💻 + 💪💪💪 Run BigChem on Multiple Nodes with TeraChem (Swarm Mode)GPU support for Docker Swarm can be implemented multiple ways. We will demonstrated the simplest way here. For more details seedocs/swarm-gpus.md. First follow the instructions above to initialize a swarm.Follow the instructionshereto installnvidia-container-runtimeon each node.Update (or create)/etc/docker/daemon.jsonon each node to usenvidiaas the default runtime with the following:{"default-runtime":"nvidia","runtimes":{"nvidia":{"path":"nvidia-container-runtime","runtimeArgs":[]}}}Restart docker on each nodesudo service docker restartRun the following command to deploy BigChem and TeraChem to all nodes.Note the second docker-compose file has a different name than the single-node file,terachem.SWARM.yaml.dockerstackdeploy-cdocker-compose.yaml-cdocker/terachem.swarm.yaml--prunebigchemBigChem and TeraChem will now be running on all nodes. It may take a few moments for the services to start since both images are large and will need to be downloaded and decompressed on all nodes.Getting into the DetailsOverviewBigChem is built onCeleryand uses a producer/consumer model for delivering and completing work tasks. You may refer to the Celery documentation for a deeper dive on technical details.Why is BigChem useful? Many computational chemistry tasks require large, distributed resources due to the scale or quantity of computations. Programming highly parallel or distributed computing tasks is challenging and requires specialized skills. BigChem makes it easy to take advantage of modern, distributed system compute paradigms without requiring expert-level understanding of distributed systems or parallelization. End users write very simple, single-threaded code, yet BigChem will efficiently distribute calculations across hundreds or thousands of compute nodes and deliver results back as if the calculations were performed on the local machine. BigChem is easy to use, easy to program, and easy to deploy across a few or thousands of nodes.BigChem has only a few key ideas that need to be mastered in order to execute distributed tasks or further develop the system to execute new tasks.A basic system overview helps demonstrate how BigChem works. End users can import BigChem'sTasksorAlgorithmsinto their code as generic Python functions. Calling these functions using.delay()or.apply_async()and passing the relevant arguments, e.g.,add.delay(2,2), tells BigChem to place the work on thebroker(messaging queue). Thebrokerdelivers theTaskto an available worker which executes theTask, confirms successful execution to thebroker, and places the result in thebackend. End users get results from thebackendby calling.get()on the future object returned from calling.delay()or.apply_async(). To end users they appear to be writing single-threaded Python code, but behind the scenes an array of BigChem workers can be executing their compute tasks at any scale on distributed resources anywhere in the world.Only onebrokerandbackendare ever running at a time. The system scales by increasing the number of workers connected to the system. Usually one worker runs on a given compute node.frombigchem.tasksimportaddfuture_result=add.delay(2,2)# sends task to broker, returns `AsyncResult`# Workers will now receive the task, execute it, and write the result to the backendresult=future_result.get()# collects task's return value from the backendresult.forget()# removes the result from the backend to free up resources for future calculationsBigChem's core object is aTask. ATaskis simply a Python function wrapped with the@bigchem.taskdecorator.Taskscan be any computing objective written in any language, calling any subprocess, or leveraging any executable. Tasks that cannot be defined as pure Python code or are written in other languages can be called using Python'ssubprocess module. Tasks can be found in thebigchem/tasks.pymodule. Once aTaskis registered with BigChem, BigChem will automatically know how to execute it remotely on worker machines and distribute it across all available resources.Taskscan also be combined into more complex computational networks to createAlgorithms.Algorithmsmake use ofTasksplus thegroup,chain, orchordprimitive to define computational tasks comprised of manyTasksexecuted simultaneously or in sequence. EachTaskin theAlgorithmwill be executed on different distributed workers in parallel. BigChem will orchestrate the execution ofTasksand collection of results across distributed resources.Algorithmscan be found inbigchem/algos.pyor created on the fly using existing tasks. Below we create anAlgorithmusing BigChem'saddandcsumTasks. The function returns aSignature, an object that contains the network of tasks to be executed and their arguments. TheSignaturecan be executed on BigChem workers by calling.delay()or.apply_async()on theSignature.frombigchem.tasksimportadd,csumfrombigchem.canvasimportgroup,Signaturedefdistributed_sum(max:int)->Signature:"""Performs the addition each integer from 0 to max with itself in parallel ondisparate worker nodes, then sums up all those values into a final number.The group tells BigChem to execute all tasks in the group in parallel.The | symbol creates a chord and BigChem will pass all the distributed resultsfrom the group as an argument to csum which will sum up the results.Each compute task in the group and the csum task may execute on differentnodes. BigChem will orchestrate the efficient distribution of tasks across nodesand coordinate collection of results from each node, without end users ordevelopers needing to consider these complex actions."""returngroup(add.s(i,i)foriinrange(max))|csum.s()# Note that we are passing arguments directly to distributed_sum and then calling# .delay() rather than passing arguments to .delay() as when calling a basic Taskfuture_result=distributed_sum(10).delay()result=future_result.get()future_result.forget()# to remove results from the backendIn summary, BigChem uses abrokerto distributeTasksto an arbitrary number of workers located anywhere in the world. Workers executeTasksremotely and write their results to thebackend. Users retrieve results from the backend by calling.get()onAsyncResultfutures objects.WorkersA worker process knows aboutTasksregistered with BigChem, where to get work (thebroker), and where to write results (thebackend). A worker must have BigChem installed to know about registeredTasksand any other software called byTasksinstalled (such as external Quantum Chemistry packages likepsi4orTeraChem).Generally, you run a single worker on a given compute node. Unless you are using GPUs or other accelerators or have an unusually complicated deployment it probably doesn't make sense to run more than one worker per node. The worker itself can scale the number of subprocesses that it uses to process tasks by setting thebigchem_worker_concurrencyenvironment variable. So one worker can have multiple independent subprocesses acceptingTasks.How do you know how many BigChem worker processes to run on each node? Generally, this depends on the nature of theTasksyou are executing. If you are executing code that makes efficient use of all CPU cores on a node, then havingbigchem_worker_concurrency=1(the default value) is appropriate as theTaskwill be making efficient use of the node. If you are executingTasksthat are single-threaded (use only 1 CPU core at a time) then you should setbigchem_worker_concurrencyto the number of cores on the machine. Settingbigchem_worker_concurrency=0will tell BigChem to automatically set the concurrency to the number of cores on the machine. Optimal performance tuning is idiosyncratic to the underlying code, so testing is key if you want to get maximum scaling performance from your code at various levels of concurrency. Scaling the number of nodes running workers will produce linear performance gains for the system.Local DevelopmentThedocker-compose.yamlfile creates a local BigChem system with a single worker for executing tasks.dockercomposeup-d--buildThis command starts up the three BigChem components: abroker, abackendand aworker. Thebrokeris a RabbitMQ instance with an admin dashboard available athttp://localhost:15672using the username and passwordguestandguest. ThebrokerreceivesTasksand by default places them on theceleryqueue for consumption by workers. Thebackendis aredisinstance. Theworkerwill get built from thedocker/worker.dockerfilefile and will containBigChem,psi4,rdkit, andxtb. Theworkerwill consume tasks from thebrokerand write results to thebackend.Services can be monitored in real time using thecontainer_namespecified in thedocker-compose.yamlfile and thedocker logscommand.dockerlogs{container_name}-fIf newTasksare added to BigChem or existing tasks are modified, the worker must be restarted to pickup the new or modifiedTasks. You can restart the worker by either rebuilding the container (docker compose up -d --build) or by just restarting the worker since the code is mounted into the container directly (docker restart bigchem_worker), generally restarting the worker should be faster.If you want to scale your worker subprocesses locally, uncomment and adjust thebigchem_worker_concurrencyvariable to your desired number or to0to automatically scale to the number of CPU cores on your machine.If you want to add additional Quantum Chemistry programs (or any program!) to the worker, modify thedocker/worker.dockerfileto install the program and then rebuild the image by runningdocker compose up -d --build.If you have a GPU on your machine and want to run TeraChem as part of BigChem, includedocker/terachem.yamlwhen you start BigChem. See theTeraChem/GPU Section Aboveabove for more details.dockercompose-fdocker-compose.yaml-fdocker/terachem.yamlup-d--buildDeploymentDeploying BigChem to a production environment consists of running abrokerand abackendthat are reachable byworkersand then as many workers as you'd like. Workers will connect to thebrokerandbackendand BigChem will automatically coordinate efficient distribution ofTasksacross all workers and the synchronization of more sophisticatedAlgorithmsthat need to pass results between workers. Deployment can be done with or without Docker, depending on the access you have to the underlying hardware. We strongly recommend deploying with Docker if you are able. The Multi-Node (Swarm) examples in the🐰 Quickstartare both examples of production deployments.All configurable environment variables for BigChem can be found inbigchem/config.py. If environment variables are set with the same names found as attributes on theSettingsobject, they will be automatically picked up and used by BigChem. Examples of this can be found in thedocker-compose.yamlfile where various settings are modified via environment variables. Variables can belower_caseor allUPPER_CASE.Broker and BackendThebrokerandbackendneed to be deployed somewhere thatworkerscan connect to via a TCP connection. This means they can be deployed on a local cluster where workers will run, or on a remote server (like a cloud server) that has a public IP address and is reachable from anywhere. If you are deploying yourbrokerandbackendon a cloud service and connecting to them over the open internet we strongly recommend you useTLSconnections to secure communication between yourworkers,broker, andbackend. For an example of how to deploy abrokerandbackendsecurely behind aTraefikreverse proxy that handlesTLSfor you, seedocker/web.yamland follow the instructions below. The configuration for running a Traefik reverse proxy can be foundhere.If you want to secure your broker/backend with usernames/passwords create the following files and populate with their correct secrets and then reference them in yourdocker-compose.yamlfile as indocker/web.yaml.rabbit.envRABBITMQ_DEFAULT_USER=RABBITMQ_DEFAULT_PASS=redis.confrequirepass${your_redis_password}Docker SwarmDeploying BigChem workers across many nodes on a cluster is very simple usingDocker Swarm. The examples in theQuickstartcan get your started.If you need to pass secrets (like usernames/password for your broker/backend) to your services use aworker.envfile as described below. Omit# pragma: allowlist secretas that is just for linting purposes in the README.md.# NOTE: depends on a single worker.env file alongside this .yaml file containing two# environment variables; fill these with the URLs of the broker and backend.# BIGCHEM_BROKER_URL=amqps://admin123:supersecret987@rabbit.mydomain.com:5671 # pragma: allowlist secret# BIGCHEM_BACKEND_URL=rediss://:password123@redis.mydomain.com:6379/0?ssl_cert_reqs=CERT_NONE # pragma: allowlist secretversion:"3.8"services:worker:image:mtzgroup/bigchem-workerenv_file:worker.envenvironment:-bigchem_worker_concurrency=2volumes:# qcengine uses /tmp by default for scratch directory-scratch:/tmpdeploy:mode:globalvolumes:scratch:This definition depends on aworker.envfile living next to thedocker-compose.workers.yamlfile that defines the URLs at which workers can locate the BigChembrokerandbackend. If the addresses are local to your cluster and don't contain secrets (like passwords) then you can just place the addresses directly in thedocker-compose.workers.yamlfile as environment variables and delete theenv_fileline from the file. Theworker.envfile should look like this but be filled with the corresponding usernames, passwords, and hosts. If you are using insecure (non-encrypted) protocols over a local network changeamqpstoamqpand change the port from5671to5672(or remove it,5672is the default port for RabbitMQ) andredisstoredisand remove?ssl_cert_reqs=CERT_NONEto denote the use of insecure protocols (you may also remove the6379port specification). Thedocker-compose.yamlfile in the root shows an example of insecure URLs without usernames, passwords, or ports.worker.envBIGCHEM_BROKER_URL=amqps://${USERNAME}:${PASSWORD}@${YOUR_DOMAIN_DOT_COM}:5671BIGCHEM_BACKEND_URL=rediss://:${PASSWORD}@${YOUR_DOMAIN_DOT_COM}:6379/0?ssl_cert_reqs=CERT_NONENOTE:The usernames and passwords in theworker.envfile should match those set in therabbit.envandredis.conffiles.The amqpS/redisS protocols. If you are connecting over the open internet you should require TLS. Also note the use of the5671secureport foramqpsconnections. This is the port on whichtraefikis listening foramqpsconnections.We do not verify the SSL certificate forredis. This is becausetraefikis dynamically generating and renewing SSL certificates so we do not have a "permanent" certificate that we can place on the client and use for verification. Very low risk of man-in-the-middle attacks here.If your workers are not connecting over the open internet they can connect directly using insecure protocols to the broker/backend rather than proxying through thetraefikreverse proxy.If you have defined yourbrokerandbackendas services in the samedocker-compose.workers.yaml(like in thedocker-compose.yamlfile in the root of this project), then you use the name of the service as the hostname and docker will route TCP connections appropriately. See theenvironmentsection of theworkerin thedocker-compose.yamlfile as an example.Theglobaldeploy mode tells Docker Swarm to place one instance of this service on each node. Thevolumesspecification creates aDocker Volumeon each node to act as a scratch disk for calculations. And we setbigchem_worker_concurrency=2so that each node has two worker subprocesses consuming tasks.You can deploy workers across your swarm by running the following on a manager node:dockerstackdeploy-cdocker-compose.workers.yaml--prunebigchemYou can inspect the worker services by running:dockerstackservicesbigchemLogs can be followed by running:dockerservicelogsbigchem_worker-fWorkers can be removed by running:dockerstackrmbigchemFor more sophisticated deployments you can tune how many workers run on each node, which queues each worker listens to (thebrokercan host multiple queues and workers can listen to one, many, or all of them,Taskscan be sent to specific queues by end users). And you can tune specific environment variables, like which GPU to use, for each individual worker. To see an example of a more sophisticated deployment seedocker/xstream.yaml. This deployment runs workers in "replicated" mode (as opposed to global mode) to have more than one worker per node, makes use of constraints like the hostname of the node to determine worker placement, and uses dynamic environment variables so that each worker is assigned exactly one GPU to use for calculations on nodes that contain multiple GPUs. TheXStreamdeployment allows independent scaling of workers on each node using thedocker service scalecommand. Using only a few lines of configuration, one can deploy 88 workers across 88 GPUs on 6 different nodes with unique environment variables for each worker using a single command:docker stack deploy -c docker/xstream.yaml --prune bigchem. SeeSwarm GPUsfor more details on how to configure GPU support for Docker Swarm.If you want to build a custom worker image with additionalTasks,Algorithms, or Quantum Chemistry programs you can modifybigchem/tasks.pyand/orbigchem/algos.pyand build a new image running the following from the root directory. Note that algorithms do not need to be written in the BigChem repo or deployed to workers. They can be written in any script without needing to deploy it to BigChem since algorithms are composed ofTasksthat BigChem already knows about and orchestration primitives (group/chain/chord) that simply tell BigChem how to execute thoseTasks.dockerbuild-twhatever_you_name_your_image:0.1.0.`You can push this image to a freeDocker Hubaccount and replace theimage:line in thedocker-compose.workers.yamlfile above with the name of your image.If you want to add additional Quantum Chemistry programs you can modify theworker.dockerfilefile to install them into your image and then perform the same steps mentioned above to build, push, and deploy your workers to your swarm.Non-Docker DeploymentDocker makes it easy to deploy a few or hundreds of workers across distributed resources, manage package dependencies, and ensure that each node is running identical code. But Docker is not required to run BigChem workers. The only thing that is required is a BigChem install and any additional executables (like Quantum Chemistry programs) that theTasksuse.You can install BigChem on a node by running the following command. This command will install BigChem and all its required Python dependencies (but not external programs likepsi4orTeraChem):python-mpipinstallbigchemOr if you are wanting to run your own modified version of BigChem, place the source code on a node and from the root directory of this codebase run:python-mpipinstall.Workers can be started by running the following:celery-Abigchem.tasksworker--without-heartbeat--without-mingle--without-gossip--loglevel=INFONote this is the same command issued in theworker.dockerfile. Additional executables--like Quantum Chemistry packages used byTasks--must be installed separately on each node. You can learn more about controlling workers in theCelery documentation.Deploying non-Dockerized workers on each node consists of installing the BigChem repo on each node, and then starting the workers with thecelerycommand. That's it!