Package: covalent-hpc-plugin
Covalent HPC PluginCovalentis a Pythonic workflow tool used to execute tasks on advanced computing hardware. This executor plugin usesPSI/Jto allow Covalent to seamlessly interface with a variety of common high-performance computing job schedulers and pilot systems (e.g.Slurm,PBS,LSF,Flux,Cobalt,RADICAL-Pilot). For workflows to be deployable, users must have SSH access to the login node, access to the job scheduler, and write access to the remote filesystem.InstallationServer EnvironmentTo use this plugin with Covalent, simply install it usingpipin whatever Python environment you use to run the Covalent server (your local machine by default):pip install covalent-hpc-pluginRun the following in Python to have Covalent automatically register the plugin:importcovalentHPC EnvironmentAdditionally, on the remote machine(s) where you plan to execute Covalent workflows with this plugin, ensure that the remote Python environment has Covalent and PSI/J installed:pip install covalent psij-pythonNote that the Python major and minor version numbers on both the local and remote machines must match to ensure reliable (un)pickling of the various objects.UsageDefault Configuration ParametersBy default, when you install thecovalent-hpc-pluginand runimport covalentfor the first time, your Covalentconfiguration file(found at~/.config/covalent/covalent.confby default) will automatically be updated to include the following sections. These are not all of the available parameters but are simply the default values.[executors.hpc]
address = ""
username = ""
ssh_key_file = "~/.ssh/id_rsa"
instance = "slurm"
launcher = "single"
inherit_environment = true
pre_launch_cmds = []
post_launch_cmds = []
shebang = "#!/bin/bash"
remote_python_exe = "python"
remote_workdir = "~/covalent-workdir"
create_unique_workdir = false
cache_dir = "~/.cache/covalent"
poll_freq = 60

[executors.hpc.environment]

[executors.hpc.resource_spec_kwargs]
node_count = 1
processes_per_node = 1
gpu_cores_per_process = 0

[executors.hpc.job_attributes_kwargs]
duration = 10You can modify various parameters in the Covalent config file as-needed to better suit your needs, such as theaddressof the remote machine, theusernameto use when logging in, thessh_key_fileto use for authentication, the type of job scheduler (instance), and much more. Note that PSI/J is a common interface to many common job schedulers, so you only need to toggle theinstanceto switch between job schedulers.A full description of the various input parameters are described in the docstrings of theHPCExecutorclass, reproduced below:https://github.com/Quantum-Accelerators/covalent-hpc-plugin/blob/25785d0c546851c4b11e5c227f2e7aebb12aba8c/covalent_hpc_plugin/hpc.py#L115-L159Defining Resource Specifications and Job AttributesTwo of the most important sets of parameters areresource_spec_kwargsandjob_attributes_kwargs, which specify the resources required for the job (e.g. number of nodes, number of processes per node, etc.) and the job attributes (e.g. duration, queue name, etc.), respectively.resource_spec_kwargsis a dictionary of keyword arguments passed to PSI/J'sResourceSpecV1classjob_attributes_kwargsis a dictionary of keyword arguments passed to PSI/J'sJobAttributesclass.The allowed types are listedhere.Using the Plugin in a Workflow: Approach 1With the configuration file appropriately set up, one can run a workflow on the HPC machine as follows:importcovalentasct@ct.electron(executor="HPCExecutor")defadd(a,b):returna+b@ct.latticedefworkflow(a,b):returnadd(a,b)dispatch_id=ct.dispatch(workflow)(1,2)result=ct.get_result(dispatch_id)Using the Plugin in a Workflow: Approach 2If you wish to modify the various parameters within your Python script rather than solely relying on the the Covalent configuration file, it is possible to do that as well by instantiating a custom instance of theHPCExecutorclass. An example with some commonly used parameters is shown below. By default, any parameters not specified in theHPCExecutorwill be inherited from the configuration file.importcovalentasctexecutor=ct.executor.HPCExecutor(address="coolmachine.university.edu",username="UserName",ssh_key_file="~/.ssh/id_rsa",instance="slurm",remote_conda_env="myenv",environment={"HELLO":"WORLD"},resource_spec_kwargs={"node_count":2,"processes_per_node":24},job_attributes_kwargs={"duration":30,# minutes"queue_name":"debug","project_name":"AccountName",},launcher="single",remote_workdir="~/covalent-workdir",)@ct.electron(executor=executor)defadd(a,b):returna+b@ct.latticedefworkflow(a,b):returnadd(a,b)dispatch_id=ct.dispatch(workflow)(1,2)result=ct.get_result(dispatch_id)Working Example: PerlmutterThe following is a minimal working example to submit a Covalent job on NERSC'sPerlmuttermachine. It assumes that you have used thesshproxyutility to generate a certificate file in order to circumvent the need for multi-factor authentication for each login.importcovalentasctexecutor=ct.executor.HPCExecutor(address="perlmutter-p1.nersc.gov",username="UserName",ssh_key_file="~/.ssh/nersc",cert_file="~/.ssh/nersc-cert.pub",remote_conda_env="myenv",job_attributes_kwargs={"project_name":"ProjectName","custom_attributes":{"slurm.constraint":"cpu","slurm.qos":"debug"},},)@ct.electron(executor=executor)defadd(a,b):returna+b@ct.latticedefworkflow(a,b):returnadd(a,b)dispatch_id=ct.dispatch(workflow)(1,2)result=ct.get_result(dispatch_id)TroubleshootingThe most common cause of issues is related to the job scheduler details (i.e. theresource_spec_kwargsand thejob_attributes_kwargs). If your job fails on the remote machine, setcleanup=Falseand then check the files left behind in the working directory as well as the~/.psijdirectory for a history and various log files associated with your attempted job submissions.Release NotesRelease notes are available in theChangelog.CreditThis plugin was developed byAndrew S. Rosen, building off of prior work by the Agnostiq team on thecovalent-slurm-plugin.If you use this plugin, be sure to cite Covalent as follows:W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand.Covalent.Zenodo, 2022.https://doi.org/10.5281/zenodo.5903364LicenseCovalent is licensed under the Apache 2.0 License. Covalent may be distributed under other licenses upon request. See theLICENSEfile or contact thesupport teamfor more details.