Package: custom-image-builder
Building a singular container for HPC using globus-computeContextOne of the execution configurations ofglobus-computerequires a registered container which is spun up to execute the user function on the HPC.HPCs do not run docker containers(due to security reasons as discussedhere) and support only an apptainer/singularity image.Installing the apptainer setup to build the singularity image locally is not a straightforward process especially on windows and mac systems as discussed in thedocumentation.Using this python library the user can specify their custom image specification to build an apptainer/singularity image
which would be used to in-turn to run their functions on globus-compute. The library registers the container and
returns the container id which would be used by the globus-compute executor to execute the user function.Prerequisite.Aglobus-compute-endpointsetup on HPC cluster.The following steps can be used to create an endpoint on the NCSA Delta Cluster, you can modify the configurations based on your use-case:Note.For the following to work we must use the globus-compute-sdk version of 2.2.0 while setting up our endpoint.
It is recommended to use python3.9 for setting up the endpoint and as the clientCreate a conda virtual env. We have created acustom-image-builderconda env on the delta cluster as follows:condacreate--namecustom-image-builder-py-3.9python=3.9

condaactivatecustom-image-builder

pipinstallglobus-compute-endpoint==2.2.0Creating a globus-compute endpoint:globus-compute-endpointconfigurecustom-image-builderUpdate the endpoint config at~/.globus_compute/custom-image-builder/config.pyto :fromparsl.addressesimportaddress_by_interfacefromparsl.launchersimportSrunLauncherfromparsl.providersimportSlurmProviderfromglobus_compute_endpoint.endpoint.utils.configimportConfigfromglobus_compute_endpoint.executorsimportHighThroughputExecutoruser_opts={'delta':{'worker_init':'conda activate custom-image-builder-py-3.9','scheduler_options':'#SBATCH --account=bbmi-delta-cpu',}}config=Config(executors=[HighThroughputExecutor(max_workers_per_node=10,address=address_by_interface('hsn0'),scheduler_mode='soft',worker_mode='singularity_reuse',container_type='singularity',container_cmd_options="",provider=SlurmProvider(partition='cpu',launcher=SrunLauncher(),# string to prepend to #SBATCH blocks in the submit# script to the scheduler eg: '#SBATCH --constraint=knl,quad,cache'scheduler_options=user_opts['delta']['scheduler_options'],worker_init=user_opts['delta']['worker_init'],# Command to be run before starting a worker, such as:# 'module load Anaconda; source activate parsl_env'.# Scale between 0-1 blocks with 2 nodes per blocknodes_per_block=1,init_blocks=0,min_blocks=0,max_blocks=1,# Hold blocks for 30 minuteswalltime='00:30:00'),)],)Start the endpoint and store the endpoint id to be used in the following exampleglobus-compute-endpointstartcustom-image-builderExampleConsider the following use-case where the user wants to execute a pandas operation on HPC using globus-compute.
They need a singularity image which would be used by the globus-compute executor. The library can be leveraged as follows:Locally you need to install the following packages, you can create a virtual env as follows:cdexample/

python3.9-mvenvvenvsourcevenv/bin/activate

pipinstallglobus-compute-sdk==2.2.0

pipinstallcustom-image-builderfromcustom_image_builderimportbuild_and_register_containerfromglobus_compute_sdkimportClient,Executordeftransform():importpandasaspddata={'Column1':[1,2,3],'Column2':[4,5,6]}df=pd.DataFrame(data)return"Successfully created df"defmain():image_builder_endpoint="bc106b18-c8b2-45a3-aaf0-75eebc2bef80"gcc_client=Client()container_id=build_and_register_container(gcc_client=gcc_client,endpoint_id=image_builder_endpoint,image_file_name="my-pandas-image",base_image_type="docker",base_image="python:3.8",pip_packages=["pandas"])print("The container id is",container_id)withExecutor(endpoint_id=image_builder_endpoint,container_id=container_id)asex:fut=ex.submit(transform)print(fut.result())Note.The singularity image require globus-compute-endpoint as one of its packages in-order to run the workers as our custom
singularity container, hence by default we require python as part of the image inorder to install globus-compute-endpoint.