Package: cloudq
CloudQCloud storage-based meta schedulerCopyright 2022-2023 National Institute of Advanced Industrial Science and Technology (AIST), Japan and
Hitachi, Ltd.This program is licensed under the Apache License, Version2.0.OverviewCloudQ is a job management system that targets on executing AI/HPC tasks on on-premise systems, cloud and supercomputers with a unified interface.
It has the following features.Input, output and status of jobs are stored on an Amazon S3 compatible object storage.User can write a jobscript in two formats;local jobscriptandmeta jobscript.
A job described in meta jobscript can run on any systems managed under CloudQ.
A job described in local jobscript runs only on a specific system, but the job can use all functions the system provides.The installation and administration of CloudQ do not require administrator privileges.CloudQ consists of two components.One isAgentwhich submits jobs and manages them on a system.
If you have accounts on multiple clouds or supercomputers, you can submit jobs to them using CloudQ by installing Agents on them.Currently, CloudQ Agent supports the following systems.ABCIAmazon Web Services (AWS)The second CloudQ component isClient.
By using Client on a user's terminal, user can submit and manage jobs on CloudQ.CloudQ provides an additional component namedBuilder.
Builder helps setting up compute clusters on clouds.Currently, CloudQ Builder for AWS is available.
CloudQ Builder for AWS creates clusters using Slurm as their job scheduler.
It installs CloudQ Agent on cluster's head node.RequirementsCloudQ ClientOS: Linux, MacOS and WindowsPython: 3.8 or newer (Tested on Python 3.8.16)AWS CLI: 2.1.30 or newer (Tested on AWS CLI 2.9.16)CloudQ AgentOS: Linux compatible OS (Tested on CentOS 7.5 and Amazon Linux 2)Python: 3.8 or newer (Tested on Python 3.8.16)AWS CLI: 2.1.30 or newer (Tested on AWS CLI 2.9.16)CloudQ Builder for AWSOS: Linux, MacOS and WindowsPython: 3.8 or newer (Tested on Python 3.8.16)AWS CLI: 2.1.30 or newer (Tested on AWS CLI 2.9.16)AWS ParallelCluster: 3.0.3InstallationInstallation Guidedescribes how to install and configure CloudQ.The following documents describe how to setup CloudQ environment in a specific scenario.Use ABCIUse ABCI as the compute resource and ABCI Cloud Storage as the job storage.Use AWSUse AWS as the compute resource and the job storage.Use ABCI and AWS (coming soon)Use ABCI and AWS as the compute resources and ABCI Cloud Storage as the job storage.UsageAgentIn an environment where CloudQ Agent does not run automatically, you need to launch CloudQ Agent as follows.$cloudqd--daemonOn compute clusters built by Builder for AWS, CloudQ Agent automatically starts.ClientSubmit a JobThe following example submits a job described in local jobscript.$cloudqclisubmit--scriptcloudq/example/ljob_tf_mnist.abci.sh\--submit_toYOUR_SYSTEM--submit_opt'SUBMIT_OPTION'Job (3f7e7681) ljob_tf_mnist.abci.sh has been submitted.The following example submits a job described in meta jobscript.$cloudqclisubmit--scriptcloudq/example/mjob_tf_mnist.shJob (e210c27c) mjob_tf_mnist.sh has been submitted.Submit a Dependent JobThe following example submits a job that depends on other jobs.$cloudqclisubmit--scriptcloudq/example/ljob_tf_mnist.abci.sh\--submit_toYOUR_SYSTEM--submit_opt'SUBMIT_OPTION'\--hold_jid'3f7e7681,e210c27c'Job (fc2d6f45) ljob_tf_mnist.abci.sh has been submitted.Submit an Array JobThe following example submits an array job.$cloudqclisubmit--scriptcloudq/example/mjob_array.sh--array_tid1-4:1Job (a38c9a9f) mjob_array.sh has been submitted.In the meta jobscript, the environment variables can be used to refer to task ID and other information.
SeeEnvironment variablesCheck the Status of a Job$cloudqclistat--ide210c27cuuid                  e210c27cjobid                 5150599name                  mjob_tf_mnist.shjobscript_type        metahold_jidarray_tidsubmit_tosubmit_optstate                 DONEworkdir               YOUR_HOME/.cloudq/cache/e210c27crun_system            abcilocal_account         YOUR_ACCOUNTlocal_group           YOUR_GROUPsubmit_command        qsub -g YOUR_GROUP mjob_tf_mnist.shtime_submit           2021/01/13 09:55:34time_receive          2021/01/13 10:05:47time_ready            2021/01/13 10:05:47time_start            2021/01/13 10:06:16time_stageout_start   2021/01/13 10:06:33time_stageout_finish  2021/01/13 10:06:33time_finish           2021/01/13 10:06:33size_input            516size_output           1329error_msgsubmit_opt_local      -g YOUR_GROUPlocal_name            mjob_tf_mnist_local.shCancel a Job$cloudqclicancel--ide210c27cJob (e210c27c) is canceled.Display Log MessagesThe following example display stdout messages of a job.$cloudqclilog--id3f7e7681<display stdout of the job>The following example display stderr messages of a job.$cloudqclilog--id3f7e7681--error<display stdout of the job>The following example display log messages of an CloudQ Agent.$cloudqclilog--agentYOUR_SYSTEM<display agent log>Stageout (Get Job Input/Output/Log Files)$cloudqclistageout--id3f7e7681<download job files in the current directory>Delete Jobs or Agent logsThe following example deletes a completed job.$cloudqclidelete--id3f7e7681Job (3f7e7681) is deleted.The following example delete all completed jobs.$cloudqclidelete--allJob (3f7e7681) is deleted.Job (e210c27c) is deleted.The following example deletes a agent log.$cloudqclidelete--agentYOUR_SYSTEMAgent log (YOUR_SYSTEM) is deleted.List Submitted/Running Jobs$cloudqclilistjob-ID                  name     state  run-system            submit at-----------------------------------------------------------------------------3f7e7681  ljob_tf_mnist.abci.s      RUN         abci  2021/01/13 09:51:22e210c27c      mjob_tf_mnist.sh      READY       abci  2021/01/13 09:55:34fc2d6f45  ljob_tf_mnist.abci.s      INIT              2021/01/13 10:02:50List Completed Jobs$cloudqclihistoryjob-ID                  name     state  run-system            submit at             start at            finish at-----------------------------------------------------------------------------------------------------------------------3f7e7681  ljob_tf_mnist.abci.s      ERROR       abci  2021/01/13 09:51:22e210c27c      mjob_tf_mnist.sh      DONE        abci  2021/01/13 09:55:34  2021/01/13 10:06:16  2021/01/13 10:06:33Builder for AWSCloudQ Builder for AWS is provided ascloudqawscommand.
It uses AWS default profile to build a cluster.
Be aware that the AWS default profile is properly set.Create a Clustercloudqaws createcreates a cluster on AWS.
Before creating a cluster, you need to set up an SSH key pairs to log in to the cluster head node.$cloudqawscreate--nameyour-cluster-name\--keypairYOUR-KEYPAIR-NAME\--cs_profileabci--cs_endpointhttps://s3.abci.ai\--cs_bucketcloudqThe stack (your-cluster-name-vpc) is creating. Please wait a minute.The stack (your-cluster-name-nodes) is creating. Please wait a minute.AWS compute cluster (your-cluster-name) has been created.List Clusterscloudqaws listshows you status of compute clusters you requested to create.$cloudqawslistCluster Name     Status     Creation Time---------------------------------------------------your-cluster-01  COMPLETED  2022/01/23 11:22:33 UTCyour-cluster-02  COMPLETED  2022/02/01 23:59:59 UTCyour-cluster-03  CREATINGyour-cluster-04  FAILEDDelete a Cluster$cloudqawsdelete--nameyour-cluster-nameThe stack (your-cluster-name-nodes) is deleting. Please wait a minute.The stack (your-cluster-name-vpc) is deleting. Please wait a minute.AWS compute cluster (your-cluster-name) has been deleted.Change log level of CloudQCommands CloudQ Client、Agent and Builder can change the log level by the respectiveconfig.inisetting or--log_leveloption.
Log levels can be specified fromINFOorDEBUG.
The value specified with the--log_leveloption takes precedence.
If neither is specified,INFOis applied as the default log level.Meta JobscriptMeta Jobscript Descriptiondescribes how to use Meta jobscript.PublicationsShinichiro Takizawa Masaaki Shimizu, Hidemoto Nakada, Toshiya Matsuba, and Ryousei Takano, "CloudQ: A Secure AI / HPC Cloud Bursting System,"  9th International Workshop on HPC User Support Tools (HUST22), November 2022.