Package: yascheduler
Yet another computing scheduler & cloud orchestration engineYascheduleris a simple job scheduler designed for submitting scientific
calculations and copying back the results from the computing clouds.Currently it supports several scientific simulation codes in chemistry
and solid state physics.
Any other scientific simulation code can be supported via the declarative
control template system (seeyascheduler.confsettings file).
There is anexample dummy C++ codewith its configuration template.InstallationUsepipand PyPI:pip install yascheduler.The last updates and bugfixes can be obtained cloning the repository:gitclonehttps://github.com/tilde-lab/yascheduler.git
pipinstallyascheduler/The installation procedure creates the configuration file located at/etc/yascheduler/yascheduler.conf.
The file contains credentials for Postgres database access, used directories,
cloud providers and scientific simulation codes (calledengines).
Please check and amend this file with the correct credentials. The database
and the system service should then be initialized withyainitscript.UsagefromyaschedulerimportYascheduleryac=Yascheduler()label="test assignment"engine="pcrystal"struct_input=str(...)# simulation control file: crystal structuresetup_input=str(...)# simulation control file: main setup, can include struct_inputresult=yac.queue_submit_task(label,{"fort.34":struct_input,"INPUT":setup_input},engine)print(result)File paths can be set using the environment variables:YASCHEDULER_CONF_PATHConfiguration file.Default:/etc/yascheduler/yascheduler.confYASCHEDULER_LOG_PATHLog file path.Default:/var/log/yascheduler.logYASCHEDULER_PID_PATHPID file.Default:/var/run/yascheduler.pidConfiguration File ReferenceDatabase Configuration[db]Connection to a PostgreSQL database.userThe username to connect to the PostgreSQL server with.passwordThe user password to connect to the server with. This parameter is optionalhostThe hostname of the PostgreSQL server to connect with.portThe TCP/IP port of the PostgreSQL server instance.Default:5432databaseThe name of the database instance to connect with.Default: Same asuserLocal Settings[local]data_dirPath to root directory of local data files.
Can be relative to the current working directory.Default:./dataExample:/srv/yadatatasks_dirPath to directory with tasks results.Default:tasksunderdata_dirExample:%(data_dir)s/taskskeys_dirPath to directory with SSH keys.Default:keysunderdata_dirExample:%(data_dir)s/keysengines_dirPath to directory with engines repository.Default:enginesunderdata_dirExample:%(data_dir)s/engineswebhook_reqs_limitMaximum number of in-flight webhook http requests.Default: 5conn_machine_limitMaximum number of concurrent SSH connection'sconnectrequests.Default: 10conn_machine_pendingMaximum number of pending SSH connection'sconnectrequests.Default: 10allocate_limitMaximum number of concurrent task or node allocation requests.Default: 20allocate_pendingMaximum number of pending task or node allocation requests.Default: 1consume_limitMaximum number of concurrent task's results downloads.Default: 20consume_pendingMaximum number of pending task's results downloads.Default: 1deallocate_limitMaximum number of concurrent node deallocation requests.Default: 5deallocate_pendingMaximum number of pending node deallocation requests.Default: 1Remote Settings[remote]data_dirPath to root directory of data files on remote node.
Can be relative to the remote current working directory (usually$HOME).Default:./dataExample:/src/yadatatasks_dirPath to directory with tasks results on remote node.Default:tasksunderdata_dirExample:%(data_dir)s/tasksengines_dirPath to directory with engines on remote node.Default:enginesunderdata_dirExample:%(data_dir)s/enginesuserDefault ssh username.Default:rootjump_userUsername of default SSHjump host(if used).jump_hostHost of default SSHjump host(if used).Providers[clouds]All cloud providers settings are set in the[cloud]group.
Each provider has its own settings prefix.These settings are common to all the providers:*_max_nodesThe maximum number of nodes for a given provider.
The provider is not used if the value is less than 1.*_userPer provider override ofremote.user.*_priorityPer provider priority of node allocation.
Sorted in descending order, so the cloud with the highest value is the first.*_idle_tolerancePer provider idle tolerance (in seconds) for deallocation of nodes.Default: different for providers, starting from 120 seconds.*_jump_userUsername of this cloud SSH jump host (if used).*_jump_hostHost of this cloud SSH jump host (if used).HetznerSettings prefix ishetzner.hetzner_tokenAPI token with Read & Write permissions for the project.hetzner_server_typeServer type (size).Default:cx51hetzner_image_nameImage name for new nodes.Default:debian-10AzureAzure Cloud should be pre-configured foryascheduler. SeeCloud Providers.Settings prefix isaz.az_tenant_idTenant ID of Azure Active Directory.az_client_idApplication ID.az_client_secretClient Secret value from theApplication Registration.az_subscription_idSubscription IDaz_resource_groupResource Group name.Default:yascheduler-rgaz_userSSH username.rootis not supported.az_locationDefault location for resources.Default:westeuropeaz_vnetVirtual network name.Default:yascheduler-vnetaz_subnetSubnet name.Default:yascheduler-subnetaz_nsgNetwork security group name.Default:yascheduler-nsgaz_vm_imageOS image name.Default:Debianaz_vm_sizeMachine size.Default:Standard_B1sUpCloudSettings prefix isupcloud.upcloud_loginUsername.upcloud_passwordPassword.Engines[engine.*]Supported engines should be defined in the section(s)[engine.name].
The name is alphanumeric string to represent the real engine name.
Once set, it cannot be changed later.platformsList of supported platform, separated by space or newline.Default:debian-10Example:mY-cOoL-OS another-cool-osplatform_packagesA list of required packages, separated by space or newline, which
will be installed by the system package manager.Default: []Example:openmpi-bin wgetdeploy_local_filesA list of filenames, separated by space or newline, which will be copied
from local%(engines_dir)s/%(engine_name)sto remote%(engines_dir)s/%(engine_name)s.
Conflicts withdeploy_local_archiveanddeploy_remote_archive.Example:dummyenginedeploy_local_archiveA name of the local archive (.tar.gz) which will be copied
from local%(engines_dir)s/%(engine_name)sto the remote machine and
then unarchived to the%(engines_dir)s/%(engine_name)s.
Conflicts withdeploy_local_archiveanddeploy_remote_archive.Example:dummyengine.tar.gzdeploy_remote_archiveThe url to the engine arhive (.tar.gz) which will be downloaded
to the remote machine and then unarchived to the%(engines_dir)s/%(engine_name)s.
Conflicts withdeploy_local_archiveanddeploy_remote_archive.Example:https://example.org/dummyengine.tar.gzExample:cp{task_path}/INPUTOUTPUT&&mpirun-np{ncpus}--allow-run-as-root\-wd{task_path}{engine_path}/Pcrystal>>OUTPUT2>&1Example:{engine_path}/gulp < INPUT > OUTPUTcheck_pnameProcess name used to check that the task is still running.
Conflicts withcheck_cmd.Example:dummyenginecheck_cmdCommand used to check that the task is still running.
Conflicts withcheck_pname. See alsocheck_cmd_code.Example:ps ax -ocomm= | grep -q dummyenginecheck_cmd_codeExpected exit code of command fromcheck_cmd.
If code matches than task is running.Default:0sleep_intervalInterval in seconds between the task checks.
Set to a higher value if you are expecting long running jobs.Default:10input_filesA list of task input file names, separated by a space or new line,
that will be copied to the remote directory of the task before it is started.
The first input is considered as themaininput.Example:INPUT sibling.fileoutput_filesA list of task output file names, separated by a space or new line,
that will be copied from the remote directory of the task after it is finished.Example:INPUT OUTPUTAiida IntegrationSee the detailed instructions for theMPDS-AiiDA-CRYSTAL workflowsas well as theansible-mpdsrepository. In essence:sshaiidauser@localhost# importantreentryscan
verdicomputersetup
verdicomputertest$COMPUTERverdicodesetupLicense