Package: waldur-slurm-agent
Waldur SLURM Integration ServiceService for Mastermind integration with SLURM cluster. The main purpose of the service is data syncronization between Waldur instance and SLURM cluster. The application uses order-related information from Waldur to manage accounts in SLURM and accounting-related info from SLURM to update usage data in Waldur.ArchitectureThis is a stateless application, which should be deployed on a machine having access to SLURM cluster data. The service consists of two sub-applications:service-pull, which fetches data from Waldur and updates a state of a SLURM cluster correspondingly (e.g. creation of SLURM accounts ordered in Waldur)service-push, which sends data from SLURM cluster to Waldur (e.g. update of resource usages)Integration with WaldurFor this, the service usesWaldur clientbased on Python and REST communication withWaldur backend. Service-pull application pulls orders data created for a specific offering linked to SLURM cluster and creates/updates/removes SLURM accounts based on the data. Service-push fetches data of usage, limits and associations from SLURM cluster and pushes it to Waldur.Integration with SLURM clusterFor this, service uses uses SLURM command line utilities (e.g.sacctandsacctmgr). The access to the binaries can be either direct or using docker client. In the latter case, the service is required to have access todockerbinary and to docker socket (e.g./var/run/docker.sock).SetupThe application supports the following environmental variables (required ones formatted with bold font):WALDUR_API_URL- URL of Waldur Mastermind API (e.g.http://localhost:8081/api/).WALDUR_API_TOKEN- token for access to Mastermind API.WALDUR_SYNC_DIRECTION- accepts two values:pushandpull. Ifpull, then application sends data from SLURM cluster to Waldur, vice versa ifpush.WALDUR_OFFERING_UUID- UUID of corresponding offering in Waldur.REQUESTS_VERIFY_SSL- flag for SSL verification for Waldur client, default istrue.SLURM_TRES_CONFIG_PATH- a path to the SLURM TRES configuration, default is./config-components.yaml.SLURM_DEPLOYMENT_TYPE- type of SLURM deployment. accepts two values:dockerandnative, default isdocker.SLURM_CUSTOMER_PREFIX- prefix used for customer's accounts, default ishpc_.SLURM_PROJECT_PREFIX- prefix used for project's accounts, default ishpc_.SLURM_ALLOCATION_PREFIX- prefix used for allocation's accounts, default ishpc_.SLURM_ALLOCATION_NAME_MAX_LEN- maximum length of account name created by the application.SLURM_DEFAULT_ACCOUNT- default account name existing in SLURM cluster for creation of new accounts. Default iswaldur.SLURM_CONTAINER_NAME- name of a headnode SLURM container; must be set if SLURM_DEPLOYMENT_TYPE is docker.SENTRY_DSN- Data Source Name for Sentry (more infohere)ENABLE_USER_HOMEDIR_ACCOUNT_CREATION- whether to create home directories for users related to accountsDeploymentTest environmentIn order to test the service, a user should deploy 2 separate instances of the service.
The first one (called service-pull) is for fetching data from Waldur with further processing and the second one (called service-push) is for sending data from SLURM cluster to Waldur.
Both instances must be configured with environment variables from e.g. .env-file.The example of .env-file for service-pull:WALDUR_SYNC_DIRECTION=pull # The setup for service-pull
WALDUR_API_URL=http://waldur.example.com/api/ # Waldur API URL
WALDUR_API_TOKEN=9e1132b9616ebfe943ddf632ca32bbb7e1109a32 # Token of a service provider in Waldur
WALDUR_OFFERING_UUID=e21a0f0030b447deb63bedf69db6742e # UUID of SLURM offering in Waldur
SLURM_DEFAULT_ACCOUNT=root # Default account for SLURM
SLURM_CONTAINER_NAME=slurmctld # Name of SLURM namenode containerThe example of .env-file for service-push:WALDUR_SYNC_DIRECTION=push # The setup for service-push
WALDUR_API_URL=http://waldur.example.com/api/ # Waldur API URL
WALDUR_API_TOKEN=9e1132b9616ebfe943ddf632ca32bbb7e1109a32 # Token of a service provider in Waldur
WALDUR_OFFERING_UUID=e21a0f0030b447deb63bedf69db6742e # UUID of SLURM offering in Waldur
SLURM_CONTAINER_NAME=slurmctld # Name of SLURM namenode containerDocker-based deploymentYou can find the Docker Compose configuration for testing inexamples/docker-compose/folder.In order to test it, you need to execute following commands in your terminal app:cdexamples/docker-compose
docker-composeup-dSystemd deploymentIf your SLURM cluster doesn't run in Docker, you need to deploy the a systemd service starting using Python module.
The agent requiressacctandsacctmgrto be accessible on a machine, so it should run on a headnode of the SLURM cluster.
Firstly, install the waldur-slurm-agent:pipinstallwaldur-slurm-agentSecondly, put systemd unit, environment and and TRES config files to the corresponding locations.
Don't forget to modify Waldur-related values the env files.# For pulling servicecpsystemd-conf/service-pull/waldur-slurm-service-pull.service/etc/systemd/system/
mkdir/etc/waldur-slurm-service/
cpsystemd-conf/service-pull/waldur-slurm-service-pull.env/etc/waldur-slurm-service/pull.env
cp./config-components.yaml.example/etc/waldur-slurm-service/tres.yaml# you can use a different path and set SLURM_TRES_CONFIG_PATH to it# For pushing servicecpsystemd-conf/service-push/waldur-slurm-service-push.service/etc/systemd/system/
cpsystemd-conf/service-push/waldur-slurm-service-push.env/etc/waldur-slurm-service/push.envAfter the preparation, run the following to apply the changes.systemctldaemon-reload
systemctlstartwaldur-slurm-service-pull
systemctlenablewaldur-slurm-service-pull# to start after rebootsystemctlstartwaldur-slurm-service-push
systemctlenablewaldur-slurm-service-push# to start after rebootOlder systemd versionsIf you want to deploy the services on a machine with systemd revision older than 240, you should use files with legacy configuration:# For pulling servicecpsystemd-conf/service-pull/waldur-slurm-service-pull-legacy.service/etc/systemd/system/waldur-slurm-service-pull.service# For pushing servicecpsystemd-conf/service-push/waldur-slurm-service-push-legacy.service/etc/systemd/system/waldur-slurm-service-push.serviceTRES configurationTo setup TRES-related info, the service uses the corresponding configuration file configured bySLURM_TRES_CONFIG_PATHenvironment variable (config-components.yamlby default). Each entry of the file incudes key-value-formatted data.
A key is a type of TRES (with optional name if type isgres) and the value contains limit, measured unit, type of accounting and label.
The service sends this data to Waldur when you run it with--load-components:python3-mwaldur_slurm.main--load-componentsIf a user wants to change this information, a custom config file should be mounted into a container and setSLURM_TRES_CONFIG_PATHvalue to a correct location.Service provider configurationThe services require existing offering in Waldur.
As a service provider owner, you should create an offering in the marketplace:Go toProvidersection on the left tab ->Add new offeringbuttonInput a name, choose a category, selectSLURM remote allocationfrom the drop-down list on the bottom and clickCreatebuttonOpen the offering page and create a plan in theAccountingsection: clickAdd planand input the necessary detailsGo toIntegrationsection, clickShow integration stepsand ensure they are completed within your SLURM cluster.