Package: slurmdocs
SlurmDocs: Automated Slurm Cluster DocumentationSlurmDocsis a Python package and CLI tool designed to streamline the process of documenting Slurm clusters. Whether you're managing a small research cluster or a large-scale HPC system, SlurmDocs automates the documentation process, saving you time and ensuring your cluster information is always up-to-date.FeaturesAutomated tflops Calculation and Visualization:SlurmDocs can automatically calculate and visualize tflops (floating-point operations per second), providing valuable insights into your cluster's computational performance.Comprehensive Node Hardware Database:SlurmDocs creates a database of node hardware compute resources, including detailed CPU and GPU information. This data can be easily exported to formats such as CSV and SQL, and it's conveniently viewable with Pandas for in-depth analysis.Reliable and Well-Documented API:SlurmDocs offers a robust and well-documented API for programmatic usage, as well as a user-friendly command-line interface (CLI).Cluster Configuration Overview:SlurmDocs provides an overview of your cluster's configuration, partitions, and nodes, giving you a clear understanding of its current state.One-Command Sync:Keep your documentation in sync with the current cluster state with a single command, ensuring that your documentation is always up-to-date.Export to Various Formats:SlurmDocs supports exporting your documentation to various formats, including HTML, PDF, and CSV, making it easy to share and present information about your Slurm cluster.InstallationYou can install SlurmDocs usingpip:pipinstallslurmdocsUsageSlurmDocs is most easily used as a CLI tool. Theslurmdocscommand can be invoked after installation with the following options:Usage:slurmdocs[OPTIONS]COMMAND[ARGS]...SlurmDocsCLIforcollectingandanalyzingSLURMclusters.

Options:--versionShowtheversionandexit.-d,--debugEnabledebuglogging.--helpShowthismessageandexit.

Commands:collectSubcommandforSlurmDocsdatabaseoperations.databaseSubcommandforSlurmDocsdatabaseoperations.statsSubcommandforSlurmDocsstatistics.These subcommands are designed for their respective use cases.Collect SubcommandThe collect subcommand is used to gather node and CPU data from each node in the cluster. To use it, you must provide the following options to invoke additional subcommands:Usage:slurmdocscollect[OPTIONS]COMMAND[ARGS]...SubcommandforSlurmDocsdatabaseoperations.

Options:-u,--usernameTEXTTheusernametouse.[required]-s,--serverTEXTTheservertouse.[required]-p,--portINTEGERTheporttouse.-k,--key-pathPATHThekeytouse.--helpShowthismessageandexit.

Commands:cpuCollectCPUinformationforaspecificnode.nodeCollectthenodeinfofilefromthecluster.sweepPopulatethedatabasewithallthecollecteddata.Cluster Connection and Data CollectionTo connect to the cluster, you need to specify the username and server, and you can optionally provide the port (defaulting to a standard port). SlurmDocs supports key-based authentication, and you can specify the path to your SSH keys (default location is~/.ssh). This allows for a secure and convenient SSH connection to the cluster. The subcommands are invoked in the following manner,slurmdocscollect-ujhondoe-sjhondoe.edu-p23-k~/.sshcpu-ncompute-10-3-pdebug-qosdebug-save~/dataThis will download the output of thelscpucommand on thecompute-10-3as a text file in the~/datadirectory. Similarly, all the data are collected for all of the available nodes. The node command collects the output of the scontrol show node and saves it as a text file.To automate the process of collecting the data and creating a database automatically, users are recommended to use the sweep command.slurmdocscollect-ujhondoe-sjhondoe.edu-p23-k~/.sshsweep-dbdoehpc-t10This will automatically download the node file and collect all the cpu data from the nodes available. This assumes the partition names in the cluster
are lower case. Optionally, user can override the partiotion and qos needed to collect the data form each compute node in cluster. This is done by providing,slurmdocscollect-ujhondoe-sjhondoe.edu-p23-k~/.sshsweep-dbdoehpc-t10-pmain-qosmain--overrideThis assumes that the "main" partition can queue jobs in all nodes with the crossponding user account.Database SubcommandAfter sweeping the data, by default the data is stored in~/.slurmdocsdirectory. User can use thedatabasesubcommand to access and maniputlate the data. Each recorded cluster has their own database which can be viewed by,slurmdocsdatabaseavailHere is a sample output from avail command:This will list all the database which are avaible. Additionally, there is a check mark to the side for each uncorroupted or non empty database.
Following additional commands are available,Usage:slurmdocsdatabase[OPTIONS]COMMAND[ARGS]...Subcommandfortheslurmdocsdatabaseoperations.

Options:--versionShowtheversionandexit.-p,--pathPATHThepathtothedatabase.--helpShowthismessageandexit.

Commands:availListtheavailabledatabases.cleanCleantheemptydatabases.coverageCoverthedatabase.createCreatethedatabasesubdirectories.destroyDestroythedatabaseincludingit'ssubdirectory.insertInsertthefileintothedatabase.integrityChecktheintegrityofthedatabase.listListthedatabasestreestructure.queryQuerythedatabase.removeRemovetheentriesfromthedatabase.updateUpdatethecertanpre-exitingdatafile.The database hierriecy are arranged in two keys (node and cpu) which is required by few subcommands like query and insert. In addition to the CLI, you can use the Python API to manipulate the data. The data is provided as a pandas.DataFrame object, which you can use for custom data analysis and processing.Stat SubcommandThe stat command is used to calculate the statistics for the database after collection process is complete. It has follwing subcommands,Usage:slurmdocsstats[OPTIONS]COMMAND[ARGS]...Subcommandfortheslurmdocsstatistics.

Options:--versionShowtheversionandexit.--helpShowthismessageandexit.

Commands:plotsGenerateplotsfromthetflopsfile.summarizeSummarizethetflopsfiletostdout.tflopsCalculatetheTFLOPSofeachnodeinthecluster.First the user need to generate the tflops file which process and parse the raw text file in the database and create a good dataframe which can be published in the website or used for further processing. The follwing command shows the usageslurmdocsstatstflops-dbjhondoe-fthtmlThis will generate the html table of the processed data. Now, this file can be used to generate plots and summary of the cluster.slurmdocsstatssummarize-tjhondoe_tflops.htmlThe follwing is the sample output of the summary output,slurmdocsstatstflopsplots-ftjhondoe_tflops.htmlThese will generate a summary in stdout and plots files in the cwd. The sample plots are,Plot : NodeName vs TFlopsPlot : Cpu Model CompositionPlot : GPUs vs TFlopsUsing the Python APIThe SlurmDocs package can be independently used as an API by importing the slurmdocs module after installation. This module is well-documented, making it easy to configure according to your specific requirements. The documentation for collecting and analyzing data using the API is available in the form of a Jupyter notebook located in the/docsdirectory.This provides flexibility for users who prefer programmatic access to SlurmDocs. The Jupyter notebook serves as a practical guide to help you leverage the API for your specific use cases and data analysis needs.Using the Python APIThe SlurmDocs package can be independently used as an API by importing theslurmdocsmodule after installation. This module is well-documented, making it easy to configure according to your specific requirements. The documentation for collecting and analyzing data using the API is available in the form of a Jupyter notebook located in the/docsdirectory.This provides flexibility for users who prefer programmatic access to SlurmDocs. The Jupyter notebook serves as a practical guide to help you leverage the API for your specific use cases and data analysis needs.Contributions and LicenseWe welcome contributions from the open-source community to enhance the functionality and features of SlurmDocs. If you're interested in contributing or have ideas for improvement, please consider the following:Feature Enhancements:You can contribute by enhancing existing features or proposing new ones. Feel free to submit feature requests or pull requests on theGitHub repository.Bug Reports:If you encounter any issues or bugs while using SlurmDocs, please report them on theissue tracker. Your feedback is valuable in improving the tool.Documentation:Contributions to documentation, including corrections, improvements, and additional examples, are highly appreciated. You can directly edit the documentation through the GitHub repository.Spread the Word:If you find SlurmDocs valuable, consider sharing it with your network, colleagues, or on social media. Your support helps us reach a wider audience.License:SlurmDocs is released under the MIT License, which encourages open-source collaboration. You can find the complete license details onopensource.org. By using or contributing to SlurmDocs, you agree to abide by the terms of this license.We appreciate your contributions and support in making SlurmDocs a valuable resource for the community.