Saravanan666/pypi-org huggingface dataset:
Package: slumber-fork
SlumberSlumber is a Python library that provides a convenient yet powerful
object-oriented interface to ReSTful APIs. It acts as a wrapper around the
excellentrequestslibrary and abstracts away the handling of URLs, serialization,
and request processing.Getting HelpVisit IRC channel (#slumber on irc.freenode.net) to get help, bounce ideas
or generally shoot the breeze.QuickStartInstall Slumber:$ pip install slumberInstall Optional Requirement:pip install pyyamlUse Slumber!RequirementsSlumber requires the following modules.Python 2.6+requestspyyaml (If you are using the optional YAML serialization)Changelogdevelopment version0.6.2Further Python 3 compatibility fixes, many thanks to Alexander Shchepetilnikov (irqed)0.6.1Removesimplejsondependency.Support range now Python 2.6 - Python 3.40.6.0All HTTP methods support multipart/form-data.  The newfilesparameter is directly passed torequests.  Thanks to Philip
Neustrom and Diego Gaustein.0.5.2Now compatible withrequests1.0 thanks to Matias Saguir.Fix bug when using default serializer.  Thanks to Andy McKay0.4.2Support decoded unicode url fragments - Thanks @collinwat0.4.1Added asessionkwarg toslumber.APIallowing passing arequestssession
that will be used instead of the slumber created one0.4.0(Backwards Incompatible)Switched fromhttplib2torequests(Backwards Incompatible)Removed the Meta class Magic(Backwards Incompatible)Removed the ability to subclassslumber.APIto specify defaults(Backwards Incompatible)New Syntax for Specifying Authentication0.3.1Fix regression where pre 0.3 urls were assumed to end in slash, and 0.3.0 presumed to end in not slash.
Urls are now assumed to end in a slash, and if you don’t want this behavior you can disable it by the
append_slash kwarg/Meta option (set to False to disable it).Fix regression caused by a mistyped variable name.0.3.0Allowed nesting resources infinitely to allow more complex api usage.Cleaned up the Meta class and allow subclassingslumber.API(Backwards Incompatible)Cleaned up the exception names.(Backwards Incompatible)Renamed theslumber.APIserialization kwarg from
default_format to format to be more consistentImproved the documentationAdded Some Tests (This could still be better)0.2.5Fixed https urls and the accidental force to port 80Fixed the assumption that all urls end in a trailing slash0.2.4Fixed Including of Changelog.rst0.2.3Updated the docs to include a section about url parameters0.2(Backwards Incompatible)Move specifying a non default serializer fromapi.resource.get(format="yaml")toapi.resource(format="yaml").get()Reworked the internalResourceapi to not clobber any kwargs passed to it. This
fixes a bug where you couldn’t useformatorurlas the name for one of
the url parameters.0.1.3Fix forResource.post()not passing kwargs toResource.get()0.1.2Initial public release of Slumber
Package: slumber-serializers
A set of Slumber serializersInstallationInstall package viapippipinstallslumber-serializersUsageimportslumberimportslumber.serializefromslumber_serializersimportCsvSerializerapi=slumber.API('/api/v1/',serializer=slumber.serialize.Serializer(default='csv',serializers=[CsvSerializer()]),format='csv')api.test(format='csv').get()Available serializersCSV serializerSerialize to and deserialize from CSV.Binary serializerSerialize to and deserialize from any binary format.
Package: sl-up
sl-upAn interactive TUI smartlog forSapling SCM, like git-sl-up but written in python3 and based oncurses.This is my own personal implementation for a tool available internally at Meta. FWIW, I do not have access to that tool's code so if they choose to open source the internal tool at any time, you should prefer that one over mine. I would ;)Installation# using pip
$ pip install sl-up

# or with pipx
$ pipx install git+https://github.com/vuduchild/sl-up.git
Package: slurchemy
PythonSQLAlchemybindings for your slurmdbd
(SLURMdatabase).InstallingEasy:$ virtualenv test-environment && source test-environment/bin/activate
$ pip install slurchemyor:$ sudo pip install slurchemyUsingAssuming you haveslurmdbdconfigured in/etc/slurm/slurmdbd.confwith:StorageType=accounting_storage/mysql
StorageHost=mysql.example.org
StoragePass=pass
StorageUser=slurm
StorageLoc=slurmdbThen in python you can do:import slurchemy
from sqlalchemy import create_engine
engine = create_engine('mysql://slurm:pass@mysql.example.org/slurmdb')
slurchemy.init_model(engine)
clusters = slurchemy.Cluster.query.all()TODOBuiltin statistics queries
Package: slurm-cola
Slurm ColaSlurm, the number-one soft drink of the future, is refreshingly addictive.slurm-cola is a tool developed within EuXFEL's detector group to manage a user's
jobs, ie: cancel them in bulk.As part of the calibration pipeline, slurm jobs are submitted perdata run.These jobs share in common their name, which allows grouping and bulk actions
on them.Although tailored for EuXFEL's calibration pipeline, there's no reason that
it might not work for you.InstallingThis package requires Python 3.6+, and can be installed so:git clone ssh://git@github.com/cydanil/slurm-cola
cd slurm-cola
pip install .Usageslurm-colais a graphical application, launch it so:$ slurm-colaIn the main window, you will be given a list of jobs for the current user.
Clicking on one will show its settings.A simple logging window is available for easier reporting during OCD calls.It's possible to specify the user:$ slurm-cola <username>Cancelling jobs for that user will require you to have the correct permissions
though.Future Developmentslurm-colais still under early development. There are plans of, in the following order:cool logoprettier job info display
Package: slurmee
slurmeeSimple Python 3 package to get information when running Python programs inslurm.Usageimportslurmeeprint(f"Slurm job id:{slurmee.get_job_id()}")print(f"Slurm job name:{slurmee.get_job_name()}")print(f"Slurm submit dir:{slurmee.get_submit_dir()}")print(f"Slurm job nodelist:{slurmee.get_job_nodelist()}")print(f"Slurm submit host:{slurmee.get_submit_host()}")print(f"Slurm job num_nodes:{slurmee.get_job_num_nodes()}")print(f"Slurm cpus on node:{slurmee.get_cpus_on_node()}")print(f"Slurm ntasks:{slurmee.get_ntasks()}")print(f"Slurm nodeid:{slurmee.get_nodeid()}")Methods returnNoneif not running inside of slurm. It can be used to check this condition e.g.ifslurmee.get_job_id():print("Running inside of slurm!")DevelopmentThis package has no external runtime dependencies. The tests can be run viamake testTo install the dependencies for generating the docs and releasing the package, in a virtual environment, runpip install -e ".[test,dev,doc]"
Package: slurmer
SlurmerThis package is designed to run heavy loads in a computer cluster. Supported task schedulers are:Running on a single nodeSLURMPRs to add more schedulers are welcome.FeaturesRun multiple tasks in parallel.Run tasks in randomised order but consistently in multiple computing nodes.Consistent pipeline to work with heavy tasks (directory creation, handling errors, postprocessing...)ExampleYou can find the documentationhere.The package can be used as follows:fromdataclassesimportdataclassfrompathlibimportPathfromtypingimportIteratorimportslurmer@dataclassclassMyTaskParameters(slurmer.TaskParameters):processing_id:int@dataclassclassMyTaskResult(slurmer.TaskResult):good_result:boolclassMyTask(slurmer.Task):def__init__(self,min_job:int,max_job:int):super().__init__()self.min_job=min_jobself.max_job=max_jobdefgenerate_parameters(self)->Iterator[MyTaskParameters]:foriinrange(self.min_job,self.max_job):yieldMyTaskParameters(processing_id=i)defmake_dirs(self):# Dirs are created before the task is runforiinrange(self.min_job,self.max_job):Path(f"out/result_{i}").mkdir(parents=True,exist_ok=True)defprocess_function(self,parameters:MyTaskParameters)->MyTaskResult:# Run your heavy code hereforiinrange(parameters.processing_id,parameters.processing_id+10):print(f"Processing{i}")returnMyTaskResult(good_result=True)defprocess_output(self,result:MyTaskResult)->bool:# Postprocess output, this is NOT run in parallel and only handles the tasks run# in the current node. If you want to postprocess everything wait for all the nodes# to finsish computing.## This function should check whether the output is the "expected" one and return True# if something went wrong.returnnotresult.good_resultThen in your main script:>>>task=MyTask(min_job=1,max_job=10)>>>task.execute_tasks()
Package: slurmflow
SlurmflowSlurmflow is a job dependency manager built to interact with slurm clusters! It serves as an abstraction between your data processing and the implementation details of Slurm. Slurmflow represents your tasks as a Directed Acyclic graph, where each node is a sbatch script to be called upon. When you "run" the dag, Slurflow recursively traverses the graph from sink (no outgoing nodes) to the source (no incoming nodes) and submits each script to slurm using the --dependency flag.ExampleHere is a simple example with three scripts (t1, t2, t3), with t3 requiring outputs of the previous two tasks. When creating each Job, all that's needed is a name for the node and the filename of a sbatch script to call. Once your jobs are created, it's possible to use the bitshift operators [<<, >>] to set dependencies. The operator 1 >> 2 denotes that 1 must finish before 2 can run, while 1 << 2 denotes the opposite. If your pipeline allows for certain steps to be run in parallel, you can add them into a list [] and perform the bitshift operations. For a more conventional syntax, you can alseo use the set_upstream or set_downstream methods like so t1.set_downstream(t2). The equivalent notation using bitshifting is t1 >> t2.Workflow.pyfromslurmflowimportJobfromslurmflowimportDAGwithDAG('simple_slurm_workflow',env=env)asd:t1=Job('Task1','task1.sh',dag=d)t2=Job('Task2','task2.sh',dag=d)t3=Job('Task3','task3.sh',dag=d)[t1,t2]>>t3d.run()d.plot()Scriptstask1.sh#!/bin/bash#SBATCH --job-name=slurmflow_test_1#SBATCH --output=slurmflow_test_1.out#SBATCH --ntasks=1#SBATCH --time=10:00#SBATCH --mem-per-cpu=1Gcatdata1.txt>data4.txt

envecho"task 1 complete"task2.sh#!/bin/bash#SBATCH --job-name=slurmflow_test_2#SBATCH --output=slurmflow_test_2.out#SBATCH --ntasks=1#SBATCH --time=10:00#SBATCH --mem-per-cpu=1Gcatdata2.txt>data3.txt

envecho"task 2 complete"task3.sh#!/bin/bash#SBATCH --job-name=slurmflow_test_3#SBATCH --output=slurmflow_test_3.out#SBATCH --ntasks=1#SBATCH --time=10:00#SBATCH --mem-per-cpu=1Gcatdata3.txt>>final.txt
catdata4.txt>>final.txt
rmdata3.txtdata4.txt

envecho"temporary files removed"Data Filesdata1.txt1
2
3
4
5data2.txt10
11
12
13
14
15final.txt10
11
12
13
14
15
1
2
3
4
5
Package: slurmgen
SlurmGen - Simple Slurm ManagerRepository:github.com/otvam/slurmgenPackage:pypi.org/project/slurmgenSummarySlurmGenis a verysimpleSlurmmanager:The job description is stored in JSON files.The JSON files are transformed into Slurm scripts.The Slurm script can be run locally or on a Slurm Cluster.SlurmGenis only supporting a smallsubsetofSlurm:Create and delete folders.Set environment variables.Set the job name and log.Set the ressources (memory, time, CPU, etc.).Define the set of commands to be executed.SlurmGen is not a binding and/or a wrapper for Slurm.
SlurmGen is only generating Slurm scripts from JSON files.
SlurmGen is written in Python without any external dependencies.ExampleAn example is located in theexamplefolder of the repository:job_def.jsoncontains the job definitionjob_tmpl.jsoncontains the template datarun_slurm.pycontains is dummy script executed by the jonslurm_outputcontains the generated Slurm script and log filedata_outputcontains data generated by the Slurm job# Create a Slurm script with a command line template.sgenjob_def.json--tmpl_str"{'title': 'test', 'var': 'Welcome to everyone'}"# Create a Slurm script with a JSON file template.sgenjob_def.json--tmpl_filejob_tmpl.json# Create a Slurm script and run the job locally.sgenjob_def.json--local--tmpl_filejob_tmpl.json# Create a Slurm script and run the job on the cluster.sgenjob_def.json--cluster--tmpl_filejob_tmpl.jsonProject LinksRepository:https://github.com/otvam/slurmgenReleases:https://github.com/otvam/slurmgen/releasesTags:https://github.com/otvam/slurmgen/tagsIssues:https://github.com/otvam/slurmgen/issuesPyPi:https://pypi.org/project/slurmgenAuthorThomas GuillodEmail:guillod@otvam.chWebsite:https://otvam.chCopyright(c) 2023 - Thomas GuillodBSD 2-Clause "Simplified" License
Package: slurm-gpustat
slurm_gpustatslurm_gpustatis a simple command line utility that produces a summary of GPU usage on a slurm cluster. The tool can be used in two ways:To query the current usage of GPUs on the cluster.To launch a daemon which will log usage over time.  This log can later be queried to provide usage statistics.InstallationInstall viapip install slurm_gpustat.  If you prefer to hack around with the source code, it's asingle python file.UsageTo print a summary of current activity:slurm_gpustatTo print a summary of current activity on particular partitions, e.g.debug&normal:slurm_gpustat -p debug,normalorslurm_gpustat --partition debug,normalTo start the logging dameon:slurm_gpustat --action daemon-startTo view a summary of logged data:`slurm_gpustat --action history`Example outputsRunningslurm_gpustatwill produce something like this:----------------------
Under SLURM management
----------------------
There are a total of 12 gpus [up]
3 rtx6k gpus
2 v100 gpus
3 m40 gpus
4 p40 gpus
----------------------
There are a total of 10 gpus [accessible]
1 rtx6k gpus
2 v100 gpus
3 m40 gpus
4 p40 gpus
----------------------
Usage by user:
user1     [total: 1 ] m40: 1
user2     [total: 1 ] p40: 1
user3     [total: 3 ] p40: 3
----------------------
There are 5 gpus available:
p40: 0
rtx6k: 1
v100: 2
m40: 2Adding--verboseto this command will produce a more detailed breakdown for the section describing gpus that are still available.  Example output:There are 18 gpus available:
m40: 5 available
 -> gnodeb3: 1 m40 [cpu: 38/40, gres/gpu: 3/4, mem: 68G/257669M] [user1,user2]
 -> gnodeb4: 2 m40 [cpu: 38/40, gres/gpu: 2/4, mem: 60G/193161M] [user1,user2]
 -> gnodec3: 2 m40 [cpu: 12/48, gres/gpu: 2/4, mem: 36G/257669M] [user1]
p40: 5 available
 -> gnodec4: 1 p40 [cpu: 20/48, gres/gpu: 3/4, mem: 216G/257669M] [user1]
 -> gnoded1: 1 p40 [cpu: 44/64, gres/gpu: 3/4, mem: 60G/385192M] [user4,user5]
 -> gnoded3: 1 p40 [cpu: 28/64, gres/gpu: 3/4, mem: 70G/385192M] [user2,user4,user5]
 -> gnoded4: 1 p40 [cpu: 28/64, gres/gpu: 3/4, mem: 112G/385192M] [user5,user4]
 -> gnodee3: 1 p40 [cpu: 36/56, gres/gpu: 3/4, mem: 60G/385192M] [user4,user2]
rtx6k: 3 available
 -> gnodef1: 1 rtx6k [cpu: 36/56, gres/gpu: 3/4, mem: 194G/257669M] [user2,user3]
 -> gnodeh2: 2 rtx6k [cpu: 16/24, gres/gpu: 2/4, mem: 96G/385345M] [user2,user1]
v100: 5 available
 -> gnodeg1: 2 v100 [cpu: 16/64, gres/gpu: 2/4, mem: 60G/191668M] [user5,user1]
 -> gnodeg2: 3 v100 [cpu: 8/64, gres/gpu: 1/4, mem: 40G/191668M] [user2]Runningslurm_gpustat --action history(after the daemon has run for a little while) will produce something like this:Historical data contains 7 samples (2020-01-03 11:51:43 to 2020-01-03 11:51:45)
GPU usage for user1:
v100  > avg: 2, max: 4
m40   > avg: 1, max: 1
total > avg: 3

GPU usage for user2:
p40m  > avg: 3, max: 4
total > avg: 3DependenciesPython >= 3.6numpybeartypeseaborncolored
Package: slurm-in-aws
slurm_in_awsMock EC2 with moto + launch instances in Docker.Usage# TODOContributingContributing SetupClone the project locallyInstall the corresponding.python-versionusing something likepyenvCreate a virtual environment named.venvwithpython -m venv .venvActivate the virtual environment withsource .venv/bin/activateInstallpoetryInstallinvokewithpip install invokeRunpoetry install --no-rootRuninvoke setupContributing TestsRunpoetry run invoke testsContributing All Checks (including tests)Runpoetry run invoke hooksBuild And Publish to PyPIpoetrybuild
poetrypublish
Package: slurmio
slurmioPython tools to readSLURMjob
parameters.InstallationPippipinstallslurmioCondacondainstall-cconda-forgeslurmioFrom source:gitclonehttps://github.com/neuroinformatics-unit/slurmiocdslurmio
pipinstall.Usage>>>fromslurmioimportSlurmJobParameters
>>>slurm_parameters=SlurmJobParameters()>>>slurm_parameters.job_id994986>>>slurm_parameters.job_name'bash'>>>slurm_parameters.partition'cpu'>>>slurm_parameters.requested_cores10>>>slurm_parameters.allocated_cores10>>>slurm_parameters.requested_nodes10>>>slurm_parameters.allocated_nodes10>>>slurm_parameters.requested_memory10240# in bytes>>>slurm_parameters.allocated_memory10240# in bytes
Package: slurm-job-resource-monitor
# Slurm Job Resource MonitorA python command line tool to monitor the resources of your slurm jobs in form of a regularly updated table.![Demo Image](https://github.com/tmachnitzki/slurm_job_resource_monitor/blob/main/demo.jpg)## InstallationInstall via pip:`bash pip installslurm-job-resource-monitor`## UsageOn a slurm managed cluster, run the following command to monitor your jobs:`bash slurm_job_monitor `To get all the possible options run:`bash slurm_job_monitor--help`The preferred way to use the tool is with the group-by-cmd option to not spam the table when running multi-cpu jobs:`bash slurm_job_monitor--group-by-cmd"sum" `## Assumptions / PreliminaryThe tool assumes that:you are running on a slurm managed cluster.The tool assumes that you have a slurm account.You can ssh in all allocated notes.For ssh within the cluster no password is needed (only ssh key).You need nvidia-smi to display GPU info of nodes (optional).You can run “ps auxfww” to display CPU info of nodes.## Some Further InfoOnly processes are displayed that have a CPU load > 0.0 $ as shown by “ps auxfww”. This is to avoid
the spamming of the table with processes that are not doing anything.The table is grouped by jobs. So if one job runs on multiple GPUs and nodes they will all be displayed
in the same “section” of row. Sections are separated by a horizontal line.There is a minimal width that the terminal needs to have to display the table correctly. The only column that
allows wrapping is the “Command” column.
Package: slurmlint
A Simple linter for Slurm Config FilesPurposeIn my experience it is very easy to make small errors in slurm configuration
files, incorrectly listing nodes in the cluster, or failing to assign nodes
to partitions, and so on. This package provides a simple command to check
slurm files that is suitable for use in automated deployment management.In addition, a python API is provided allowing checking arbitrary strings
of text. Utility functions such as explanding slurm-style hostname lists
are also given.UsageTheslurmlintcommand takes a slurm configuration file as an argument
and returns a simple report with errors (if any). If there were errors
then the return code is 1, otherwise 0.$ slurmlint slurm.conf
753 compute nodes configured

Errors detected:
Line 126 - Duplicate node definition: ng1031, ng908, ng909, ...
Line 127 - Duplicate node definition: ng1031
Line 128 - Duplicate node definition: ng908, ng909, ng910
Line 131 - Defined node has no partition: ng1259
Line 131 - Duplicate node definition: ng1256LimitationsThis utility does not fully understand the grammar of slurm configuration
files and does not guarantee the validity of a configuration file. It only
serves to help find certain kinds of simple errors.Hostname ListsThis linter applies stricter rules than Slurm for lists of hostnames.
Hostnames must be valid hostnames according toRFC-1123, that is they may contain
a-z, 0-9, dot, and the minus sign, at most 253 ASCII characters, and must
be comprised of "labels" from 1-63 characters connected by dots.Hostnames can be separated by commas in a list.The standard (???) rules for brackets apply. If a list entry uses brackets
to denote some numerical range or comma separated list, there must be
brackets at the end of the hostname. One can also have a bracketed range
somewhere else in the hostname provided that there are brackets at the end.
Brackets may contain valid non-negative integers separated by commas, or
ranges such as1-8. A leading zero in a range denotes a fixed width, i.e.001-020.Note that this linter does not use the slurm parser and will result in
"false positive" warnings where allowed hostname ranges still result in
warnings.
Package: slurm-mongo
Slurm-mongoSlurm-Mongo is a Python package that downloads SLURM usage data from a supercomputer and uploads it to MongoDB.InstallationYou can install Slurm-Mongo using pip:pipinstallslurm-mongoUsageYou can use Slurm-Mongo by running theslurm_mongocommand followed by the start date, end date, and supercomputer host name in user@hostname format:slurm_mongostart_dateend_datesupercomputer_host_nameExampleslurm_mongo'2017-01-01''2017-01-02''shinde@snellius.surf.nl'By default, Slurm-Mongo will connect to a MongoDB instance running onlocalhost:27017and will store the data in a database called "Snellius" and a collection called "Usage". You can customize the MongoDB connection and database/collection names by using the optional--connection,--port,--database, and--collectionflags:slurm_mongostart_dateend_datesupercomputer_host_name--connectionmongo.example.com--port12345--databaseMySlurm--collectionMyUsageLicenseSlurm-Mongo is released under the MIT License. See LICENSE for more information.IssuesIf you encounter any issues with Slurm-Mongo, please report them on the GitHub issue tracker.ContactContact the author Dr. Ravindra Shinde atr.l.shinde@utwente.nl
Package: slurmpter
Slurmpter (Slurm Scripter) is a package to build Slurm submit files of a workflow of jobs easily. This package usesPyCondoras the backend. The user interface of slurmpter is very similar to that ofPyCondorexcept for some arguments dedicated for Slurm.DocumentationDocumentation ofslurmptercan be found athttps://slurmpter.readthedocs.io/en/latest/index.html.InstallationPyPIThe latest release ofslurmptercan be installed withpip:pipinstallslurmpterCondaThe latest release ofslurmptercan be installed withconda:condainstall-cisaac-cfwongslurmpterUseful Linksslurmpter @ GitLabslurmpter mirror @ GitHubIssue trackerCopyright (c) 2020 Isaac Chun Fung WONG
Package: slurmpy
quick and dirty lib for submitting jobs to slurm via python2/python3.fromslurmpyimportSlurms=Slurm("job-name",{"account":"my-account","partition":"my-parition"})s.run("""dolotsofstuff""")The above will submit the job tosbatchautomatically write the script toscripts/and automatically write logs/{name}.err and logs/{name}.out. It will have today's
date in the log and script names.The script to run() can also contain$variableswhich are filled with the cmd_kwarg dict.
E.g.echo $namecould be filled withcmd_kwargs={'name': 'sally'}A command can be tested (not sent to queue) by setting the_cmdare torunas e.g. "ls".
The default issbatchwhich submits jobs to slurm.DependenciesEach timeslurmpy.Slurm().run()is called, it returns the job-id of the submitted job. This
can then be sent to a subsequent job:s = Slurm()
s.run(..., depends_on=[job_id])to indicate that this job should not run until the the job withjob_idhas finished successfully.Installpipinstallslurmpy--user
Package: slurmqueen
SlurmQueenA Python 3 library for automatically running experiments on a black-box tool using a Slurm cluster.
In particular, this harness allows you to:Define an experiment by declaring a list of tasks to run on a tool.Generate bash scripts that run each task.Run those bash scripts on a Slurm cluster and download the results.Analyze the results through an SQL interface.Seethis jupyter notebookfor detailed usage.InstallationFor most users, the recommended method to install is via pip:pip install slurmqueenDependencies:paramikopandaspathlib (included in Python 3.4 or higher)ipywidgets (optional)notebook (required to run the example notebook)Seerequirements.txtfor detailed version information, if needed.