Package: gpu-watchmen
WatchmenA simple and easy-to-use toolkit for GPU scheduling.DependenciesPython>= 3.6requests >= 2.24.0pydantic >= 1.7.1gpustat >= 0.6.0flask >= 1.1.2apscheduler >= 3.6.3InstallationInstall dependencies.$pipinstall-rrequirements.txtInstall watchmen.Install from source code:$pipinstall-e.Or you can install the stable version package from pypi.$pipinstallgpu-watchmen-ihttps://pypi.org/simpleQuick StartStart the serverThe default port of the server is62333$python-mwatchmen.serverIf you want the server to be running backend, try:$nohuppython-mwatchmen.server1>watchmen.log2>&1&There are some configurations for the serverusage: server.py [-h] [--host HOST] [--port PORT]
                 [--queue_timeout QUEUE_TIMEOUT]
                 [--request_interval REQUEST_INTERVAL]
                 [--status_queue_keep_time STATUS_QUEUE_KEEP_TIME]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           host address for api server
  --port PORT           port for api server
  --queue_timeout QUEUE_TIMEOUT
                        timeout for queue waiting (seconds)
  --request_interval REQUEST_INTERVAL
                        interval for gpu status requesting (seconds)
  --status_queue_keep_time STATUS_QUEUE_KEEP_TIME
                        hours for keeping the client status. set `-1` to keep
                        all clients' statusModify the source code in your project:fromwatchmenimportWatchClientclient=WatchClient(id="short description of this running",gpus=[1],server_host="127.0.0.1",server_port=62333)client.wait()When the program goes on afterclient.wait(), you are in the working queue.
Watchmen supports two requesting mode:queuemode means you are waiting for the gpus ingpusarguments.schedulemode means you are waiting for the server to sparereq_gpu_numof available GPUs ingpus.
You can check examples inexample/for further reading.# single card queue mode$cdexample&&pythonsingle_card_mnist.py--id="single"--cuda=0--wait# single card schedule mode$cdexample&&pythonsingle_card_mnist.py--id="single schedule"--cuda=0,2,3--req_gpu_num=1--wait_mode="schedule"--wait# queue mode$cdexample&&pythonmulti_card_mnist.py--id="multi"--cuda=2,3--wait# schedule mode$cdexample&&pythonmulti_card_mnist.py--id='multi card scheduling wait'--cuda=1,0,3--req_gpu_num=2--wait="schedule"Check the queue in browser.Open the following link to your browser:http://<server ip address>:<server port>, for example:http://192.168.126.143:62333.And you can get a result like the demo below.
Please be aware that the page is not going to change dynamically, so you can refresh the page manually to check the latest status.Home page: GPU statusWorking queue:Finished queue:Reminder when program is finished.watchmenalso support email and other kinds of reminders for message informing.
For example, you can send yourself an email when the program is finished.fromwatchmen.reminderimportsend_email...# your code heresend_email(host="smtp.163.com",# email host to login, like `smtp.163.com`port=25,# email port to login, like `25`user="***@163.com",# user email address for login, like `***@163.com`password="***",# password or auth code for loginreceiver="***@outlook.com",# receiver email addresshtml_message="<h1>Your program is finished!</h1>",# content, html format supportedsubject="Proram Finished Notice"# email subject)To get more reminders, please checkwatchmen/reminder.py.UPDATEv0.3.8: changeOKstatus to be shown only in the finished queue, and showreadyin the working queue. Fix severe bug when schedulingv0.3.7: much faster due to lock free changes! fix timeout and schedule bugv0.3.6: fix front-end api hostname bugv0.3.5: fix front-end api port bugv0.3.4: refreshed interface, addregister_timefield, fixcheck_finishedbugv0.3.3: fixcheck_finishedbug in server end, quit the main thread if the sub-thread is quit, and remove the backend cmd in the main threadv0.3.2: fixWatchClientbugv0.3.1: changeClientintoWatchClient, fixClientCollectionandsend_emailbugv0.3.0: support gpu scheduling, fix blank input output, fixcheck_gpus_existencev0.2.2: fix html package data, add multi-card exampleTODOimport user authentication modules to help the working queue delete operationsread programs' pids to help reading program working status and kill tasks remotelytest and support distributed model parallel configurations (withpython -m torch.distributed.launch)prettify the web page and divide functions into different tabsgpu using stats for each user and processquit the main thread if the sub-thread is quitchangeClientintoWatchClient, in case of any ambiguityClientCollection/__contains__function should not includefinished_queue, to help theidreleasessubject bug inreminder/send_email()add schedule feature, so clients only have to request for a number and range of gpus, and the server will assign the gpu num to clientsadd remindersadd webui html supportadd examples