Package: cudam
cudamCuda Mangement - multi-process, scheduled jobs, distributed processingcommand to check all cuda server statusdate>>cuda_status.txt&&echo'cuda1'>>cuda_status.txt&&sshcuda1'nvidia-smi'>>cuda_status.txt&&echo'cuda2'>>cuda_status.txt&&sshcuda2'nvidia-smi'>>cuda_status.txt&&echo'cuda3'>>cuda_status.txt&&sshcuda3'nvidia-smi'>>cuda_status.txt&&echo'cuda4'>>cuda_status.txt&&sshcuda4'nvidia-smi'>>cuda_status.txt&&echo'cuda5'>>cuda_status.txt&&sshcuda5'nvidia-smi'>>cuda_status.txt&&echo'cuda6'>>cuda_status.txt&&sshcuda6'nvidia-smi'>>cuda_status.txt&&echo'cuda11'>>cuda_status.txt&&sshcuda11'nvidia-smi'>>cuda_status.txtserver-client mode to utilize multi-GPUs across Multi-Machinesserver side - develop the code that runs on a single GPU# here is a dumb function to evaluate densenet# it should be replaced by the actual code of evaluationdefevaluate_densenet(model):acc=0.99returnaccclient size - develop the code to send the models to server for evaluationAdd available GPU servers in the server list configuration file# configuration of server list
cuda4,8000
cuda4,8001
cuda5,8000
cuda5,8001
cuda5,8002The client code that concurrently evaluates modelsfromcudam.cudam_socket.clientimportGPUClientPoolDEFAULT_RUN_CODE_WORK_DIRECTORY="/home/www/server"# the folder where the server side code residesDEFAULT_RUN_CODE_PATH="server_file"# the file name of the server side codeSERVER_LIST_CONFIG='config/server_list.txt'# the configuration file of the server listdefpool_evaluate_densenet(model_list):# generat the arguments which will passed to client poolarr_args=[]forminmodel_list:singe_args={'model':m}arr_args.append({'path':DEFAULT_RUN_CODE_PATH,'entry':"evaluate_densenet",'work_directory':DEFAULT_RUN_CODE_WORK_DIRECTORY,'args':singe_args,'use_cuda':True# whether to use GPU or not})# init client poolserver_list=GPUClientPool.load_server_list_from_file(SERVER_LIST_CONFIG)pool=GPUClientPool(server_list)# perform evaluationeval_result=pool.run_code_batch(arr_args)returneval_result# main entranceif__name__=='__main__':model_list=[]# dumb model list which needs to be replaced by real modelspool_evaluate_densenet(model_list)start the serverAfter installation of this package,cudam_server.pyshould be automatically copied to the bin path; if not, please manually copy this file to the root folder of the project. The server can be started by running the following command:nohuppythoncudam_server.py-s1-icuda1-p8000-g0>&log/nohup_cuda_1_8000_0.log&run the client side python code to evaluate a batch of modelstask managertask template#!/usr/bin/env bashwhilegetoptsg:option;docase"${option}"ing)GPU_ID=${OPTARG};;esacdoneprint_help(){printf"Parameter g(GPU ID) is mandatory\n"printf"g values - GPU ID"exit1}if[-z"${GPU_ID}"];thenprint_helpfiecho"start task on GPU:$GPU_ID"# the root directory of your python scriptcd~/code/psocnn/# the main python script accepting the gpu ID in -g argumentpython3main.py-g${GPU_ID}task folder structuretask manager# start task managernohupcudam_task_manager.py-n2-s2-i60-f300&# snap gpucudam_snap_gpu.py-s2-l60-g1install cumdam for a specific user and can not add the local path into executable PATHSwitch to the root folder of your projectInstall cudam packagepipinstall--usercudamCreate a soft link of the executable fileln-s/home/{YOURUSER}/.local/bin/cudam_task_manager.pycudam_task_manager.py
ln-s/home/{YOURUSER}/.local/bin/cudam_snap_gpu.pycudam_snap_gpu.pyRun the task manager# run interactivelypythoncudam_task_manager.py-n2-s2-i60-f300# run in backgroundnohuppythoncudam_task_manager.py-n2-s2-i60-f300&