Package: gpuctl
GPU Monitoring and Failure Notification/RecoveryWhile in minging crytocurrency with GPU, although the GPU has built-in thermal control,
but it might not be fitted in some environment and cause hash rate decreases or even GPU
faults.  This project provides a tool to:According the detected temperature to change fan speedExecute custom action scripts while current temperature of GPU(s) is over a defined thresholdExecute custom action scripts while current hash rate of GPU(s) is under a defined thresholdCurrently, the example of action script are based on theminer's remote managemen API, for different miner, it might be needed to customize locally.Environment and InstallationIntel CPU/16G RAM
Ubuntu 18.4/Python3Environment setupFor configuring Nvidia's GPU (ie. fan), it is needed to set the following environment.In ~/.profile and/or ~/.bashrc, add:exportDISPLAY=:0exportNO_AT_BRIDGE=1exportXAUTHORITY=/var/run/lightdm/root/:0In some shell script examples, it is needed to invoke the miner executive:exportPHOENIX_PATH=<path-to-phoenixminer-root>exportNSF_PATH=<path-to-nsfxminer-root>and also the wallet and pool address:exportETH_WALLET=<your-ethereum-wallet-address># 0xexportETH_POOL=<your-favorite-pool># ex. asia1.ethermine.org:4444Installation from PyPISetup Python vitual environment:cdgpuctl
python3-mvenvvenvsource./venv/bin/activatepipinsallgpuctlInstallation from sourcegitclonehttps://github.com/Ed-Yang/gpuctlSetup Python vitual environment:cdgpuctl
python3-mvenvvenvsource./venv/bin/activate
pipinstall-rrequirements.txtInstall gpuctl:pipinstall.After completed the above procedure, before you run the gpuctl, you only need to to run:source./venv/bin/activateInteraction with minersFor the the under rate detection feature, it need to retreive the current hashrate of miner through the network management API
of miner.The following exapmple setup a netowrk management port on TCP port 3333.ethminer/nsfminer, additional parameters:--HWMON 2 --api-port -3333or--HWMON 2 --api-bind 127.0.0.1:3333Phoenixminer, additional parameters (check section 3):-cdm 2 -cdmport 3333To test if the miner's network management function is correctly enabled:```shell
./scripts/get-rate.sh 3333
```Example of script for running PhonenixMiner and nsfminerFor utilizing the miner's script, it have to define:```
export PHOENIX_PATH=<path-to-phoenixminer-root>
export NSF_PATH=<path-to-nsfxminer-root>

export ETH_WALLET=<your-ethereum-wallet-address> # 0x
export ETH_POOL=<your-favorite-pool> # ex. asia1.ethermine.org:4444
```

pm-dev.sh <n>: start PhoenixMiner on device <n>, n is starting from zero
pm-all.sh : start PhoenixMiner all GPU devices
nsf-dev.sh <n>: start nsfminer on device <n>, n is starting from zero
nsf-all.sh : start PhoenixMiner all GPU devicesGpuCtlSome parameters are applying to every GPU on whole system (like interval, curve, etc.),
if it is necessary to provide specific setting for a GPU, it is able to run seperate
gpuctl instance with expected parametets.Usagegpuctl--helpusage:gpuctl[-h][-l][-sSLOTS][-a][-n][--intervalINTERVAL][--waitWAIT][--set-speed[0-100]][-fFAN][-dDELTA][--curveCURVE][--tempTEMP][--tasTAS][--scan][-v][-V]optionalarguments:
-h,--helpshowthishelpmessageandexit-l,--listlistallGPUcards
-sSLOTS,--slotsSLOTSusePCIslotnametolocateGPU(ie.0000:01:00.0/0000:01:00.1)-a,--amdonlyuseAMDGPU
-n,--nvidiaonlyuseNvidiaGPU
--intervalINTERVALmonitoringinterval
--waitWAITsecondsbeforereportfailure
--set-speed[0-100]setthefanspeed(0~100)-fFAN,--fanFANiftemperatureisexceedthanFANonce,activatefancontrol(default:70)-dDELTA,--deltaDELTAsetfanspeediftemperaturediff%isoverDELTA(defaut:2)--curveCURVEsettemp/fan-speedcurve(ie.0:0/10:10/80:100)--tempTEMPovertemperatureactionthreshold
--tasTASovertemperatureactionscript
--scanlistminerthroughnetworkinquiry
-v,--verboseshowdebugmessage
-V,--versionshowversioninfoSlot NameThe slot name of each GPU card could be found by using "lspci -D" command.
In the following output, the slot name of AMD GPU card is "0000:01:00.0".```shell
lspci -D
```

```shell
0000:01:00.0 VGA compatible controller....
```Action scriptsA few examples of action script are provided for reference, besides it is feasible to write a script to send syslog, email or telegram message, etc.scripts/gpu-failure.sh: shutdown system and schedule rebootscripts/restart.sh: restart minerscripts/reboot.sh: reboot rigIf a failure is detected, the gpuctl will invoke the given script with slot name as argument.ExamplesExample 1) List status of GPU cardsgpuctl--listIDSlotNameVendorPCI-IDTemp.FanPWRWorking
-------------------------------------------------------10000:01:00.0NVIDIA[10DE:1C03]54c10%73.63wTrue20000:0b:00.0AMD[1002:67DF]61c60%80.00wTrue30000:0d:00.0NVIDIA[10DE:1C03]47c0%76.18wTrueRetreive GPU's status through network management API:Note, if the miner is running as root priviledge, it must invoke with "sudo".sudogpuctl--scanMiner:nsfminer-1.3.12+,tcpport:3333,pid:10208Uptime:960s
Rate(kh)TempFan================1880058c11%2693960c60%1993348c0%Example 2) Set the fan speed of all GPUs to 50%sudogpuctl--set-speed50IDSlotNameVendorPCI-IDTemp.FanPWRWorking
--------------------------------------------------------10000:01:00.0NVIDIA[10DE:1C03]61c50%74.76wTrue20000:0b:00.0AMD[1002:67DF]78c47%81.00wTrue30000:0d:00.0NVIDIA[10DE:1C03]49c50%72.60wTrueExample 3) If the temperature of a GPU is over 65c, activate the fan speed control for the specific GPU.sudogpuctl--fan65IDSlotNameVendorPCI-IDTemp.FanPWRWorking
--------------------------------------------------------10000:01:00.0NVIDIA[10DE:1C03]60c50%76.08wTrue20000:0b:00.0AMD[1002:67DF]77c80%81.00wTrue30000:0d:00.0NVIDIA[10DE:1C03]49c50%74.41wTrue


gpuctl:started12:22:25INFOqueryintervel:5wait-time:12012:22:25INFOtemperaturethreshold:8512:22:25INFOfancontrolthreshold:6512:22:25INFOscript:None12:22:30INFO[0000:0b:00.0/AMD]currenttemp.75csetspeed84%12:22:35INFO[0000:0b:00.0/AMD]currenttemp.72csetspeed72%12:22:50INFO[0000:0b:00.0/AMD]currenttemp.76csetspeed84%12:23:00INFO[0000:0b:00.0/AMD]currenttemp.73csetspeed72%12:23:10INFO[0000:0b:00.0/AMD]currenttemp.70csetspeed72%12:23:36INFO[0000:0b:00.0/AMD]currenttemp.67csetspeed61%Example 4)nsfminerIf the temeprature of a GPU is over 75c for 10s, call action script "restart.sh" to restart minerNote, the Phoenix might not be corectly restared through network command, so this mechanism only suitable for nsfminer.Tsudogpuctl--temp75--wait10--tas./scripts/restart.shIDSlotNameVendorPCI-IDTemp.FanPWRWorking
--------------------------------------------------------10000:01:00.0NVIDIA[10DE:1C03]52c11%73.90wTrue20000:0b:00.0AMD[1002:67DF]56c60%80.00wTrue30000:0d:00.0NVIDIA[10DE:1C03]46c0%75.48wTrue


gpuctl:started12:49:00INFOqueryintervel:5wait-time:1012:49:00INFOtemperaturethreshold:7512:49:00INFOfancontrolthreshold:None12:49:00INFOscript:./scripts/restart.sh12:51:07WARNING[0000:0b:00.0/AMD]overtemperature75c,exec./scripts/restart.sh12:51:07INFOexecscript./scripts/restart.sh0000:0b:00.0no_waitFalse

IDSlotNameVendorPCI-IDTemp.FanPWRWorking
--------------------------------------------------------10000:0b:00.0AMD[1002:67DF]78c60%82.00wTrueIn the nsfminer's terminal:12:51:07minerAPI:Methodminer_restartrequested12:51:07minerRestartminers...12:51:07minerShuttingdownminers...12:51:07minerSpinningupminers...Example 5) For every GPU, if its temeprature is over 75c for 10s, call failure action scriptNote, invoking gpu-failure.sh will shutdown the system and schedulee to restart at 5 minutes later.sudogpuctl--temp75--wait10--tas./scripts/gpu-failure.shIDSlotNameVendorPCI-IDTemp.FanPWRWorking
--------------------------------------------------------10000:01:00.0NVIDIA[10DE:1C03]56c11%75.99wTrue20000:0b:00.0AMD[1002:67DF]61c60%81.00wTrue30000:0d:00.0NVIDIA[10DE:1C03]48c0%75.49wTrue


gpuctl:started13:11:17INFOqueryintervel:5wait-time:1013:11:17INFOtemperaturethreshold:7513:11:17INFOfancontrolthreshold:None13:11:17INFOscript:./scripts/gpu-failure.sh13:12:23WARNING[0000:0b:00.0/AMD]overtemperature75c,exec./scripts/gpu-failure.sh13:12:23INFOexecscript./scripts/gpu-failure.sh0000:0b:00.0no_waitFalseEthCtlUsageethctl--helpusage:ethctl[-h][-l][-bBASE][--intervalINTERVAL][-wWAIT][-tTEMP][-rRATE][--rmodeRMODE][-dDELAY][-sSCRIPT][-v][-V]optionalarguments:
-h,--helpshowthishelpmessageandexit-l,--listlistallminers
-bBASE,--baseBASEtcpportoffset,ie.deviceNislistenedonbase+N
--intervalINTERVALmonitoringinterval
-wWAIT,--waitWAITcountdowninterval
-tTEMP,--tempTEMPovertemperatureactionthreshold
-rRATE,--rateRATEunderratethreshold(default:2000kh)--rmodeRMODEfailurerestart,0:none,1:netrestart,2:kill-dDELAY,--delayDELAYdelaybeforecallingtheactionscript
-sSCRIPT,--scriptSCRIPTcallingtoscriptonfailure
-v,--verboseshowdebugmessage
-V,--versionshowversioninfoExamplesExample 1) List all miner with network management function enabledethctl--listMiner:nsfminer-1.3.12+,tcpport:3333,pid:19526Uptime:92100s
Rate(kh)TempFan================1879458c50%2662462c47%1989847c50%Example 2)PhoenixMinerif the highest of temeprature of miner's GPU(s) is over 75c,
or the lowest hashrate under 2 Mh/s call for 120s,
kill the miner's process and call action script to restart minerUse PhoenixMiner 5.3b as example:Start miner:

```shell
./scripts/pm-all.sh
```

Open a new terminal and run:

```shell
ethctl -t 75 -r 2000 --rmode 2 -d 120 -s ./scripts/pm-all.sh
```

```shell
Miner : 'PM 5.3b'   
Uptime: 0s
Rate(kh) Temp Fan  
======== ==== ==== 
       0  48c  48%


ethctl: started

23:29:52 INFO     query intervel: 5 wait-time: 120
23:29:52 INFO     temperature: 75c hashrate: 2000 kh/s
23:29:52 INFO     restart mode: 2 script: ./scripts/pm-all.sh


23:30:43 INFO     add miner 'PM 5.3b - ETH':3333 pid 27381
23:33:20 WARNING  'PM 5.3b - ETH':3333 dev 0 rate 0 under threshold
23:33:20 WARNING  'PM 5.3b - ETH':3333 dev 0 temp [72] rate [0]
23:33:20 INFO     'PM 5.3b - ETH':3333 dev 0 restarting pid 27381 mode 2 
23:33:20 INFO     'PM 5.3b - ETH':3333 miner removed
23:33:20 INFO     'PM 5.3b - ETH':3333 delay 120 exec ./scripts/pm-all.sh 0 3333 72 0
```Example 3)nsfminerif the highest of temeprature of miner's GPU(s) is over 75c,
or the lowest hashrate under 2 Mh/s call for 60s,
restart the miner's processUse PhoenixMiner 5.3b as example:Start miner:

```shell
./scripts/nsf-dev.sh 1
```

Open a new terminal and run:

```shell
ethctl -t 75 -r 2000 --rmode 2 -w 30 --delay 120 -s ./scripts/nsf-all.sh
```

```shell
Miner : 'PM 5.3b'   
Uptime: 0s
Rate(kh) Temp Fan  
======== ==== ==== 
    0  48c  48%


ethctl: started

23:29:52 INFO     query intervel: 5 wait-time: 120
23:29:52 INFO     temperature: 75c hashrate: 2000 kh/s
23:29:52 INFO     restart mode: 2 script: ./scripts/pm-all.sh


23:30:43 INFO     add miner 'PM 5.3b - ETH':3333 pid 27381
23:33:20 WARNING  'PM 5.3b - ETH':3333 dev 0 rate 0 under threshold
23:33:20 WARNING  'PM 5.3b - ETH':3333 dev 0 temp [72] rate [0]
23:33:20 INFO     'PM 5.3b - ETH':3333 dev 0 restarting pid 27381 mode 2 
23:33:20 INFO     'PM 5.3b - ETH':3333 miner removed
23:33:20 INFO     'PM 5.3b - ETH':3333 delay 120 exec ./scripts/pm-all.sh 0 3333 72 0
```Run Test Causepython3-munittestdiscovertestsDiagnosticsMonitor AMD GPU card# HiveOSsudowatch-c-n2amd-infoMonitor Nvidia GPU card# HiveOSsudowatch-c-n2nvidia-infoorsudowatch-c-n2nvidia-smiQ/Anvidia Unable to init server: Could not connect: Connection refusedIn ~/.profile and/or ~/.bashrc, add:exportDISPLAY=:0exportXAUTHORITY=/var/run/lightdm/root/:0(nvidia-settings:15781): dbind-WARNING **: 04:46:56.622....In ~/.profile and/or ~/.bashrc, add:exportNO_AT_BRIDGE=1ReferenceFan controller for amdgpusEthminer's APIGPUFanPyOpenCL SamplesAssociating OpenCL device ids with GPUs