Repository: microsoft/nni

Issue:
failed to train with remote mode
hello, everyone,
      I used the experiment examples/trials/weight_sharing/ga_squad/config_remote.yml, and I modified the machineList with my own machine, but the experiment was waiting and the log is below. Is there any suggestion?
```
[9/29/2019, 4:40:41 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:41 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:44 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:44 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:47 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:47 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:50 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:50 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:53 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:53 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:56 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:56 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:59 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:59 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:02 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:02 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:05 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:05 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:08 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:08 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:11 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:11 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:14 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:14 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:17 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:17 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:20 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:20 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:23 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:23 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:26 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:26 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:29 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:29 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:32 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:32 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:35 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
```


Discussion:
Can you check if `nvidia-smi` is available on your machine?
Hello @justttry
The log shows that you might have enough GPU resource to run this code. If you want to run in CPU, you can change the setting in `config.yml`.
@martsalz nvidia-smi is available and the print is below. I want to use the GPUs, but it seems to find no GPUs. @xuehui1991 

> +-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+====================
|   0  Tesla P40           On   | 00000000:04:00.0 Off |                    0 |
| N/A   26C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           On   | 00000000:05:00.0 Off |                    0 |
| N/A   24C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           On   | 00000000:08:00.0 Off |                    0 |
| N/A   26C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P40           On   | 00000000:09:00.0 Off |                    0 |
| N/A   28C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla P40           On   | 00000000:85:00.0 Off |                    0 |
| N/A   29C    P8    17W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla P40           On   | 00000000:86:00.0 Off |                    0 |
| N/A   29C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla P40           On   | 00000000:89:00.0 Off |                    0 |
| N/A   28C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla P40           On   | 00000000:8A:00.0 Off |                    0 |
| N/A   29C    P8    16W / 250W |      0MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
Do other applications/users use the GPUs in parallel? If yes, please specify `useActiveGpu=true` in the `config.yml`. By default, NNI will use the GPU only if there is no another active process in the GPU. See:

https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/ExperimentConfig.md

Please show the `config.yml`
I had the same problem...
@justttry Would you please paste the config.yml as @martsalz  mentioned.  
There is a similar issue #1852 which has already been fixed, please check it.
Hi @chicm-ms,
I also met the same problem. I followed the instructions in #1852 , but got no luck.  
1) There is no such file 'tmp/root/nni/scripts/gpu_metrics'  in my remote machine although I can generate it manually with `bash -c echo $$ > /tmp/root/nni/scripts/pid ; METRIC_OUTPUT_DIR=/tmp/root/nni/scripts python3 -m nni_gpu_tool.gpu_metrics_collector`.  
2) I also tried  'rm -rf /tmp/root' and 'rm -rf /tmp/nni'
3) nvidia-smi works fine in my remote machines.
4) I ran the nnictl command in my local machine.

nni Environment:

nni version: 1.4
nni mode(local|pai|remote): remote
OS: linux
python version: 3.7.3
is conda or virtualenv used?: yes
is running in docker?: yes

The config.yml and nnimanager.log are attached as follows

 config.yml 

`authorName: default
debug: true
experimentName: example_cifar_pytorch
trialConcurrency: 2
maxExecDuration: 1h
maxTrialNum: 128
#choice: local, remote, pai
trainingServicePlatform: remote
searchSpacePath: search_space.json
#choice: true, false
useAnnotation: false
tuner:
    builtinTunerName: TPE
    classArgs:
        optimize_mode: maximize

trial:
  command: python3 cifar.py
  codeDir: .
  gpuNum: 1

machineList:
    - ip: 10.22.228.140
      username: root
      port: 9003
      sshKeyPath: /root/.ssh/id_rsa
    - ip: 10.22.226.194
      username: root
      port: 9000
      sshKeyPath: /root/.ssh/id_rsa`

 nnimanager.log 
`[3/11/2020, 10:17:16 AM] INFO [ 'NNIManager setClusterMetadata, key: machine_list, value: [{"ip":"10.22.228.140","username":"root","port":9003,"sshKeyPath":"/root/.ssh/id_rsa"},{"ip":"10.22.226.194","username":"root","port":9000,"sshKeyPath":"/root/.ssh/id_rsa"}]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'Connecting to remote machines: [{"ip":"10.22.228.140","username":"root","port":9003,"sshKeyPath":"/root/.ssh/id_rsa"},{"ip":"10.22.226.194","username":"root","port":9000,"sshKeyPath":"/root/.ssh/id_rsa"}]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [mkdir -p /tmp/nni/experiments/oxNT5WxA]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [mkdir -p /tmp/nni/experiments/oxNT5WxA]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [(umask 0 ; mkdir -p /tmp/root/nni/scripts)]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [chmod 777 /tmp/nni /tmp/nni/* /tmp/nni/scripts/*]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [bash -c \'echo $$ > /tmp/root/nni/scripts/pid ; METRIC_OUTPUT_DIR=/tmp/root/nni/scripts python3 -m nni_gpu_tool.gpu_metrics_collector\']' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [(umask 0 ; mkdir -p /tmp/root/nni/scripts)]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [chmod 777 /tmp/nni /tmp/nni/* /tmp/nni/scripts/*]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [bash -c \'echo $$ > /tmp/root/nni/scripts/pid ; METRIC_OUTPUT_DIR=/tmp/root/nni/scripts python3 -m nni_gpu_tool.gpu_metrics_collector\']' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'PUT: /experiment/cluster-metadata: body:\n{\n    "trial_config": {\n        "command": "python3 cifar.py",\n        "codeDir": "/mnt/cephfs_new_wj/cv/wangxing/source_code/bytennf/examples/trials/cifar-rcnsa-advisor/.",\n        "gpuNum": 1\n    }\n}' ]
[3/11/2020, 10:17:16 AM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python3 cifar.py","codeDir":"/mnt/cephfs_new_wj/cv/wangxing/source_code/bytennf/examples/trials/cifar-rcnsa-advisor/.","gpuNum":1}' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'POST: /experiment: body:\n{\n    "authorName": "default",\n    "experimentName": "example_cifar_pytorch",\n    "trialConcurrency": 2,\n    "maxExecDuration": 3600,\n    "maxTrialNum": 128,\n    "searchSpace": "{\\"in_channel_1\\": {\\"_type\\": \\"choice\\", \\"_value\\": [8, 16, 24, 32]}, \\"in_channel_2\\": {\\"_type\\": \\"choice\\", \\"_value\\": [8, 16, 24, 32]}, \\"in_channel_3\\": {\\"_type\\": \\"choice\\", \\"_value\\": [8, 16, 24, 32]}, \\"in_channel_4\\": {\\"_type\\": \\"choice\\", \\"_value\\": [8, 16, 24, 32]}, \\"hidden_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [128, 256, 512, 1024]}}",\n    "trainingServicePlatform": "remote",\n    "tuner": {\n        "builtinTunerName": "TPE",\n        "classArgs": {\n            "optimize_mode": "maximize"\n        }\n    },\n    "versionCheck": false,\n    "clusterMetaData": [\n        {\n            "key": "machine_list",\n            "value": [\n                {\n                    "ip": "10.22.228.140",\n                    "username": "root",\n                    "port": 9003,\n                    "sshKeyPath": "/root/.ssh/id_rsa"\n                },\n                {\n                    "ip": "10.22.226.194",\n                    "username": "root",\n                    "port": 9000,\n                    "sshKeyPath": "/root/.ssh/id_rsa"\n                }\n            ]\n        },\n        {\n            "key": "trial_config",\n            "value": {\n                "command": "python3 cifar.py",\n                "codeDir": "/mnt/cephfs_new_wj/cv/wangxing/source_code/bytennf/examples/trials/cifar-rcnsa-advisor/.",\n                "gpuNum": 1\n            }\n        }\n    ]\n}' ]
[3/11/2020, 10:17:16 AM] INFO [ 'Starting experiment: oxNT5WxA' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'Setup tuner...' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'dispatcher command: python3 -m nni --exp_params eyJhdXRob3JOYW1lIjoiZGVmYXVsdCIsImV4cGVyaW1lbnROYW1lIjoiZXhhbXBsZV9jaWZhcl9weXRvcmNoIiwidHJpYWxDb25jdXJyZW5jeSI6MiwibWF4RXhlY0R1cmF0aW9uIjozNjAwLCJtYXhUcmlhbE51bSI6MTI4LCJ0cmFpbmluZ1NlcnZpY2VQbGF0Zm9ybSI6InJlbW90ZSIsInR1bmVyIjp7ImJ1aWx0aW5UdW5lck5hbWUiOiJUUEUiLCJjbGFzc0FyZ3MiOnsib3B0aW1pemVfbW9kZSI6Im1heGltaXplIn19LCJ2ZXJzaW9uQ2hlY2siOmZhbHNlLCJjbHVzdGVyTWV0YURhdGEiOlt7ImtleSI6Im1hY2hpbmVfbGlzdCIsInZhbHVlIjpbeyJpcCI6IjEwLjIyLjIyOC4xNDAiLCJ1c2VybmFtZSI6InJvb3QiLCJwb3J0Ijo5MDAzLCJzc2hLZXlQYXRoIjoiL3Jvb3QvLnNzaC9pZF9yc2EifSx7ImlwIjoiMTAuMjIuMjI2LjE5NCIsInVzZXJuYW1lIjoicm9vdCIsInBvcnQiOjkwMDAsInNzaEtleVBhdGgiOiIvcm9vdC8uc3NoL2lkX3JzYSJ9XX0seyJrZXkiOiJ0cmlhbF9jb25maWciLCJ2YWx1ZSI6eyJjb21tYW5kIjoicHl0aG9uMyBjaWZhci5weSIsImNvZGVEaXIiOiIvbW50L2NlcGhmc19uZXdfd2ovY3Yvd2FuZ3hpbmcvc291cmNlX2NvZGUvYnl0ZW5uZi9leGFtcGxlcy90cmlhbHMvY2lmYXItcmNuc2EtYWR2aXNvci8uIiwiZ3B1TnVtIjoxfX1dfQ==' ]
[3/11/2020, 10:17:16 AM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[3/11/2020, 10:17:16 AM] INFO [ 'Add event listeners' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'Send tuner command: INITIALIZE: {"in_channel_1": {"_type": "choice", "_value": [8, 16, 24, 32]}, "in_channel_2": {"_type": "choice", "_value": [8, 16, 24, 32]}, "in_channel_3": {"_type": "choice", "_value": [8, 16, 24, 32]}, "in_channel_4": {"_type": "choice", "_value": [8, 16, 24, 32]}, "hidden_size": {"_type": "choice", "_value": [128, 256, 512, 1024]}}' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'ipcInterface command type: [IN], content:[{"in_channel_1": {"_type": "choice", "_value": [8, 16, 24, 32]}, "in_channel_2": {"_type": "choice", "_value": [8, 16, 24, 32]}, "in_channel_3": {"_type": "choice", "_value": [8, 16, 24, 32]}, "in_channel_4": {"_type": "choice", "_value": [8, 16, 24, 32]}, "hidden_size": {"_type": "choice", "_value": [128, 256, 512, 1024]}}]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[3/11/2020, 10:17:16 AM] INFO [ 'RestServer start' ]
[3/11/2020, 10:17:16 AM] INFO [ 'RestServer base port is 6667' ]
[3/11/2020, 10:17:16 AM] INFO [ 'Run remote machine training service.' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:16 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:17 AM] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[3/11/2020, 10:17:17 AM] DEBUG [ 'ipcInterface command type: [GE], content:[2]' ]
[3/11/2020, 10:17:17 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"in_channel_1": 24, "in_channel_2": 32, "in_channel_3": 8, "in_channel_4": 24, "hidden_size": 128}, "parameter_index": 0}' ]
[3/11/2020, 10:17:17 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"in_channel_1": 16, "in_channel_2": 24, "in_channel_3": 16, "in_channel_4": 32, "hidden_size": 512}, "parameter_index": 0}' ]
[3/11/2020, 10:17:17 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:17 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:18 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:18 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:19 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:19 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:20 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:20 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:21 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[3/11/2020, 10:17:21 AM] INFO [ 'submitTrialJob: form: {"sequenceId":0,"hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"in_channel_1\\": 24, \\"in_channel_2\\": 32, \\"in_channel_3\\": 8, \\"in_channel_4\\": 24, \\"hidden_size\\": 128}, \\"parameter_index\\": 0}","index":0}}' ]
[3/11/2020, 10:17:21 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"in_channel_1": 24, "in_channel_2": 32, "in_channel_3": 8, "in_channel_4": 24, "hidden_size": 128}, "parameter_index": 0}, jobDetail: {"id":"P9vW6","status":"WAITING","submitTime":1583921841578,"workingDirectory":"/tmp/nni/experiments/oxNT5WxA/trials/P9vW6","form":{"sequenceId":0,"hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"in_channel_1\\": 24, \\"in_channel_2\\": 32, \\"in_channel_3\\": 8, \\"in_channel_4\\": 24, \\"hidden_size\\": 128}, \\"parameter_index\\": 0}","index":0}},"tags":[],"gpuIndices":[]}' ]
[3/11/2020, 10:17:21 AM] INFO [ 'submitTrialJob: form: {"sequenceId":1,"hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"in_channel_1\\": 16, \\"in_channel_2\\": 24, \\"in_channel_3\\": 16, \\"in_channel_4\\": 32, \\"hidden_size\\": 512}, \\"parameter_index\\": 0}","index":0}}' ]
[3/11/2020, 10:17:21 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"in_channel_1": 16, "in_channel_2": 24, "in_channel_3": 16, "in_channel_4": 32, "hidden_size": 512}, "parameter_index": 0}, jobDetail: {"id":"Cbna8","status":"WAITING","submitTime":1583921841582,"workingDirectory":"/tmp/nni/experiments/oxNT5WxA/trials/Cbna8","form":{"sequenceId":1,"hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"in_channel_1\\": 16, \\"in_channel_2\\": 24, \\"in_channel_3\\": 16, \\"in_channel_4\\": 32, \\"hidden_size\\": 512}, \\"parameter_index\\": 0}","index":0}},"tags":[],"gpuIndices":[]}' ]
[3/11/2020, 10:17:21 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:21 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:22 AM] WARNING [ 'Scheduler: trialJob id P9vW6, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[3/11/2020, 10:17:22 AM] INFO [ 'Right now no available GPU can be allocated for trial P9vW6, will try to schedule later' ]
[3/11/2020, 10:17:22 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:22 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:23 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:23 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:24 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:24 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:25 AM] WARNING [ 'Scheduler: trialJob id P9vW6, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[3/11/2020, 10:17:25 AM] INFO [ 'Right now no available GPU can be allocated for trial P9vW6, will try to schedule later' ]
[3/11/2020, 10:17:25 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:25 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:26 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[3/11/2020, 10:17:26 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:26 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:27 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:27 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:28 AM] WARNING [ 'Scheduler: trialJob id P9vW6, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[3/11/2020, 10:17:28 AM] INFO [ 'Right now no available GPU can be allocated for trial P9vW6, will try to schedule later' ]
[3/11/2020, 10:17:28 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:28 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:29 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:29 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:30 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:30 AM] DEBUG [ 'remoteExeCommand: command: [tail -n 1 /tmp/root/nni/scripts/gpu_metrics]' ]
[3/11/2020, 10:17:31 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[3/11/2020, 10:17:31 AM] WARNING [ 'Scheduler: trialJob id P9vW6, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[3/11/2020, 10:17:31 AM] INFO [ 'Right now no available GPU can be allocated for trial P9vW6, will try to schedule later' ]
`
@xingwangsfu Are your remote machines in docker containers?  
Would you please check following?
1. nni version on remote remote machine
2. Whether  process `nni_gpu_tool.gpu_metrics_collector` is running on remote machine before and after nni is started. If the process exist before nni is started, you can kill it and then restart nni experiment.
3. Whether /tmp/root/nni/scripts diretory and /tmp/root/nni/scripts/pid  file exist on remote machine if you do not generate gpu metrics manually while nni is started.
Update on this @chicm-ms :
I have to run nni in conda on remote machine. But the remote mode always called the '/usr/bin/python3' to generate gpu_metric file. That's why the problem happened. To solve this problem, I also installed nni with '/usr/bin/python3'. Then, the error disappeared. I think this problem is related to #1712 . 

Is there a way to force nni to use the user specified python3 to generate gpu_metric instead of '/usr/bin/python3' on the remote machine?
> Update on this @chicm-ms :
> I have to run nni in conda on remote machine. But the remote mode always called the '/usr/bin/python3' to generate gpu_metric file. That's why the problem happened. To solve this problem, I also installed nni with '/usr/bin/python3'. Then, the error disappeared. I think this problem is related to #1712 .
> 
> Is there a way to force nni to use the user specified python3 to generate gpu_metric instead of '/usr/bin/python3' on the remote machine?

@chicm-ms @QuanluZhang @SparkSnail - this sounds a fair ask =).
@scarlett2018  Yes, I also think this is a reasonable requirement, currently the python executable is hard coded as `python3`: 
https://github.com/microsoft/nni/blob/master/src/nni_manager/training_service/common/util.ts#L205



Resolving patch:
diff --git a/tools/nni_gpu_tool/gpu_metrics_collector.py b/tools/nni_gpu_tool/gpu_metrics_collector.py
--- a/tools/nni_gpu_tool/gpu_metrics_collector.py
+++ b/tools/nni_gpu_tool/gpu_metrics_collector.py
@@ -10,27 +10,31 @@
 
 from xml.dom import minidom
 
+
 def check_ready_to_run():
     if sys.platform == 'win32':
         pgrep_output = subprocess.check_output(
             'wmic process where "CommandLine like \'%nni_gpu_tool.gpu_metrics_collector%\' and name like \'%python%\'" get processId')
         pidList = pgrep_output.decode("utf-8").strip().split()
-        pidList.pop(0) # remove the key word 'ProcessId'
+        pidList.pop(0)  # remove the key word 'ProcessId'
         pidList = list(map(int, pidList))
         pidList.remove(os.getpid())
         return not pidList
     else:
-        pgrep_output = subprocess.check_output('pgrep -fxu "$(whoami)" \'python3 -m nni_gpu_tool.gpu_metrics_collector\'', shell=True)
+        pgrep_output = subprocess.check_output('pgrep -afu "$(whoami)" \'python3 -m nni_gpu_tool.gpu_metrics_collector\'', shell=True)
         pidList = []
         for pid in pgrep_output.splitlines():
-            pidList.append(int(pid))
-        pidList.remove(os.getpid())
+            pid = pid.decode()
+            if "pgrep " in pid or pid.startswith('%s ' % os.getpid()) or pid.startswith('%s ' % os.getppid()):
+                continue
+            pidList.append(pid)
         return not pidList
 
+
 def main(argv):
     metrics_output_dir = os.environ['METRIC_OUTPUT_DIR']
     if check_ready_to_run() == False:
-        # GPU metrics collector is already running. Exit
+        print("GPU metrics collector is already running. exiting...")
         exit(2)
     cmd = 'nvidia-smi -q -x'.split()
     while(True):
@@ -44,6 +48,7 @@ def main(argv):
         # TODO: change to sleep time configurable via arguments
         time.sleep(5)
 
+
 def parse_nvidia_smi_result(smi, outputDir):
     try:
         old_umask = os.umask(0)
@@ -70,13 +75,14 @@ def parse_nvidia_smi_result(smi, outputDir):
                 outPut["gpuInfos"].append(gpuInfo)
             print(outPut)
             outputFile.write("{}\n".format(json.dumps(outPut, sort_keys=True)))
-            outputFile.flush();
-    except:
+            outputFile.flush()
+    except Exception as error:
         # e_info = sys.exc_info()
-        print('xmldoc paring error')
+        print('gpu_metrics_collector error: %s' % error)
     finally:
         os.umask(old_umask)
 
+
 def gen_empty_gpu_metric(outputDir):
     try:
         old_umask = os.umask(0)


Test patch: