Repository: microsoft/nni

Issue:
Error: Dispatcher stream error, tuner may have crashed. When using PolicyBasedRL to do nas and set concurrency>1
**Hello guys, i am using PolicyBasedRL strategy to do multi trial NAS experiments, when I set concurrency=2, the tuner crased, and concurrency=1 works just fine. Here is the error message**:
```
[2022-09-08 02:09:28] ERROR: 1
Traceback (most recent call last):
  File "/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni/runtime/msg_dispatcher_base.py", line 97, in command_queue_worker
    self.process_command(command, data)
  File "/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni/runtime/msg_dispatcher_base.py", line 143, in process_command
    command_handlers[command](data)
  File "/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni/nas/execution/common/integration.py", line 223, in handle_report_metric_data
    self.invoke_callback('intermediate_metric', data['parameter_id'], self._process_value(data['value']))
  File "/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni/nas/execution/common/integration.py", line 119, in invoke_callback
    self.process_queued_callbacks()
  File "/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni/nas/execution/common/integration.py", line 104, in process_queued_callbacks
    self.intermediate_metric_callback(*call_args)  # pylint: disable=not-callable
  File "/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni/nas/execution/pytorch/graph.py", line 130, in _intermediate_metric_callback
    model = self._running_models[trial_id]
KeyError: 1
[2022-09-08 02:09:29] Dispatcher exiting...
[2022-09-08 02:09:30] Dispatcher terminiated
Error: Dispatcher stream error, tuner may have crashed.
    at EventEmitter.<anonymous> (/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni_node/core/nnimanager.js:647:32)
    at EventEmitter.emit (node:events:526:28)
    at WebSocketChannelImpl.handleError (/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:107:22)
    at WebSocket.<anonymous> (/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:41:37)
    at WebSocket.emit (node:events:538:35)
    at WebSocket.emitClose (/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:246:10)
    at Socket.socketOnClose (/home/liangkai/miniconda3/envs/camp/lib/python3.8/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:1127:15)
    at Socket.emit (node:events:526:28)
    at TCP.<anonymous> (node:net:687:12)
```


**Environment**:
- NNI version:2.9/2.8
- Training service (local):
- Client OS: 
- Server OS (for remote mode [only):](url)
- Python version: 3.8
- PyTorch/TensorFlow version: pytorch 1.21.1
- Is conda/virtualenv/venv used?: conda
- Is running in Docker?: no


**Configuration**:
 - Experiment config (remember to remove secrets!):
 - Search space:


**Log message**:

 - nnimanager.log: [nnimanager.log](https://github.com/microsoft/nni/files/9522943/nnimanager.log)

 - dispatcher.log: [dispatcher.log](https://github.com/microsoft/nni/files/9522946/dispatcher.log)
 - nnictl stdout and stderr:
 


<!--
Where can you find the log files:
LOG: https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/HowToDebug.md#experiment-root-director
STDOUT/STDERR: https://nni.readthedocs.io/en/stable/reference/nnictl.html#nnictl-log-stdout
-->



Discussion:
I think a possible problem might be that your trial sends intermediate results too fast, before the trial is even registered in the python end. Would you try to reduce the intermediate result frequency, or try to wait for a couple of seconds before sending the first intermediate result?
time.sleep(5) before report intermediate result doesn't help. When using Random strategy concurrency=2 works just fine, so i think there is something wrong with the PolicyBasedRL strategy
I suspect there might be some kind of racing conditions in the RL implementation.

If you have access to NNI code (which might be found in your site-packages), try to move `submit_models` into the thread lock and see whether that helps.

![image](https://user-images.githubusercontent.com/8463288/189290643-949a63ad-4d8d-4ca8-a6b8-08bdab19743f.png)

Great！Problem solved, thank you!
annother problem is this model describtion with PolicyBasedRL strategy, its a little hard to read ^_^
![image](https://user-images.githubusercontent.com/9530526/189298019-a0329c02-f920-492d-848d-9782e69a587a.png)
BTW. is there any method to load a well trained strategy? Like ENAS NASNet doing, when finish searching, man can load the controller and sample 100 archs and evaluate them.

If what you want is a one-shot strategy like ENAS here, you should use `strategy.ENAS`. PolicyBasedRL is a multi-trial strategy and no weights are shared at all.
it's nothing about weights sharing, i mean that the controller should could be called many times to sample new archittectures no matter its one-shot experiment or multi trial experiment.
IIUC, what you want here is a checkpoint of the strategy, which you can reload and test what it has learned. Otherwise, you would only need to lengthen the search and take the last 100 trials as the final samples.

Checkpoint is a nice feature, which we have been actively working on. In v2.9, we just supported checkpointing for RandomOneShot strategy. We will support more strategies in future.
Thank you for your advice and your hard work👍
________________________________
发件人: Yuge Zhang ***@***.***>
发送时间: Saturday, September 10, 2022 10:23:05 AM
收件人: microsoft/nni ***@***.***>
抄送: Kai Liang ***@***.***>; Manual ***@***.***>
主题: Re: [microsoft/nni] Error: Dispatcher stream error, tuner may have crashed. When using PolicyBasedRL to do nas and set concurrency>1 (Issue #5117)


IIUC, what you want here is a checkpoint of the strategy, which you can reload and test what it has learned. Otherwise, you would only need to lengthen the search and take the last 100 trials as the final samples.

Checkpoint is a nice feature, which we have been actively working on. In v2.9, we just supported checkpointing for RandomOneShot strategy. We will support more strategies in future.

—
Reply to this email directly, view it on GitHub<https://github.com/microsoft/nni/issues/5117#issuecomment-1242597565>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACIWZHWILXBT22JJGFMCLALV5PWITANCNFSM6AAAAAAQHIZQWY>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


Resolving patch:
diff --git a/nni/nas/strategy/_rl_impl.py b/nni/nas/strategy/_rl_impl.py
--- a/nni/nas/strategy/_rl_impl.py
+++ b/nni/nas/strategy/_rl_impl.py
@@ -136,8 +136,8 @@ def step(self, action: int) -> Tuple[ObservationType, float, bool, dict]:
         if self.cur_step == self.num_steps:
             with _thread_lock:
                 model = get_targeted_model(self.base_model, self.mutators, self.sample)
-            _logger.info(f'New model created: {self.sample}')
-            submit_models(model)
+                _logger.info(f'New model created: {self.sample}')
+                submit_models(model)
             wait_models(model)
             if model.status == ModelStatus.Failed:
                 return self.reset(), 0., False, {}


Test patch: