Cannot sample n_sample > prob_dist.size(-1) samples without replacement

username: antreas97
cooked: <p>Hello. I am trying to use imitation learning. However when I try to enter the recorded .demo file the following error, related to torch, appears. Thanks.</p>
<p>RuntimeError: cannot sample n_sample &gt; prob_dist.size(-1) samples without replacement</p>
<p>ERROR LOG BELOW<br>
–‐----------------------------<br>
Version information:<br>
ml-agents: 0.27.0,<br>
ml-agents-envs: 0.27.0,<br>
Communicator API: 1.5.0,<br>
PyTorch: 1.9.0+cu111<br>
[INFO] Listening on port 5004. Start training by pressing the Play button in the Unity Editor.<br>
[INFO] Connected to Unity environment with package version 2.0.0-pre.3 and communication version 1.5.0<br>
[INFO] Connected new brain: ZARAgoal?team=1<br>
[WARNING] Deleting TensorBoard data events.out.tfevents.1624485266.AndreasPC.18212.0 that was left over from a previous run.<br>
[INFO] Hyperparameters for behavior name ZARAgoal:<br>
trainer_type:   ppo<br>
hyperparameters:<br>
batch_size:   128<br>
buffer_size:  2048<br>
learning_rate:        0.0003<br>
beta: 0.01<br>
epsilon:      0.2<br>
lambd:        0.95<br>
num_epoch:    3<br>
learning_rate_schedule:       linear<br>
network_settings:<br>
normalize:    False<br>
hidden_units: 256<br>
num_layers:   2<br>
vis_encode_type:      simple<br>
memory:       None<br>
goal_conditioning_type:       hyper<br>
reward_signals:<br>
extrinsic:<br>
gamma:      0.99<br>
strength:   1.0<br>
network_settings:<br>
normalize:        False<br>
hidden_units:     128<br>
num_layers:       2<br>
vis_encode_type:  simple<br>
memory:   None<br>
goal_conditioning_type:   hyper<br>
gail:<br>
gamma:      0.99<br>
strength:   0.01<br>
network_settings:<br>
normalize:        False<br>
hidden_units:     128<br>
num_layers:       2<br>
vis_encode_type:  simple<br>
memory:   None<br>
goal_conditioning_type:   hyper<br>
learning_rate:      0.0003<br>
encoding_size:      None<br>
use_actions:        False<br>
use_vail:   False<br>
demo_path:  Demos/ZARAdemos/<br>
init_path:      None<br>
keep_checkpoints:       5<br>
checkpoint_interval:    500000<br>
max_steps:      100000<br>
time_horizon:   64<br>
summary_freq:   60000<br>
threaded:       False<br>
self_play:      None<br>
behavioral_cloning:<br>
demo_path:    Demos/ZARAdemos/<br>
steps:        50000<br>
strength:     1.0<br>
samples_per_update:   0<br>
num_epoch:    None<br>
batch_size:   None<br>
d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\nn\init.py:388: UserWarning: Initializing zero-element tensors is a no-op<br>
warnings.warn(“Initializing zero-element tensors is a no-op”)<br>
d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\nn\init.py:426: UserWarning: Initializing zero-element tensors is a no-op<br>
warnings.warn(“Initializing zero-element tensors is a no-op”)<br>
Traceback (most recent call last):<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\trainer_controller.py”, line 176, in start_learning<br>
n_steps = self.advance(env_manager)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\trainer_controller.py”, line 234, in advance<br>
new_step_infos = env_manager.get_steps()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\env_manager.py”, line 124, in get_steps<br>
new_step_infos = self._step()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\subprocess_env_manager.py”, line 298, in _step<br>
self._queue_steps()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\subprocess_env_manager.py”, line 291, in _queue_steps<br>
env_action_info = self._take_step(env_worker.previous_step)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\subprocess_env_manager.py”, line 429, in _take_step<br>
all_action_info[brain_name] = self.policies[brain_name].get_action(<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\policy\torch_policy.py”, line 212, in get_action<br>
run_out = self.evaluate(decision_requests, global_agent_ids)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\policy\torch_policy.py”, line 178, in evaluate<br>
action, log_probs, entropy, memories = self.sample_actions(<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\policy\torch_policy.py”, line 140, in sample_actions<br>
actions, log_probs, entropies, memories = self.actor.get_action_and_stats(<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\networks.py”, line 626, in get_action_and_stats<br>
action, log_probs, entropies = self.action_model(encoding, masks)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\action_model.py”, line 194, in forward<br>
actions = self._sample_action(dists)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\action_model.py”, line 84, in _sample_action<br>
discrete_action.append(discrete_dist.sample())<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\distributions.py”, line 114, in sample<br>
return torch.multinomial(self.probs, 1)<br>
RuntimeError: cannot sample n_sample &gt; prob_dist.size(-1) samples without replacement</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “C:\Users\antre\AppData\Local\Programs\Python\Python39\lib\runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “C:\Users\antre\AppData\Local\Programs\Python\Python39\lib\runpy.py”, line 87, in <em>run_code<br>
exec(code, run_globals)<br>
File "D:\Desktop\Crowds-and-ML-Agents\venv\Scripts\mlagents-learn.exe_<em>main</em></em>.py", line 7, in <br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\learn.py”, line 250, in main<br>
run_cli(parse_command_line())<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\learn.py”, line 246, in run_cli<br>
run_training(run_seed, options)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\learn.py”, line 125, in run_training<br>
tc.start_learning(env_manager)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\trainer_controller.py”, line 201, in start_learning<br>
self._save_models()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\trainer_controller.py”, line 80, in _save_models<br>
self.trainers[brain_name].save_model()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\trainer\rl_trainer.py”, line 185, in save_model<br>
model_checkpoint = self._checkpoint()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents_envs\timers.py”, line 305, in wrapped<br>
return func(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\trainer\rl_trainer.py”, line 157, in <em>checkpoint<br>
export_path, auxillary_paths = self.model_saver.save_checkpoint(<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\model_saver\torch_model_saver.py”, line 59, in save_checkpoint<br>
self.export(checkpoint_path, behavior_name)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\model_saver\torch_model_saver.py”, line 64, in export<br>
self.exporter.export_policy_model(output_filepath)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\model_serialization.py”, line 159, in export_policy_model<br>
torch.onnx.export(<br>
File "d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\onnx_<em>init</em></em>.py", line 275, in export<br>
return utils.export(model, args, f, export_params, verbose, training,<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\onnx\utils.py”, line 88, in export<br>
_export(model, args, f, export_params, verbose, training, input_names, output_names,<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\onnx\utils.py”, line 689, in _export<br>
_model_to_graph(model, args, verbose, input_names,<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\onnx\utils.py”, line 458, in _model_to_graph<br>
graph, params, torch_out, module = _create_jit_graph(model, args,<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\onnx\utils.py”, line 422, in _create_jit_graph<br>
graph, torch_out = _trace_and_get_graph_from_model(model, args)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\onnx\utils.py”, line 373, in _trace_and_get_graph_from_model<br>
torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\jit_trace.py”, line 1160, in _get_trace_graph<br>
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\jit_trace.py”, line 127, in forward<br>
graph, out = torch._C._create_graph_by_tracing(<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\jit_trace.py”, line 118, in wrapper<br>
outs.append(self.inner(*trace_inputs))<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\torch\nn\modules\module.py”, line 1039, in _slow_forward<br>
result = self.forward(*input, **kwargs)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\networks.py”, line 664, in forward<br>
) = self.action_model.get_action_out(encoding, masks)<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\action_model.py”, line 171, in get_action_out<br>
discrete_out_list = [<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\action_model.py”, line 172, in <br>
discrete_dist.exported_model_output()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\distributions.py”, line 136, in exported_model_output<br>
return self.sample()<br>
File “d:\desktop\crowds-and-ml-agents\venv\lib\site-packages\mlagents\trainers\torch\distributions.py”, line 114, in sample<br>
return torch.multinomial(self.probs, 1)<br>
RuntimeError: cannot sample n_sample &gt; prob_dist.size(-1) samples without replacement</p>

username: ptrblck
cooked: <p>This error is raised, if <code>n_sample</code> is larger than the provided positive probabilities in <code>multinomial</code> and would thus fail, if <code>replacement</code> is set to <code>False</code>.<br>
Check the values of <code>self.probs</code> and make sure that the sampling would be possible.</p>

username: antreas97
cooked: <p>Hello, thanks for your reply. Finally, I found the solution to the problem. It was a setting related to mlagents where I have set the branch size to 0 instead of 1. I write it in case someone encounter the same issue🙂</p>