Multi-gpu training crashes in A6000

adelaide: <p>Hi,</p>
<p>I am trying to train <a href="https://github.com/facebookresearch/dino" rel="noopener nofollow ugc">dino</a> with 2 <strong>A6000</strong> gpus. The code works fine when I train on a single gpu but crashes when I use 2 gpus. My python version is 3.8.11, pytorch version is 1.9.0, torch.version.cuda: 11.1.<br>
Does anyone have any idea how to debug this error or to solve this problem? Thanks in Advance!</p>
<p>Command:<br>
python -m torch.distributed.launch --nproc_per_node=2 main_dino.py --arch resnet50 --optimizer sgd --weight_decay 1e-4 --weight_decay_end 1e-4 --global_crops_scale 0.14 1 --local_crops_scale 0.05 0.14 --data_path /home/ma/uname/dataset/imagenet/train --output_dir /temp</p>
<p>Error message:<br>
/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py:163: DeprecationWarning: The ‘warn’ method is deprecated, use ‘warning’ instead<br>
logger.warn(<br>
The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run</p>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
<p>WARNING:torch.distributed.run:–use_env is deprecated and will be removed in future releases.<br>
Please read local_rank from <code>os.environ('LOCAL_RANK')</code> instead.<br>
INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:<br>
entrypoint       : main_dino.py<br>
min_nodes        : 1<br>
max_nodes        : 1<br>
nproc_per_node   : 2<br>
run_id           : none<br>
rdzv_backend     : static<br>
rdzv_endpoint    : 127.0.0.1:29500<br>
rdzv_configs     : {‘rank’: 0, ‘timeout’: 900}<br>
max_restarts     : 3<br>
monitor_interval : 5<br>
log_dir          : None<br>
metrics_cfg      : {}</p>
<p>INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd<br>
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:52: FutureWarning: This is an experimental API and will be changed in future.<br>
warnings.warn(<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=0<br>
master_addr=127.0.0.1<br>
master_port=29500<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_0/0/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_0/1/error.json<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
| distributed init (rank 0): env://<br>
| distributed init (rank 1): env://<br>
[W ProcessGroupNCCL.cpp:1569] Rank 0 using best-guess GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.<br>
[W ProcessGroupNCCL.cpp:1569] Rank 1 using best-guess GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.</p>
<p>[E ProcessGroupNCCL.cpp:566] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1807903 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:566] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1807905 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1807903 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1807905 milliseconds before timing out.<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 23796) of binary: /home/ma/uname/maanaconda3/envs/mdetr/bin/python<br>
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=1<br>
master_addr=127.0.0.1<br>
master_port=29500<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_1/0/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_1/1/error.json<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Traceback (most recent call last):<br>
File “main_dino.py”, line 472, in <br>
train_dino(args)<br>
File “main_dino.py”, line 134, in train_dino<br>
utils.init_distributed_mode(args)<br>
File “/home/ma/uname/code/dino_orig/utils.py”, line 468, in init_distributed_mode<br>
Traceback (most recent call last):<br>
File “main_dino.py”, line 472, in <br>
train_dino(args)<br>
File “main_dino.py”, line 134, in train_dino<br>
utils.init_distributed_mode(args)<br>
File “/home/ma/uname/code/dino_orig/utils.py”, line 468, in init_distributed_mode<br>
dist.init_process_group(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 547, in init_process_group<br>
dist.init_process_group(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 547, in init_process_group<br>
_store_based_barrier(rank, store, timeout)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 219, in _store_based_barrier<br>
_store_based_barrier(rank, store, timeout)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 219, in _store_based_barrier<br>
raise RuntimeError(<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=4, timeout=0:30:00)<br>
raise RuntimeError(<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=4, timeout=0:30:00)<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 23980) of binary: /home/ma/uname/maanaconda3/envs/mdetr/bin/python<br>
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=2<br>
master_addr=127.0.0.1<br>
master_port=29500<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_2/0/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_2/1/error.json<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Traceback (most recent call last):<br>
File “main_dino.py”, line 472, in <br>
train_dino(args)<br>
File “main_dino.py”, line 134, in train_dino<br>
utils.init_distributed_mode(args)<br>
File “/home/ma/uname/code/dino_orig/utils.py”, line 468, in init_distributed_mode<br>
dist.init_process_group(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 547, in init_process_group<br>
_store_based_barrier(rank, store, timeout)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 219, in _store_based_barrier<br>
raise RuntimeError(<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=6, timeout=0:30:00)<br>
Traceback (most recent call last):<br>
File “main_dino.py”, line 472, in <br>
train_dino(args)<br>
File “main_dino.py”, line 134, in train_dino<br>
utils.init_distributed_mode(args)<br>
File “/home/ma/uname/code/dino_orig/utils.py”, line 468, in init_distributed_mode<br>
dist.init_process_group(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 547, in init_process_group<br>
_store_based_barrier(rank, store, timeout)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 219, in _store_based_barrier<br>
raise RuntimeError(<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=6, timeout=0:30:00)<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 24138) of binary: /home/ma/uname/maanaconda3/envs/mdetr/bin/python<br>
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=3<br>
master_addr=127.0.0.1<br>
master_port=29500<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_3/0/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_q72lm7ip/none_glcjvhtd/attempt_3/1/error.json<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Using cache found in /home/ma/uname/.cache/torch/hub/facebookresearch_xcit_master<br>
Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
File “main_dino.py”, line 472, in <br>
File “main_dino.py”, line 472, in <br>
train_dino(args)<br>
File “main_dino.py”, line 134, in train_dino<br>
train_dino(args)<br>
File “main_dino.py”, line 134, in train_dino<br>
utils.init_distributed_mode(args)<br>
File “/home/ma/uname/code/dino_orig/utils.py”, line 468, in init_distributed_mode<br>
utils.init_distributed_mode(args)<br>
File “/home/ma/uname/code/dino_orig/utils.py”, line 468, in init_distributed_mode<br>
dist.init_process_group(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 547, in init_process_group<br>
dist.init_process_group(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 547, in init_process_group<br>
_store_based_barrier(rank, store, timeout)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 219, in _store_based_barrier<br>
_store_based_barrier(rank, store, timeout)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 219, in _store_based_barrier<br>
raise RuntimeError(<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=8, timeout=0:30:00)<br>
raise RuntimeError(<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=8, timeout=0:30:00)<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 24711) of binary: /home/ma/uname/maanaconda3/envs/mdetr/bin/python<br>
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed<br>
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish<br>
/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:70: FutureWarning: This is an experimental API and will be changed in future.<br>
warnings.warn(<br>
INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.0004601478576660156 seconds<br>
{“name”: “torchelastic.worker.status.FAILED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: 0, “group_rank”: 0, “worker_id”: “24711”, “role”: “default”, “hostname”: “ma-gpu04”, “state”: “FAILED”, “total_run_time”: 7237, “rdzv_backend”: “static”, “raw_error”: “{“message”: “”}”, “metadata”: “{“group_world_size”: 1, “entry_point”: “python”, “local_rank”: [0], “role_rank”: [0], “role_world_size”: [2]}”, “agent_restarts”: 3}}<br>
{“name”: “torchelastic.worker.status.FAILED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: 1, “group_rank”: 0, “worker_id”: “24712”, “role”: “default”, “hostname”: “ma-gpu04”, “state”: “FAILED”, “total_run_time”: 7237, “rdzv_backend”: “static”, “raw_error”: “{“message”: “”}”, “metadata”: “{“group_world_size”: 1, “entry_point”: “python”, “local_rank”: [1], “role_rank”: [1], “role_world_size”: [2]}”, “agent_restarts”: 3}}<br>
{“name”: “torchelastic.worker.status.SUCCEEDED”, “source”: “AGENT”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: null, “group_rank”: 0, “worker_id”: null, “role”: “default”, “hostname”: “ma-gpu04”, “state”: “SUCCEEDED”, “total_run_time”: 7237, “rdzv_backend”: “static”, “raw_error”: null, “metadata”: “{“group_world_size”: 1, “entry_point”: “python”}”, “agent_restarts”: 3}}<br>
/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py:354: UserWarning:</p>
<hr>
<pre><code>           CHILD PROCESS FAILED WITH NO ERROR_FILE
</code></pre>
<hr>
<p>CHILD PROCESS FAILED WITH NO ERROR_FILE<br>
Child process 24711 (local_rank 0) FAILED (exitcode 1)<br>
Error msg: Process failed with exitcode 1<br>
Without writing an error file to &lt;N/A&gt;.<br>
While this DOES NOT affect the correctness of your application,<br>
no trace information about the error will be available for inspection.<br>
Consider decorating your top level entrypoint function with<br>
torch.distributed.elastic.multiprocessing.errors.record. Example:</p>
<p>from torch.distributed.elastic.multiprocessing.errors import record</p>
<p><span class="mention">@record</span><br>
def trainer_main(args):<br>
# do train</p>
<hr>
<p>warnings.warn(_no_error_file_warning_msg(rank, failure))<br>
Traceback (most recent call last):<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py”, line 173, in <br>
main()<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py”, line 169, in main<br>
run(args)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/run.py”, line 621, in run<br>
elastic_launch(<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 116, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 348, in wrapper<br>
return f(*args, **kwargs)<br>
File “/home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 245, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>
<hr>
<pre><code>      main_dino.py FAILED
</code></pre>
<h1>
<a name="root-cause-0-time-2021-09-17_134215-rank-0-local_rank-0-exitcode-1-pid-24711-error_file-na-msg-process-failed-with-exitcode-1-1" class="anchor" href="#root-cause-0-time-2021-09-17_134215-rank-0-local_rank-0-exitcode-1-pid-24711-error_file-na-msg-process-failed-with-exitcode-1-1"></a>=======================================<br>
Root Cause:<br>
[0]:<br>
time: 2021-09-17_13:42:15<br>
rank: 0 (local_rank: 0)<br>
exitcode: 1 (pid: 24711)<br>
error_file: &lt;N/A&gt;<br>
msg: “Process failed with exitcode 1”</h1>
<p>Other Failures:<br>
[1]:<br>
time: 2021-09-17_13:42:15<br>
rank: 1 (local_rank: 1)<br>
exitcode: 1 (pid: 24712)<br>
error_file: &lt;N/A&gt;<br>
msg: “Process failed with exitcode 1”</p>
<hr>

cbalioglu: <p>Hi,</p>
<p>In the error output, if you look closely, you will see a recommendation to gather more information about the root cause:</p>
<aside class="quote no-group" data-username="adelaide" data-post="1" data-topic="132279">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ecae2f/48.png" class="avatar"> adelaide:</div>
<blockquote>
<p>CHILD PROCESS FAILED WITH NO ERROR_FILE<br>
Child process 24711 (local_rank 0) FAILED (exitcode 1)<br>
Error msg: Process failed with exitcode 1<br>
Without writing an error file to &lt;N/A&gt;.<br>
While this DOES NOT affect the correctness of your application,<br>
no trace information about the error will be available for inspection.<br>
Consider decorating your top level entrypoint function with<br>
torch.distributed.elastic.multiprocessing.errors.record. Example:</p>
<p>from torch.distributed.elastic.multiprocessing.errors import record</p>
<p><span class="mention">@record</span><br>
def trainer_main(args):</p>
</blockquote>
</aside>
<p>You can find more information about the <code>record</code> decorator <a href="https://pytorch.org/docs/stable/elastic/errors.html" rel="noopener nofollow ugc">here</a>. If you decorate your main function with <code>@record</code> you should get a full stack trace of the actual exception.</p>
<p>Having said that I would also suggest running your code with the <code>--max_restarts=0</code> option. This was a bug in v1.9 that is fixed in v1.9.1. Without that option, your training will be restarted up to 3 times before ultimately failing.</p>