Aborted in smolvla start offline training

username: cyc
cooked: <p>I encountered a problem. When I started training with the smolvla model, it aborted. The following is the error message</p>
<p>environment：</p>
<p>ubuntu：22.04</p>
<p>memory：128G</p>
<p>GPU：RTX5090 48G CUDA12.8</p>
<p>pytorch：2.9.0.dev20250904+cu128</p>
<p>INFO 2025-09-08 14:07:07 ts/train.py:144 Creating optimizer and scheduler<br>
INFO 2025-09-08 14:07:07 ts/train.py:156 Output dir: outputs/train/my_smolvla<br>
INFO 2025-09-08 14:07:07 ts/train.py:159 cfg.steps=20000 (20K)<br>
INFO 2025-09-08 14:07:07 ts/train.py:160 dataset.num_frames=11939 (12K)<br>
INFO 2025-09-08 14:07:07 ts/train.py:161 dataset.num_episodes=50<br>
INFO 2025-09-08 14:07:07 ts/train.py:162 num_learnable_params=99880992 (100M)<br>
INFO 2025-09-08 14:07:07 ts/train.py:163 num_total_params=450046212 (450M)<br>
INFO 2025-09-08 14:07:07 ts/train.py:177 cfg.num_workers=4<br>
INFO 2025-09-08 14:07:07 ts/train.py:178 cfg.batch_size=64<br>
INFO 2025-09-08 14:07:07 ts/train.py:204 Start offline training on a fixed dataset<br>
Traceback (most recent call last):<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1275, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/queue.py”, line 180, in get<br>
self.not_empty.wait(remaining)<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/threading.py”, line 324, in wait<br>
gotit = waiter.acquire(True, timeout)<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py”, line 73, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 21350) is killed by signal: Aborted.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/ubuntu/lerobot/src/lerobot/scripts/train.py”, line 297, in<br>
main()<br>
File “/home/ubuntu/lerobot/src/lerobot/scripts/train.py”, line 293, in main<br>
train()<br>
File “/home/ubuntu/lerobot/src/lerobot/configs/parser.py”, line 225, in wrapper_inner<br>
response = fn(cfg, *args, **kwargs)<br>
File “/home/ubuntu/lerobot/src/lerobot/scripts/train.py”, line 207, in train<br>
batch = next(dl_iter)<br>
File “/home/ubuntu/lerobot/src/lerobot/datasets/utils.py”, line 630, in cycle<br>
yield next(iterator)<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 732, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1482, in _next_data<br>
idx, data = self._get_data()<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1434, in _get_data<br>
success, data = self._try_get_data()<br>
File “/home/ubuntu/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1288, in _try_get_data<br>
raise RuntimeError(<br>
RuntimeError: DataLoader worker (pid(s) 21350) exited unexpectedly</p>
<p>I tried num_workers=0, which caused a aborted (core dumped).The memory is not fully used and the GPU is sufficient.How can I locate this problem and solve it? Sincerely waiting for your reply.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="cyc" data-post="1" data-topic="222933">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/2acd7d/48.png" class="avatar"> cyc:</div>
<blockquote>
<p>I tried num_workers=0, which caused a aborted (core dumped).</p>
</blockquote>
</aside>
<p>Try to narrow down what exactly fails e.g. by running your script via <code>gdb</code> and by collecting the stacktrace:</p>
<pre><code class="lang-auto">gdb --args python script.py args
...
r
...
bt
</code></pre>