RuntimeError: Broken pipe using NVIDIA Megatron-LM

username: Jakson
cooked: <p>Hello!<br>
I’m experimenting with distributed training using NVIDIA <a href="https://github.com/NVIDIA/Megatron-LM" rel="nofollow noopener">Megatron-LM project</a>. And I get an error when running the script bash <a href="https://github.com/NVIDIA/Megatron-LM/blob/master/scripts/pretrain_gpt2_model_parallel.sh" rel="nofollow noopener">scripts/pretrain_gpt2_model_parallel.sh</a></p>
<p>Traceback looks like</p>
<pre><code class="lang-auto">File "pretrain_gpt2.py", line 625, in &lt;module&gt;
    main()
  File "pretrain_gpt2.py", line 569, in main
    args.eod_token = get_train_val_test_data(args)
  File "pretrain_gpt2.py", line 536, in get_train_val_test_data
    group=mpu.get_model_parallel_group())
  File "/home/ubuntu/Env/ml/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: Broken pipe
Traceback (most recent call last):
  File "pretrain_gpt2.py", line 625, in &lt;module&gt;
    main()
  File "pretrain_gpt2.py", line 569, in main
    args.eod_token = get_train_val_test_data(args)
  File "pretrain_gpt2.py", line 536, in get_train_val_test_data
    group=mpu.get_model_parallel_group())
  File "/home/ubuntu/Env/ml/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: Broken pipe
Traceback (most recent call last):
  File "pretrain_gpt2.py", line 625, in &lt;module&gt;
    main()
  File "pretrain_gpt2.py", line 569, in main
    args.eod_token = get_train_val_test_data(args)
  File "pretrain_gpt2.py", line 536, in get_train_val_test_data
    group=mpu.get_model_parallel_group())
  File "/home/ubuntu/Env/ml/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: Broken pipe
Traceback (most recent call last):
  File "pretrain_gpt2.py", line 625, in &lt;module&gt;
    main()
  File "pretrain_gpt2.py", line 569, in main
    args.eod_token = get_train_val_test_data(args)
  File "pretrain_gpt2.py", line 536, in get_train_val_test_data
    group=mpu.get_model_parallel_group())
  File "/home/ubuntu/Env/ml/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: Broken pipe
</code></pre>
<p>The error occurs in the file  <a href="https://github.com/NVIDIA/Megatron-LM/blob/master/pretrain_gpt2.py" rel="nofollow noopener">pretrain_gpt2.py</a></p>
<p>Could anybody help me with this issue?</p>

username: ptrblck
cooked: <p>I would recommend to create an issue in the GitHub repository directly, as the authors of the code might help there.</p>

username: Jakson
cooked: <p>I did, but unfortunately I didn’t get an answer.<br>
The error traceback refers to <code>lib/python3.6/site-packages/torch/distributed/distributed_c10d.py</code> . That’s why I thought I might be able to get a hint here.</p>

username: ptrblck
cooked: <p>I assume you’ve created <a href="https://github.com/NVIDIA/Megatron-LM/issues/15" rel="nofollow noopener">this issue</a>?<br>
It looks like your datasets are empty and the actual error message is:</p>
<pre><code class="lang-python">TypeError: iteration over a 0-d array
</code></pre>
<p>when calculating the dataset lengths.<br>
I’ll also post in the issue directly.</p>

username: Jakson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> That’s a great idea, I’ll check it out . Thank you</p>