Training suddenly stops with IndexError: scatter_(): Expected dtype int64 for index

Allan_Jie: <p>It works prettt well until the 23rd epochs. not sure what is happening here</p>
<pre><code class="lang-bash">--training batch:  39%|███▉      | 999/2567 [03:57&lt;08:04,  3.23it/s]epoch: 21, iteration: 1000, current mean loss: 0.17
--training batch:  78%|███████▊  | 1999/2567 [07:58&lt;02:09,  4.38it/s]epoch: 21, iteration: 2000, current mean loss: 0.17
--training batch: 100%|██████████| 2567/2567 [10:12&lt;00:00,  4.19it/s]Finish epoch: 21, loss: 426.59, mean loss: 0.17

--validation: 100%|██████████| 321/321 [00:43&lt;00:00,  7.32it/s][Info] Acc.:71.52 
[Info] val Acc.:78.27 
--training batch:  39%|███▉      | 999/2567 [03:59&lt;05:56,  4.39it/s]epoch: 22, iteration: 1000, current mean loss: 0.15
--training batch:  78%|███████▊  | 1999/2567 [07:52&lt;02:03,  4.58it/s]epoch: 22, iteration: 2000, current mean loss: 0.14
--training batch: 100%|██████████| 2567/2567 [10:05&lt;00:00,  4.24it/s]Finish epoch: 22, loss: 367.22, mean loss: 0.14

--validation: 100%|██████████| 321/321 [00:43&lt;00:00,  7.35it/s][Info] Acc.:69.11 
[Info] val Acc.:74.63 

--training batch:  39%|███▉      | 999/2567 [04:01&lt;06:07,  4.26it/s]epoch: 23, iteration: 1000, current mean loss: 0.13
--training batch:  78%|███████▊  | 1999/2567 [07:59&lt;02:38,  3.58it/s]epoch: 23, iteration: 2000, current mean loss: 0.13
--training batch: 100%|█████████▉| 2566/2567 [10:13&lt;00:00,  4.18it/s]
Traceback (most recent call last):
  File "universal_main.py", line 422, in &lt;module&gt;
    main()
  File "universal_main.py", line 398, in main
    constant_values=constant_values)
  File "universal_main.py", line 138, in train
    scaler.scale(loss).backward()
  File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
IndexError: scatter_(): Expected dtype int64 for index.
</code></pre>

soulitzer: <p>Seems quite weird, to see where it is coming from, can you try to run it again but with the <code>TORCH_SHOW_CPP_STACKTRACES=1</code> environment variable enabled?</p>
<p>Enabling anomaly mode <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.10.0 documentation</a> may also help.</p>

Allan_Jie: <p>Sure., I can run the experiment again.  It will take some time to run. Probably a few hours later. I will paste the results here.</p>
<p>I was also wondering if there are any issues with vanishing/exploding gradient issues.</p>
<p>Notes for the training process:</p>
<pre><code class="lang-python">    for epoch in range(num_epochs):
        total_loss = 0
        model.train()
        for iter, feature in tqdm(enumerate(train_dataloader, 1), desc="--training batch", total=len(train_dataloader)):
            optimizer.zero_grad()
            with torch.cuda.amp.autocast(enabled=bool(config.fp16)):
                loss = model(.....).loss
            if config.fp16:
                scaler.scale(loss).backward()
                scaler.unscale_(optimizer)
            else:
                loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            total_loss += loss.item()
            if config.fp16:
                scaler.step(optimizer)
                scaler.update()
            else:
                optimizer.step()
            scheduler.step()
            model.zero_grad()
</code></pre>

Allan_Jie: <p>here comes the error</p>
<pre><code class="lang-bash">
--training batch:  39%|███▉      | 999/2567 [03:57&lt;06:04,  4.30it/s]epoch: 23, iteration: 1000, current mean loss: 0.13
--training batch:  78%|███████▊  | 1999/2567 [07:54&lt;02:37,  3.61it/s]epoch: 23, iteration: 2000, current mean loss: 0.13
--training batch: 100%|█████████▉| 2566/2567 [10:07&lt;00:00,  4.22it/s]
Traceback (most recent call last):
  File "universal_main.py", line 422, in &lt;module&gt;
    if __name__ == "__main__":
  File "universal_main.py", line 398, in main
    dev=conf.device, tokenizer=tokenizer, num_labels=num_labels,
  File "universal_main.py", line 138, in train
    scaler.scale(loss).backward()
  File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
IndexError: scatter_(): Expected dtype int64 for index.
Exception raised from scatter_add_ at /pytorch/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1082 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fa0603bda22 in /usr/local/lib/python3.7/dist-packages/torch/lib/libc10.so)
frame #1: at::native::scatter_add_(at::Tensor&amp;, long, at::Tensor const&amp;, at::Tensor const&amp;) + 0x11b (0x7fa061ac337b in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: &lt;unknown function&gt; + 0xbccd62 (0x7fa015338d62 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cuda_cu.so)
frame #3: at::redispatch::scatter_add_(c10::DispatchKeySet, at::Tensor&amp;, long, at::Tensor const&amp;, at::Tensor const&amp;) + 0xb5 (0x7fa0620ddb15 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x350ddad (0x7fa063d49dad in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: at::redispatch::scatter_add_(c10::DispatchKeySet, at::Tensor&amp;, long, at::Tensor const&amp;, at::Tensor const&amp;) + 0xb5 (0x7fa0620ddb15 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x2fae5bb (0x7fa0637ea5bb in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #7: at::Tensor::scatter_add_(long, at::Tensor const&amp;, at::Tensor const&amp;) const + 0x151 (0x7fa062568bc1 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #8: at::native::gather_backward(at::Tensor const&amp;, at::Tensor const&amp;, long, at::Tensor const&amp;, bool) + 0x87 (0x7fa061ad7b67 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #9: &lt;unknown function&gt; + 0x1b8556d (0x7fa0623c156d in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #10: at::gather_backward(at::Tensor const&amp;, at::Tensor const&amp;, long, at::Tensor const&amp;, bool) + 0x194 (0x7fa061ef4a04 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #11: torch::autograd::generated::GatherBackward::apply(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;&amp;) + 0x1b7 (0x7fa063722837 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #12: &lt;unknown function&gt; + 0x355a3fa (0x7fa063d963fa in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #13: torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;) + 0x1477 (0x7fa063d91c77 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #14: torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) + 0x47b (0x7fa063d929fb in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #15: torch::autograd::Engine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) + 0x89 (0x7fa063d8ac49 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) + 0x53 (0x7fa1051b94b3 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_python.so)
frame #17: &lt;unknown function&gt; + 0xbbb2f (0x7fa10639eb2f in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #18: &lt;unknown function&gt; + 0x7fa3 (0x7fa107ea1fa3 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #19: clone + 0x3f (0x7fa1079e84cf in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>