IndexError: Key out of bounds with size 0, while using DistributedDataParallel

Manas_Tripathi: <p>I will explain the entire issue.<br>
I am using a 4 gpu machine each of 20gb for finetuning mistral.<br>
When I train a model, the gpu distributes some memory over all the gpus but fully utilises gpu 1. As soon as the memory is exhausted it throws an error Cuda OutofMemory instead of switching to the next gpu<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82658d67b47749fbc4605b62a1e5706a52886ce8.png" data-download-href="https://discuss.pytorch.org/uploads/default/82658d67b47749fbc4605b62a1e5706a52886ce8" title="Screenshot 2024-06-10 140100"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82658d67b47749fbc4605b62a1e5706a52886ce8.png" alt="Screenshot 2024-06-10 140100" data-base62-sha1="iBxE3Ajb4oS3ERcmAAkzhoh6VXq" width="572" height="500" data-dominant-color="111212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-06-10 140100</span><span class="informations">802×701 35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I tried to use DistributedDataParallel but now I am facing this IndexError mentioned above:</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend="nccl")
 Set device based on local rank
local_rank = int(os.environ["LOCAL_RANK"])#
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)
.
.
model = nn.parallel.DistributedDataParallel(model, device_ids=[local_rank], output_device=local_rank, find_unused_parameters=True)


</code></pre>
<p>Command to run  torchrun --nproc_per_node=4 --nnodes=1 --node_rank=0 mistral-opt.py</p>
<p>There are no missing values in my dataset. I am not sure what is happening.Need some help</p>
<p>An IndexError occurred: Invalid key: 41784 is out of bounds for size 0<br>
Skipping this error and continuing execution.<br>
An IndexError occurred: Invalid key: 26752 is out of bounds for size 0<br>
Skipping this error and continuing execution.<br>
An IndexError occurred: Invalid key: 35187 is out of bounds for size 0<br>
Skipping this error and continuing execution.<br>
An IndexError occurred: Invalid key: 38692 is out of bounds for size 0<br>
Skipping this error and continuing execution.</p>

ptrblck: <aside class="quote no-group" data-username="Manas_Tripathi" data-post="1" data-topic="204402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/manas_tripathi/48/70646_2.png" class="avatar"> Manas_Tripathi:</div>
<blockquote>
<p>As soon as the memory is exhausted it throws an error Cuda OutofMemory instead of switching to the next gpu</p>
</blockquote>
</aside>
<p>I don’t understand why the script should switch to the next GPU somehow. Did you debug why the imbalanced memory usage is seen in the first place?</p>
<aside class="quote no-group" data-username="Manas_Tripathi" data-post="1" data-topic="204402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/manas_tripathi/48/70646_2.png" class="avatar"> Manas_Tripathi:</div>
<blockquote>
<p>There are no missing values in my dataset. I am not sure what is happening.Need some help</p>
</blockquote>
</aside>
<p>Narrow down which line of code raises the warning and then check all corresponding inputs of the operation.</p>

Manas_Tripathi: <p>So, the machine has 4 gpus each of 20gb memory, The OutofMemory error occurs when Gpu 1 runs of out of memory, but as you can see the other gpus are still having a lot of unused memory, I wanted to know as to why it is not utilising the remaining memory instead to allocate space.<br>
Also, I am new to this so can you tell how I can debug the imbalanced memory usage.</p>
<p>The error occurs in : trainer.train() , I am trying to finetune Mistral on a dataset</p>

ptrblck: <aside class="quote no-group" data-username="Manas_Tripathi" data-post="3" data-topic="204402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/manas_tripathi/48/70646_2.png" class="avatar"> Manas_Tripathi:</div>
<blockquote>
<p>I wanted to know as to why it is not utilising the remaining memory instead to allocate space.</p>
</blockquote>
</aside>
<p>PyTorch does not automatically allocate memory on different devices and since each memory allocation is explicit, you would need to narrow down why the memory usage is higher on GPU0.<br>
You could start by adding debug <code>print</code> statements narrowing down where the memory increases unexpectedly via <code>print(torch.cuda.memory_allocated())</code> (or similar calls).</p>

Manas_Tripathi: <p><a href="https://discuss.pytorch.org/t/uneven-gpu-utilization-during-training-backpropagation/36117">https://discuss.pytorch.org/t/uneven-gpu-utilization-during-training-backpropagation/36117</a><br>
I found a thread related to the issue I am facing, I will try to see if I can change the output devices for gradient accumulation and forward/backward passes.<br>
Thanks!</p>

ptrblck: <p>The linked issue uses <code>nn.DataParallel</code> which is known to create an imbalanced usage since the default GPU device will be used to gather all inputs and outputs. In your title you are mentioning <code>DistributedDataParallel</code>, which does not suffer from this issue since each device uses its own process for the model training including data loading and processing.</p>

Manas_Tripathi: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="204402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>for the model training including data loading and processing</p>
</blockquote>
</aside>
<p>Yes I figured out this error was occuring in the tokenization step.Thanks alot!</p>