How to train a neural network on a list of tensors with different shapes

fatcat: <p>I have a list of tensors with different shapes; the list is denoted by <code>examples_train</code>, and the corresponding label list is <code>labels_train</code>; there are about 1200 examples in the list <code>examples_train</code>.  I want to fit my model on this list. I cannot use data loader since the tensors in the list have different shapes.</p>
<p>My current method is to compute loss of the examples one by one, and sum up the losses to do backpropagation. The code is:</p>
<pre><code class="lang-auto">training_indices = np.arange(len(examples_train))
np.random.shuffle(training_indices) #shuffle the examples
optimizer.zero_grad()
loss = 0.
for idx in training_indices:
      example = example_train[idx]
      label = labels_train[idx]
     #Add batch dimension
      example = example.unsqueeze(0)
      label = label.unsqueeze(0)
      example = example.cuda()
      label = label.cuda()
      logits = model(example)
      current_loss = F.cross_entropy(logits, label)
      loss += current_loss
loss = loss/len(examples_train)
loss.backward()
optimizer.step()
</code></pre>
<p>The code above works, but sometimes there might be  “CUDA out of memory” error, and the optimisation is also very slow. How can I fix this problem? Should I partition the list <code>examples_train</code> into smaller sub lists(mini batches)?</p>

huahuanZ: <ol>
<li>
<p>can you use dataloader for variable length data?<br>
Definitely yes, you should implement a collate_fn for the dataloader (commonly we pad the data to fit the size and be arranged as mini-batch)</p>
</li>
<li>
<p>why does the GPU memory increase in iterations?<br>
the line <code>loss +=...</code> makes torch store all intermediate tensors from step 1 until you call backward ()</p>
</li>
</ol>

fatcat: <p>Thanks for your reply.<br>
Actually in my case each example need to multiply with another corresponding tensor in the forward process, which I didn’t mention above, so it still seems impossible to use dataloader.</p>
<p>For the  GPU memory increasing problem, can I partition the <code>examples_train</code> list into small lists(mini-batches) and do backpropagation for each list to solve this problem?</p>
<p>Best Regards.</p>

huahuanZ: <p>Can you give a code snippet example for the problem? It’s not very clear how the “corresponding tensor” is used.</p>

fatcat: <p>Actually my code is to implement a graph neural network and the examples are graphs including different number of nodes. There are node2edge and edge2node operations in the forward process, which need matrix which is specific for each graph.</p>

fatcat: <p>To solve this problem, can I delete the intermediate tensors after I added them to the loss variable?<br>
I.e.</p>
<pre><code class="lang-auto">loss += current_loss
del current_loss
</code></pre>

huahuanZ: <p>In your case, if it’s impossible to batch the data. Using gradient accumulating is OK. But as you mentioned, you should partition the entire dataset into mini-batch size, and accumulating gradients for limited steps (the same as “batch size”), otherwise the GPU memory usage is going to explode.<br>
Your don’t need to manually delete the loss variable. After <code>loss.backward()</code>, the computation graph is released by pytorch itself.</p>
<p>And just a hint, accumulating gradients by step is not identical to mini-batching update when there are layers like batch-norm in your network.</p>