Is my data too big?

CallMeMisterOwl: <p>Hi everyone,<br>
I’m trying to train a GNN using pytorch_geometric. I already know my data is rather large, but CUDA tries to allocate 107.80 GiB during training !<br>
Currently I only have one 16 GB GPU available and my batches are around 8 GiB big (measured with <code>sys.getsizeof(tensor.storage())</code>. 
I assumed that pytorch would just iterate over the batches, somehow recognise that there is only space for one batch and therefore only load one batch.
<br>
Now, I can trim down my graph and reduce the memory footprint of the data, but I would prefer not.<br>
Also I’m getting a <code>ValueError</code>, but that occurs while handling the CUDA exception, hence it can be ignored.<br>
<code>ValueError: Encountered a CUDA error. Please ensure that all indices in 'edge_index' point to valid indices in the interval [0, 951740) in your node feature matrix and try again.</code></p>
<p>Please let me know what you think.</p>

ptrblck: <aside class="quote no-group" data-username="CallMeMisterOwl" data-post="1" data-topic="168955">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/callmemisterowl/48/52563_2.png" class="avatar"> CallMeMisterOwl:</div>
<blockquote>
<p>I assumed that pytorch would just iterate over the batches, somehow recognise that there is only space for one batch and therefore only load one batch.</p>
</blockquote>
</aside>
<p>No, PyTorch will not automatically reduce the batch size or any other parameter. If you are running out of memory on the GPU, reduce the batch size of your <code>DataLoader</code> manually.</p>

ConvolutionalAtom: <p>If I understand correctly, for your case, only data (input) tensor alone costs 8GiB of memory.<br>
But, for computations done during forward prop and backward intermediate tensor values are stored as well.<br>
In case you already did not, you should also calculate memory of these intermediate tensor values (due to computation graph) to get more accurate measure of memory being used<br>
Of course, there are other factors due to cuda, optimizers etc.</p>

CallMeMisterOwl: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/convolutionalatom">@ConvolutionalAtom</a> I thank you both for your answers.<br>
Firstly, <a class="mention" href="/u/ptrblck">@ptrblck</a> I’m sorry, I was tired yesterday and it seems like I did a bad job articulate my thoughts effectively. I know that the batch size is a fixed value, that has to be assigned before training. I already reduced the batch size to 10, which is already too low for my standards. The issue is that I assumed that a batch which is approximately 8GiB in size would fit into the 16GiB GPU memory <strong>and</strong> leave enough space for calculations. Unfortunately that is not the case since CUDA tries to allocate 107.80 GiB. I don’t understand why pytorch needs that much GRAM tbh.</p>
<p><a class="mention" href="/u/convolutionalatom">@ConvolutionalAtom</a></p>
<aside class="quote no-group" data-username="ConvolutionalAtom" data-post="3" data-topic="168955">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/convolutionalatom/48/45668_2.png" class="avatar"> ConvolutionalAtom:</div>
<blockquote>
<p>In case you already did not, you should also calculate memory of these intermediate tensor values (due to computation graph) to get more accurate measure of memory being used<br>
Of course, there are other factors due to cuda, optimizers etc.</p>
</blockquote>
</aside>
<p>I understand and no I did not consider that the tensors used for computation could be that big. I will figure out how to measure the memory footprint of those tensors and I hope that will clear up where the 107 GiB come from.</p>

ptrblck: <p>Depending on the model architecture the intermediate activations would need a huge amount of memory as <a class="mention" href="/u/convolutionalatom">@ConvolutionalAtom</a> explained.<br>
A simple conv layer would be a good example, as it’s often not reducing the spatial size significantly while increasing the number of channels in the output activation.<br>
An 8GB input would thus create an even larger output.<br>
Take a look at <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">this post</a> which estimates the memory usage of a resnet.</p>

CallMeMisterOwl: <p>Unfortunately I couldn’t get the code in the post to run properly. The forward act is always 0, but I think I get what you indented to show with the post → memory usage of the calculation should not be underestimated.<br>
I will trim down my graphs and look into multi-gpu training, hope that will solve my issue.</p>
<p>Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>