Pytorch GPU Memory Usage

legoh: <p>Hi guys, I’m not really sure why this is happening but if I measure my data object, it’s about 265mb in the GPU. If I measure the model, it’s also about 300mb. But once I start training, pytorch uses up almost all my GPU memory. I can’t really understand why. I’ve set <code>pin_memory=False</code> in my <code>DataLoader</code> already and it still displays this behavior. Is there a way to trace the memory usage properly for each object? I’m only doing a single sample every time as I can’t move to batches since all my GPU is being used up.</p>

ptrblck: <p>Besides the data and model parameters, the CUDA context will use some memory as well as the intermediate activations, which are needed to calculate the backward pass.<br>
Also note that PyTorch uses a caching allocator, which will reuse the memory.<br>
<code>nvidia-smi</code> will thus show the complete memory usage, while <code>torch.cuda.memory_allocated()</code> will give you the allocated memory only.</p>

legoh: <p>Hmm if that’s the case, it’d mean that majority of the GPU allocation is for the intermediate stages of forward / backward passes? Hence, will it be likely that I can improve the GPU allocation if I reduce the data types of the inputs? e.g. converting from float 64 to float 32.</p>

ptrblck: <aside class="quote no-group" data-username="legoh" data-post="3" data-topic="78565">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/82dd89/48.png" class="avatar"> legoh:</div>
<blockquote>
<p>it’d mean that majority of the GPU allocation is for the intermediate stages of forward / backward passes?</p>
</blockquote>
</aside>
<p>That might be the case.<br>
E.g. for a single conv layer, the output might contain more elements than the input, if <code>out_channels &gt; in_chanels</code> and you don’t reduce the spatial size.</p>
<aside class="quote no-group" data-username="legoh" data-post="3" data-topic="78565">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/82dd89/48.png" class="avatar"> legoh:</div>
<blockquote>
<p>converting from float 64 to float 32.</p>
</blockquote>
</aside>
<p>Yes. PyTorch uses <code>float32</code> by default, so if you’ve called <code>double()</code> on your model (and don’t strictly need the precision), I would recommend to use FP32.</p>

legoh: <p>Okay got it. Here’s something that’s really puzzling to me though. 1 sample of my DataLoader appears to have ~250Mb (when I first load 1 sample into the GPU). But oddly, if i increment my batch size to 4, I get this error</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 7.41 GiB (GPU 0; 11.17 GiB total capacity; 8.34 GiB already allocated; 2.28 GiB free; 8.60 GiB reserved in total by PyTorch)
</code></pre>
<p>Which is really odd, does this mean my data object is actually larger? In this instance, 7.41/4 = 1.8 Gb?</p>

ptrblck: <p>Yes, it means that not only the data sample itself uses additional memory, but also all intermediate activations.</p>

legoh: <p>I see ok. So one way is to reduce the size of the tensors that in the object. I keep a tensor that is used to index another tensor in the object (e.g. A[b: ]). In this case, does b have to be of type LongTensor? Or can I reduce this to a smaller integer type?</p>