A huge difference of memory usage on different GPUs

sunxm: <p>Hi all,</p>
<p>I have a problem about memory consulting on different GPUs. I implement a model containing convolution layers and LSTM. I try to train it using both the GPU on my workstation and also the GPU on the server. However, it consults different size of memory on different GPUs, which confuses me.</p>
<p>The GPU on my workstation is GeForce GTX. When the model uses this GPU, it takes 5286MB in total. However, when it runs on the TITAN X on the server, it takes up to 7539MB. There is a huge difference.</p>
<p>Especially, when it runs on TITAN X, it consults only 5315MB after first time backward and it consults 7539MB since the second time backward. I do not understand why the second time backward consults so much memory? Since the memory usage holds the same after the second backward, I cannot conclude there exists memory leaks.</p>
<p>Can anyone share any thought about this situation?</p>
<p>Best</p>

SimonW: <p>Do you have identical cuda, cudnn and pytorch versions?</p>

sunxm: <p>Yes, they are the same.</p>

SimonW: <p>If the numbers you mentioned are observed via nvidia-smi, then it is not an accurate depiction of the actual memory usage since pytorch use a caching allocator <a href="http://pytorch.org/docs/0.3.0/notes/cuda.html#memory-management">http://pytorch.org/docs/0.3.0/notes/cuda.html#memory-management</a>. Moreover, the cudnn may choose different algorithms basing on different architectures. Since your model contains conv layers and an LSTM, it uses cudnn heavily.</p>

sunxm: <p>Thank you for explanation.</p>
<p>I get all the numbers from nvidia-smi. I have tried out using <code>torch.cuda.empty_cache()</code> to free all unused memory and I get the same result as I stated before.</p>
<p>I know that memory allocation will be different on different architectures, but is it responsible for such a huge difference of memory usage, i.e. 5286MB v.s. 7539MB, on two kinds of GPUs?</p>
<p>Also, I do not understand why the algorithm consults much more memory since the second backward compared with the first backward on TITAN X, while it consults the same memory no matter which time backward on GeForce GTX.</p>

falmasri: <p>I have same problem</p><aside class="quote" data-post="1" data-topic="37863">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falmasri/48/11265_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/strange-behavior-in-memory-usage/37863">Strange behavior in memory usage</a> 
  </div>
  <blockquote>
    I’m running the same model at the same time each one of them on different GPU. 
memory usage on Titan XP is 11665 MB (96%) 
memory usage on Titan X is 4047 MB (33%) 
“model convergence also not the same”
  </blockquote>
</aside>


talkhaldi: <p>I have the same problem. When I load the same transformer model (Roberta-Large)  on Titan X and V100 I get vastly different results:</p>
<ul>
<li>Gradient Checkpointing True, fp16 off, TitanX:   8034G</li>
<li>Gradient Checkpointing True, fp16 off, V100:    10861G</li>
<li>Gradient Checkpointing True, fp16 on, V100:    10613G</li>
</ul>
<p>I tried with same cuda/cudnn on both machines (Tried Cuda 10.1/10.2, and cudnn 7605/7603). I also tried both pytorch 1.6 and 1.3. It seems there are some Volta specific kernels that consume more memory on V100?  Does anyone know how to reduce the GPU memory consumption on the Volta cards? I don’t mind sacrificing some speed in my use case.</p>

ptrblck: <p>You could try to disable cudnn via <code>torch.backends.cudnn.enabled = False</code> at the beginning of your script in order to use the PyTorch-native implementations.</p>

talkhaldi: <p>I tried it right after import torch, the GPU memory consumption is exactly the same (digit by digit). Any other advices?</p>

ptrblck: <p>In that case your initial code doesn’t seem to have used architecture-specific kernels.<br>
You could trade compute for memory by using e.g. <code>torch.utils.checkpoint</code> or by lowering the batch size, if this fits your use case.</p>

talkhaldi: <p>Thanks for the quick reply!</p>
<p>My batch size is already 1, so no luck there. torch.utils.checkpoint is interesting! My model consists of transformers and classification heads on top. Transformers are already using gradient_checkpointing, but I’ll try to checkpoint the classification heads now!</p>
<p>Thanks for the tip <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

talkhaldi: <p>A quick question about torch.utils.checkpoint.</p>
<p>My classification head is:</p>
<pre><code>class ClassificationHead(nn.Module):
  def __init__(self, config):
    super().__init__()
    self.dense = nn.Linear(config.hidden_size, config.hidden_size)
    self.dropout = nn.Dropout(config.hidden_dropout_prob)
    self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
 forward(self, inputs):
   ....
</code></pre>
<p>I use it in another bigger model:</p>
<pre><code>sf_logits = checkpoint(self.sf_classifier, sequence_output)
</code></pre>
<p>But I’m getting an error: “TypeError: ‘module’ object is not callable”.</p>
<p>In the documentation, it doesn’t say what is the type of the function to be passed.</p>

talkhaldi: <p>Nevermind, a stupid mistake on my part ^^’</p>
<p>I was doing:</p>
<pre><code>from torch.utils import checkpoint
</code></pre>
<p>instead of:</p>
<pre><code>from torch.utils.checkpoint import checkpoint
</code></pre>
<p>It was a good try, but it only saved 5Mb in my case.</p>
<p>I still appreciate it, thanks!</p>