SentenceBERT cuda out of memory problems

Ra-V: <p>Hello, I have cuda memory problems while trying to fine tune Siamese BERT on quora question dataset. I am using SentenceTransformers library (<a href="https://github.com/UKPLab/sentence-transformers" rel="nofollow noopener">https://github.com/UKPLab/sentence-transformers</a>). I launched VM on GCP with 4 GPUS (NVIDIA TESLA P100), and I tried to fine tune it with following code:</p>
<pre><code class="lang-auto">train_batch_size = 64
model_name = 'bert-base-nli-mean-tokens'
num_epochs = 4
model_save_path = 'output/training_stsbenchmark_continue_training-'+model_name+'-'+datetime.now().strftime("%Y-%m-%d_%H-%M-%S")

warmup_steps = math.ceil(len(train_data)*num_epochs/train_batch_size*0.1) #10% of train data for warm-up
logging.debug("Warmup-steps: {}".format(warmup_steps))


# Training the model
model.fit(train_objectives=[(train_dataloader, train_loss)],
          evaluator=evaluator,
          epochs=num_epochs,
          evaluation_steps=1000,
          warmup_steps=warmup_steps,
          output_path=model_save_path)
</code></pre>
<p>after 342 iterations I got following error:</p>
<pre><code class="lang-auto">/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, hidden_states)
    326     def forward(self, hidden_states):
    327         hidden_states = self.dense(hidden_states)
--&gt; 328         hidden_states = self.intermediate_act_fn(hidden_states)
    329         return hidden_states
    330 

/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in gelu(x)
    131         Also see https://arxiv.org/abs/1606.08415
    132     """
--&gt; 133     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
    134 
    135 

RuntimeError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 15.90 GiB total capacity; 13.44 GiB already allocated; 13.88 MiB free; 1.80 GiB cached)
</code></pre>
<p>I tried using more GPUs but it always failed, and I started to wonder if maybe there is a problem with not optimal memory allocation in the SentenceTransformers library. On the other hand, BERT model is pretty huge, so considering its siamese model it can be just too big, and I might need to use even more GPUs or distributed training. I’d be really grateful for every suggestion <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I’m not familiar with the mentioned repository, but by just skimming through the code it seems multiple GPUs won’t be used?<br>
The <code>fit()</code> function points to <a href="https://github.com/UKPLab/sentence-transformers/blob/b307063782ae2f1014a73262e1d29c910f9b4cf3/sentence_transformers/SentenceTransformer.py#L292">this line of code</a>, which will only use the default device.</p>
<p>You could try to lower the batch size and see, if the model still converges as you wish.<br>
Also, you could try to use <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a> to trade compute for memory, but I’m not sure how easy it would be to implement it in this particular repository, as some higher level abstractions seem to be used.</p>

Ra-V: <p>Thanks, I’m not really familiar with the topic of using multiple GPUs, I thought that this syntax by default splits onto every available GPU. So if I’d like to use all, I should be moving it to every device explicitly, or use <code>model = nn.DataParallel(model)</code>? In case of repository, I think I will have to fork it and change just my nessecary functions/classes, and maybe later on work on it in depth</p>

Ra-V: <p>I tried using <code>nn.DataParallel</code>, but I ran into the same problem. The diagnostics of GPUs looks like this:</p>
<pre><code class="lang-auto">
Sun Jan 26 12:16:30 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   49C    P0    36W / 250W |  16233MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  Off  | 00000000:00:05.0 Off |                    0 |
| N/A   40C    P0    28W / 250W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P100-PCIE...  Off  | 00000000:00:06.0 Off |                    0 |
| N/A   40C    P0    28W / 250W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P100-PCIE...  Off  | 00000000:00:07.0 Off |                    0 |
| N/A   42C    P0    25W / 250W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      7706      C   /opt/anaconda3/bin/python                  16223MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>after I used such syntax in fit method of the SentenceTransformer class:</p>
<pre><code class="lang-auto">        if device is None:
            device = "cuda" if torch.cuda.is_available() else "cpu"
            logging.info("Use pytorch device: {}".format(device))
        self.device = torch.device(device)
        self = nn.DataParallel(self)
        self.to(device)
</code></pre>
<p>It looks like everything is still passed only to first GPU.</p>
<p>EDIT: I added nn.DataParallel in appropriate block of code (for every loss_model in loss_models, with appropriate changes in them so that they work) and it works, models are being split onto every GPU <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

chaochaoSZ: <p>Hi, Ra-V. Can you show more detail about the loss_model changes?</p>

Ra-V: <p>Sure, I think everything is in this commit in data_parallel branch of my fork of original repository:<br>
<aside class="onebox githubcommit">
  <header class="source">
      <a href="https://github.com/rafaljanwojcik/sentence-transformers/commit/b8b493d466498e8facae90e9bc4da6cd29453e11" target="_blank" rel="nofollow noopener">github.com/rafaljanwojcik/sentence-transformers</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/rafaljanwojcik/sentence-transformers/commit/b8b493d466498e8facae90e9bc4da6cd29453e11" target="_blank" rel="nofollow noopener">Merge branch 'data_parallel' of https://github.com/rafaljanwojcik/sentence-transformers into data_parallel</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2020-01-26" data-time="15:31:18" data-timezone="UTC">03:31PM - 26 Jan 20 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/rafaljanwojcik" target="_blank" rel="nofollow noopener">
          <img alt="rafaljanwojcik" src="https://avatars0.githubusercontent.com/u/48239833?v=4" class="onebox-avatar-inline" width="20" height="20">
          rafaljanwojcik
        </a>
        
      </div>

      <div class="lines" title="changed 2 files with 17 additions and 43 deletions">
        <a href="https://github.com/rafaljanwojcik/sentence-transformers/commit/b8b493d466498e8facae90e9bc4da6cd29453e11" target="_blank" rel="nofollow noopener">
          <span class="added">+17</span>
          <span class="removed">-43</span>
        </a>
      </div>
    </div>

  </div>
</div>




  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Essentially, I changed these lines of code:</p>
<pre><code class="lang-auto">92        self = nn.DataParallel(self)
93        self.to(device)
...
294      device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
295      for loss_model in loss_models:
296      loss_model.to(device)
</code></pre>
<p>to:</p>
<pre><code class="lang-auto">300  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
301  loss_models = list(map(lambda x: nn.DataParallel(x), loss_models))
302  for loss_model in loss_models:
303      loss_model.to(device)
</code></pre>
<p>I also changed the CosineSimilarityLoss, but it was mainly because of different training objective I wanted to use, so it’s not nessecary for multiple GPUs. Let me know if you need help, it’s a bit messy but it works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

chaochaoSZ: <p>Hi, Ra-V. I have read your committed. Only see the change in:</p>
<p>self.device = torch.device(device)<br>
self = nn.DataParallel(self)<br>
self.to(device)</p>
<p>I don’t see the change of  device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”).<br>
Should I add these by myself?</p>
<p>Thank you.</p>

Ra-V: <p>Hello, I’m not sure if you’ve read the same commit, there is a line I pasted above, here is a screenshot of it:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d56c13d70e54fc5bfb8a53d7123180acb2bbeb28.png" data-download-href="https://discuss.pytorch.org/uploads/default/d56c13d70e54fc5bfb8a53d7123180acb2bbeb28" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d56c13d70e54fc5bfb8a53d7123180acb2bbeb28_2_689x462.png" alt="image" data-base62-sha1="us1fSLljPDmga4VqggcsoyW4fb2" width="689" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d56c13d70e54fc5bfb8a53d7123180acb2bbeb28_2_689x462.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d56c13d70e54fc5bfb8a53d7123180acb2bbeb28_2_1033x693.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d56c13d70e54fc5bfb8a53d7123180acb2bbeb28_2_1378x924.png 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2461×1651 318 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>there is a line 300 with code for setting up device</p>
<pre><code class="lang-auto"> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
</code></pre>
<p>If this is what you’re asking about <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Ra-V: <p>Hello, could you link to/paste whole code with your ‘training_nli_bert.py’ script? There is a problem with shapes of tensors you pass (outputs of your model have different dimensions than your targets)</p>

chaochaoSZ: <p>Thanks, sir. Now the problem have been solved.</p>

Ra-V: <p>Great, good luck on the project!</p>

samm: <p>excuse me what if I have a single GPU . is there any solution for it ?</p>