Fp16 training with feedforward network slower time and no memory reduction

Najeeb_Nabwani: <p>Hello,<br>
I’m doing mixed-precision training (from the native amp in pytorch 1.6) on feedforward neural networks. Both the training time and memory consumed have increased as a result.</p>
<p>The GPU is RTX 2080Ti. I tried to have all of the dimensions in multiples of 8 as well.</p>
<p>The training time is less important to me, I mainly want to decrease the memory footprint as much as possible since I’m using large feedforward neural networks only.</p>
<p>Thanks.</p>

Henry_Chibueze: <p>Well if ur memory consumption is so high that it irritates u then I suggest u downsample ur data with convolutional layers (if the problem is not a regression model problem) and try to update most of ur code variables inplace</p>

Najeeb_Nabwani: <p>Not really applicable in my situation. But I’m wondering more about why fp16 isn’t reducing my memory at all…</p>

Henry_Chibueze: <p>The good effects of half precision floating points may just be negligible in this case<br>
U said u were using mixed precision right? Then just making it a single precision might give u what u want I guess</p>

Najeeb_Nabwani: <p>Currently single precision is indeed faster. But I need fp16 primarily to reduce my memory footprint for when I want to run bigger networks.</p>

Henry_Chibueze: <p>The fp16 should still decrease ur memory foot print even if it’s by a small factor.<br>
It’s possible that the decrease is so small its negligible.</p>

Najeeb_Nabwani: <p>Unfortunately that’s exactly the problem, even on a model that takes 9GB, there is a memory <em>increase</em> when using fp16. I even tried pytorch lightning ‘precision=16’ (which uses native amp) and still no decrease. I’m guessing it’s related to the fact that I’m strictly using feedforward networks but I’m not entirely sure why.</p>

Henry_Chibueze: <p>It might be the feed forward networks though I’m not entirely sure<br>
Well what I’ll suggest is if u have the time try creating a conv network and in one Instance use single precision and in another use mix precision and check the difference in memory usage</p>

ptrblck: <p>Are you seeing a speedup or slowdown?<br>
I’m a bit confused, since you’ve mentioned both.</p>
<p>Also, how are you measuring the memory usage?<br>
Note that <code>nvidia-smi</code> shows the total allocated memory (CUDA context + cache + allocated memory), so you should use <code>torch.cuda.memory_allocated()</code> instead.</p>

Najeeb_Nabwani: <p>It seems that it doesn’t work well with feed forward. I don’t see a speedup nor reduced memory. However, when using other networks such as conv networks (as suggested) then there is indeed a significant speedup and memory reduction.</p>

seungjun: <p>torch.cuda.amp.autocast works in this way:</p>
<ol>
<li>cast the layer into fp16 if the corresponding operation is fp16-safe.<br>
<a href="https://pytorch.org/docs/stable/amp.html#autocast-op-reference" rel="nofollow noopener">https://pytorch.org/docs/stable/amp.html#autocast-op-reference</a><br>
For example, batch normalization will stay in fp32.</li>
<li>run in the configured precision levels<br>
If the layer input does not match the target precision, it will convert the type automatically.</li>
</ol>
<p>Using mixed-precision could sometimes be slow due to the type-casting operation.<br>
At inference time, using pure half-precision is faster than using amp.</p>
<p>But I’m not sure why your memory consumption is higher.<br>
I always see less memory consumption with mixed-precision training on RTX 2080 Ti.<br>
Maybe the above step 2 needs features in both fp32 and fp16 for precision switching layers.</p>
<p>You may want to refer to this <a href="https://arxiv.org/pdf/1710.03740.pdf" rel="nofollow noopener">paper</a>.</p>

stas: <p>Found a 10x memory increase with native fp16 (pt1.6) - no such problem with nvidia-apex (pt1.5)<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/huggingface/transformers/issues/8403" target="_blank" rel="noopener nofollow ugc">github.com/huggingface/transformers</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huggingface/transformers/issues/8403" target="_blank" rel="noopener nofollow ugc">[s2s finetune] huge increase in memory demands with --fp16</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-11-08" data-time="07:14:19" data-timezone="UTC">07:14AM - 08 Nov 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/stas00" target="_blank" rel="noopener nofollow ugc">
          <img alt="stas00" src="https://avatars2.githubusercontent.com/u/10676103?v=4" class="onebox-avatar-inline" width="20" height="20">
          stas00
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">While working on #8353 I discovered that --fp16 causes a 10x+ increase in gpu memory demands.
e.g. I can run bs=12 w/o...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>