AMP autocast not faster than FP32

username: krishansubudhi
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e555d4b7436c74e34e7ab53094473034b5aa436c.png" data-download-href="https://discuss.pytorch.org/uploads/default/e555d4b7436c74e34e7ab53094473034b5aa436c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e555d4b7436c74e34e7ab53094473034b5aa436c_2_566x500.png" alt="image" data-base62-sha1="wINeqf8WJQavYlyklmdAaNHQU8I" width="566" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e555d4b7436c74e34e7ab53094473034b5aa436c_2_566x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e555d4b7436c74e34e7ab53094473034b5aa436c_2_849x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e555d4b7436c74e34e7ab53094473034b5aa436c_2_1132x1000.png 2x" data-dominant-color="171819"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1445×1276 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so you would have to synchronize the code before starting and stopping the timer. The <code>torch.utils.benchmark</code> utility provides this functionality as well as e.g. warmup iterations.</p>

username: krishansubudhi
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="111757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.utils.benchmark</p>
</blockquote>
</aside>
<p>Still not seeing speed up.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.cuda.amp import autocast
print(torch.__version__)

!nvidia-smi

from transformers import AutoModel
model = AutoModel.from_pretrained('bert-base-uncased').to(1)
model.train()

x=torch.randint(0,1000, (8,128)).to(1)

def autocast_forward(model, x, fp16 = False):
    with autocast(enabled = fp16):
        return model.forward(x)

</code></pre>
<pre data-code-wrap="python"><code class="lang-python">
import torch.utils.benchmark as benchmark

t0 = benchmark.Timer(
    stmt='autocast_forward(model, x, False)',
    setup='from __main__ import autocast_forward',
    globals={'x': x, 'model':model})

t1 = benchmark.Timer(
    stmt='autocast_forward(model, x, True)',
    setup='from __main__ import autocast_forward',
    globals={'x': x, 'model':model})

print(t0.timeit(100))
print(t1.timeit(100))
</code></pre>
<pre><code>&lt;torch.utils.benchmark.utils.common.Measurement object at 0x7f9e30ccd520&gt;
autocast_forward(model, x, False)
  19.95 ms
  1 measurement, 100 runs , 1 thread
&lt;torch.utils.benchmark.utils.common.Measurement object at 0x7f9d9fa651c0&gt;
autocast_forward(model, x, True)
  23.53 ms
  1 measurement, 100 runs , 1 thread
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">
</code></pre>

username: ptrblck
cooked: <p>The 15% speedup seems to be low, as the FP16 kernels are not fully saturating the GPU, which is visible in the Nsight Systems profile in the whitespaces between kernel calls. While the FP16 cublas kernels are faster, the overall speedup is lower due to this.</p>

username: krishansubudhi
cooked: <p>I don’t see 15% speedup. autocast = false is faster than autocast=true</p>

username: ptrblck
cooked: <p>Ah, thanks. I’ve misunderstood the timings, but saw a minor speedup on an A100 using your benchmark.<br>
In any case, it seems the workload is heavily CPU bound. Also there are some unexpected <code>device_to_device</code> kernels, so I’ll look into it a bit more.</p>

username: aminsaied
cooked: <p>For what it’s worth - I reproed this on Tesla M60 GPU and saw the same behavior as OP - that with autocast enabled forward pass is marginally <em>slower</em> than without.</p>
<pre><code class="lang-auto"># fp16=True
  92.23 ms
  1 measurement, 100 runs , 1 thread
# fp16=False
  79.00 ms
  1 measurement, 100 runs , 1 thread
</code></pre>
<p>I also tried including backward pass, and observed the times were very similar:</p>
<pre><code class="lang-auto">model = BertForSequenceClassification.from_pretrained("bert-base-uncased").to(0)

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt").to(0)
labels = torch.tensor([1]).unsqueeze(0).to(0)

def autocast_backward(model, inputs, labels, fp16=False):
    with autocast(enabled=fp16):
        outputs = model(**inputs, labels=labels)
        loss = outputs.loss
        loss.backward()

t0 = benchmark.Timer(
    stmt="autocast_backward(model, x, False)",
    setup="from __main__ import autocast_backward",
    globals={"inputs": inputs, "model": model, "labels": labels,},
)

t1 = benchmark.Timer(
    stmt="autocast_backward(model, x, True)",
    setup="from __main__ import autocast_backward",
    globals={"inputs": inputs, "model": model, "labels": labels,},
)

print(t1.timeit(100))
print(t0.timeit(100))
</code></pre>
<p>gave</p>
<pre><code class="lang-auto"># fp16=True
  304.61 ms
  1 measurement, 100 runs , 1 thread
# fp16=False
  304.99 ms
  1 measurement, 100 runs , 1 thread
</code></pre>

username: krishansubudhi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  any updates on the issue?</p>

username: jsleep
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> , have you gotten a chance to look into this? Anyone working with transformers is better off using apex amp over native amp until there’s some kind of fix for this here.</p>

username: ptrblck
cooked: <p>Sorry for the late reply.<br>
I’ve reran the transformer model with a source build using the latest CUDA version and see a speedup on the V100s you are using as: AMP: 16.79ms, FP32: 20.02ms, so I assume cublas might have updated their heuristics and pick faster (TC) kernels.</p>
<p><a class="mention" href="/u/aminsaied">@aminsaied</a> I wouldn’t expect to see a large difference between AMP and FP32 on Maxwell GPUs.</p>
<p><a class="mention" href="/u/jsleep">@jsleep</a> I don’t agree, as apex/amp is deprecated and used the same autocast list in the then recommended <code>O1</code> mode.</p>

username: jadeantonis
cooked: <p>I’m having the same issue on the A6000:</p>
<pre><code class="lang-auto">import torch
from torch.cuda.amp import autocast

from transformers import AutoModel
model = AutoModel.from_pretrained('bert-base-uncased').to(1)
model.train()

x=torch.randint(0,1000, (8,128)).to(1)

def autocast_forward(model, x, fast_dtype=None):
    if fast_dtype is None:
        return model.forward(x)

    with autocast(fast_dtype=fast_dtype):
        return model.forward(x)

import torch.utils.benchmark as benchmark

t0 = benchmark.Timer(
    stmt='autocast_forward(model, x)',
    setup='from __main__ import autocast_forward',
    globals={'x': x, 'model':model})

t1 = benchmark.Timer(
    stmt='autocast_forward(model, x, fast_dtype=torch.float16)',
    setup='from __main__ import autocast_forward',
    globals={'x': x, 'model':model})

t2 = benchmark.Timer(
    stmt='autocast_forward(model, x, fast_dtype=torch.bfloat16)',
    setup='from __main__ import autocast_forward',
    globals={'x': x, 'model':model})

print(t0.timeit(1000))  # prints 8.91 ms
print(t1.timeit(1000))  # prints 9.18 ms
print(t2.timeit(1000))  # prints 9.13 ms
</code></pre>
<p>My goal is to use bf16, not fp16, but the issue is there for both</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> is there any new insight on this?</p>

username: jadeantonis
cooked: <p>and for backward pass:</p>
<pre><code class="lang-auto">from torch.utils import benchmark
import torch
from torch.cuda.amp import autocast
from transformers import BertForSequenceClassification, BertTokenizer

model = BertForSequenceClassification.from_pretrained("bert-base-uncased").to(0)

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt").to(0)
labels = torch.tensor([1]).unsqueeze(0).to(0)

def autocast_backward(model, inputs, labels, fast_dtype=None):
    if fast_dtype is None:
        outputs = model(**inputs, labels=labels)
        loss = outputs.loss
        loss.backward()
        return

    with autocast(fast_dtype=fast_dtype):
        outputs = model(**inputs, labels=labels)
        loss = outputs.loss
        loss.backward()

    return

t0 = benchmark.Timer(
    stmt="autocast_backward(model, inputs, labels)",
    setup="from __main__ import autocast_backward",
    globals={"inputs": inputs, "model": model, "labels": labels,},
)

t1 = benchmark.Timer(
    stmt="autocast_backward(model, inputs, labels, torch.float16)",
    setup="from __main__ import autocast_backward",
    globals={"inputs": inputs, "model": model, "labels": labels,},
)
t2 = benchmark.Timer(
    stmt="autocast_backward(model, inputs, labels, torch.bfloat16)",
    setup="from __main__ import autocast_backward",
    globals={"inputs": inputs, "model": model, "labels": labels,},
)

print(t0.timeit(100))  # prints 20.52
print(t1.timeit(100))  # prints 22.33
print(t2.timeit(100))  # prints 26.43 :(
</code></pre>

username: ptrblck
cooked: <p>Your A6000 would use TF32 by default and would thus already speedup your wokload using TensorCores. <a href="https://github.com/pytorch/pytorch/issues/57806">This post</a> has additional information, but skip the “channels-last” part since you are working on a language model.<br>
Additionally, <a href="https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525/2">this post</a> discusses a similar issue and provides a profile of the workload.</p>