[amp]automatic mixed precision training slower than the normal model

username: Hu_Penglong
cooked: <p>i’m trying   to use the  automatic mixed precision training  to speed update the training speed.But it seems i get the opposite result. The pytorch is 1.6, cuda is 10.1, gpu is Tesla k80 and Tesla T4,cudnn 8.0.2</p>
<p>the baseline version code is</p>
<pre><code class="lang-auto">import torch

N, D_in, D_out = 64, 1024, 512
x = torch.randn(N, D_in, device="cuda")
y = torch.randn(N, D_out, device="cuda")
model = torch.nn.Linear(D_in, D_out).cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
for to in range(500):
    y_pred = model(x)
    loss = torch.nn.functional.mse_loss(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>time is<br>
real    0m3.402s<br>
user    0m2.926s<br>
sys     0m4.955s</p>
<p>the amp version code is</p>
<pre><code class="lang-auto">import torch
from  torch.cuda.amp import autocast, GradScaler
N, D_in, D_out = 64, 1024, 512
x = torch.randn(N, D_in, device="cuda")
y = torch.randn(N, D_out, device="cuda")
scaler = GradScaler()
model = torch.nn.Linear(D_in, D_out).cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
for to in range(500):
    with autocast():
        y_pred = model(x)
        loss = torch.nn.functional.mse_loss(y_pred, y)
    scaler.scale(loss).backward()
    optimizer.zero_grad()
    scaler.step(optimizer)
    scaler.update()
</code></pre>
<p>the time is<br>
real    0m3.584s<br>
user    0m3.131s<br>
sys     0m4.832s</p>

username: Hu_Penglong
cooked: <p>Is it because，Tesla 80k do not have Tensor Core? and amp must have tensor core support.<br>
I change a Tesla T4, get the same result.</p>

username: ptrblck
cooked: <p>I would recommend to profile the code directly by synchronizing the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timer and calculate the average time. Currently you are also profiling the startup time etc.</p>
<aside class="quote no-group" data-username="Hu_Penglong" data-post="2" data-topic="102614">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hu_penglong/48/13528_2.png" class="avatar"> Hu_Penglong:</div>
<blockquote>
<p>Is it because，Tesla 80k do not have Tensor Core?</p>
</blockquote>
</aside>
<p>Yes, this would explain the slowdown on the K80 due to the added overhead in amp.</p>
<aside class="quote no-group" data-username="Hu_Penglong" data-post="2" data-topic="102614">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hu_penglong/48/13528_2.png" class="avatar"> Hu_Penglong:</div>
<blockquote>
<p>I change a Tesla T4, get the same result.</p>
</blockquote>
</aside>
<p>I’ll try to grab a node with a T4 to reproduce it.</p>

username: Hu_Penglong
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your reply.  T4 is slow , i guess it is because the batch_size is too small. I reproduce the code in <a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html</a> .<br>
The conclusion is mixed precision is slow in Tesla k80, and faster in T4.<br>
I wanna to speed up the interference speed, and i only have k80 dev environment and t4 product environment. it is ok to use Tesla k80 to train and use T4 to interference in amp ?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Hu_Penglong" data-post="4" data-topic="102614">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hu_penglong/48/13528_2.png" class="avatar"> Hu_Penglong:</div>
<blockquote>
<p>it is ok to use Tesla k80 to train and use T4 to interference in amp ?</p>
</blockquote>
</aside>
<p>Yes, that should be OK. There is no requirement to use the same GPU for inference and training.</p>

username: Hu_Penglong
cooked: <p>But the k80 do not support fp16.</p>

username: ptrblck
cooked: <p>It shouldn’t matter or are you seeing different results on the T4 in comparison to the K80?</p>

username: Hu_Penglong
cooked: <p>In interference the result is bad when i used torch.cuda.amp.autocast();<br>
if i remove the code, the result is right.<br>
I guess it is because i can’t use mixed precision training in Tesla 80k.</p>

username: ptrblck
cooked: <p>Could you compare the absolute and relative error on this device with and without <code>autocast</code>?</p>

username: Hu_Penglong
cooked: <p>actually i’m running a gpt2lm model, the diff is big.</p>