Why backward is very slow if the code is excuted on 3080 compared to 2080?

Shanyan_Guan: <p>I met a strange problem that the same code executed on 3080 is much slower than on 2080.</p>
<p>Detailed environments on 3080:<br>
CUDA 11.1<br>
Pytorch 1.8<br>
learn2learn 0.1.5</p>
<p>Detailed environments on 2080:<br>
CUDA 10.2<br>
Pytorch 1.1<br>
learn2learn 0.1.3</p>
<p>My code clip can be summarized as:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
learner = model.clone()
loss1 = torch.nn.MSE(learner(x), y)
learner.adapt(loss1)

loss2 = torch.nn.MSE(learner(x), y)
loss2.backward()
optimizer.step()
</code></pre>
<p>And I found most of the time is spent on the <code>loss2.backward()</code> step. Can anyone provide hints about this problem? Maybe the reason is pytorch 1.8 has changed some features related to clone()?</p>

Shanyan_Guan: <p>Some snapthot is avaiable on <a href="https://github.com/learnables/learn2learn/issues/233" rel="noopener nofollow ugc">this site</a></p>

ptrblck: <p>Your 3080 might see a performance regression due to an older cudnn version (you would be using 8.0.5) and/or due to <a href="https://github.com/pytorch/pytorch/issues/50153">this issue</a>.<br>
While we are still working on fixing both issues (first the static linking then updating cudnn, if no major regressions are found), you could build from source using the latest cudnn (and CUDA) version and compare the performance.</p>

Shanyan_Guan: <p>Thanks for your quick reply!<br>
The version of installed pytorch is <code>py3.6_cuda11.1_cudnn8.0.5_0</code> . Should I upgrade the cudnn version?</p>

ptrblck: <p>You could update cudnn, but would need to build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>

Shanyan_Guan: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I built pytorch from source using CUDA11.1 and cudnn8.1.1. But the problem still exists.</p>
<p>I find that when the code is executed on 3080, GPU-Util is 100%. However, it is 20% when the code is executed on 2080.</p>
<p>The snapshot (on 3080):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e80dc4d484a7020005a39e126d4c33a0e14b33d.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e80dc4d484a7020005a39e126d4c33a0e14b33d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e80dc4d484a7020005a39e126d4c33a0e14b33d_2_690x127.png" alt="image" data-base62-sha1="kkDSLhtOeD2raIes2hl9sx4dHCB" width="690" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e80dc4d484a7020005a39e126d4c33a0e14b33d_2_690x127.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e80dc4d484a7020005a39e126d4c33a0e14b33d_2_1035x190.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e80dc4d484a7020005a39e126d4c33a0e14b33d_2_1380x254.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1750×324 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Shanyan_Guan: <p>The bug is located in the following code of <a href="https://github.com/learnables/learn2learn/blob/master/learn2learn/utils/__init__.py#L51" rel="noopener nofollow ugc">this function</a>. However, I cannot dig it out…</p>
<pre><code class="lang-auto">    # Then, recurse for each submodule
    for module_key in module._modules:
        module._modules[module_key] = update_module(
            module._modules[module_key],
            updates=None,
            memo=memo,
        )
</code></pre>

ptrblck: <p>I’m not sure I understand the last issue. Are you still seeing a slowdown on the 3080 compared to the 2080?<br>
If so, could you post a code snippet, which you’ve used to profile the workload (and which we could execute to reproduce it using random inputs)?<br>
Do you think the method from your last post is causing a high GPU utilization?</p>

Shanyan_Guan: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="115193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not sure I understand the last issue. Are you still seeing a slowdown on the 3080 compared to the 2080?</p>
</blockquote>
</aside>
<p>yes, it’s still slower on the 3080.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="115193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If so, could you post a code snippet, which you’ve used to profile the workload (and which we could execute to reproduce it using random inputs)?</p>
</blockquote>
</aside>
<p>Sorry, the project replies on complex environment, It’s not easy to post online.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="115193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Do you think the method from your last post is causing a high GPU utilization?</p>
</blockquote>
</aside>
<p>Yes. At first I thought it was an iteration problem, but I found that the most time-consuming step was to update the parameters (<a href="https://github.com/learnables/learn2learn/blob/master/learn2learn/utils/__init__.py#L112" rel="noopener nofollow ugc">Line 112- Line 122</a>).</p>

Shanyan_Guan: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , here is the cleaner code to reproduce the error. I found if I delete code related to <code>self.register_buffer</code>, it works fine.</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn
from tqdm import tqdm
import learn2learn as l2l
import torchvision.models.resnet as resnet

def set_dropout_eval(m):
    classname = m.__class__.__name__
    if classname.find('Dropout') != -1:
        m.eval()

def freeze_dropout(model):
    model.apply(set_dropout_eval)
    return model

class Model(nn.Module):
    def __init__(self,):
        super(Model, self).__init__()
        self.model = resnet.resnet50(pretrained=True)
        init_pose = torch.randn(1, 3,256,256)
        self.register_buffer('init_pose', init_pose)
        self.register_buffer('init_pose2', init_pose)
        self.register_buffer('init_pose3', init_pose)
    
    def forward(self, x):
        batch_size = x.shape[0]
        init_pose = self.init_pose.expand(batch_size, -1, -1, -1)
        init_pose2 = self.init_pose2.expand(batch_size, -1, -1, -1)
        init_pose3 = self.init_pose3.expand(batch_size, -1, -1, -1)

        x = x + init_pose
        x = x + init_pose2
        x = x + init_pose3
        
        out = self.model(x)
        return out

device = torch.device('cuda')
model = Model()
model = l2l.algorithms.MAML(model, lr=0.001, first_order=False).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001)

X = torch.randn(16,3,256,256).to(device)
Y = torch.randn(16,1000).to(device)
criterion = nn.MSELoss().to(device)

# traning
for i in tqdm(range(10000), total=10000):
    model.train()
    model = freeze_dropout(model)
    X = torch.randn(16,3,256,256).to(device)
    Y = torch.randn(16,1000).to(device)
    learner = model.clone()
    optimizer.zero_grad()
    # inner update
    loss1 = criterion(learner(X), Y)
    learner.adapt(loss1)
    # outer update
    loss2 = criterion(learner(X), Y)
    loss2.backward()
    optimizer.step()

</code></pre>

ptrblck: <p>I’m still unsure, how you are profiling the code, but note that CUDA operations are executed asynchronously, so you would have to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timers.<br>
To profile certain operations, you could use the <code>torch.utils.benchmark</code> utilities or have a look at <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59?u=ptrblck">this post</a> to profile the code with Nsight Systems to isolate the bottlenecks.</p>

Shanyan_Guan: <p>Thanks! Here is the revised code.</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn
from tqdm import tqdm
import learn2learn as l2l
import torchvision.models.resnet as resnet

def set_dropout_eval(m):
    classname = m.__class__.__name__
    if classname.find('Dropout') != -1:
        m.eval()

def freeze_dropout(model):
    model.apply(set_dropout_eval)
    return model

class Model(nn.Module):
    def __init__(self,):
        super(Model, self).__init__()
        self.model = resnet.resnet50(pretrained=True)
        self.init_pose = torch.randn(1, 3,256,256).cuda()
        self.register_buffer('init_pose', init_pose)
        self.register_buffer('init_pose2', init_pose)
        self.register_buffer('init_pose3', init_pose)
    
    def forward(self, x):
        batch_size = x.shape[0]
        init_pose = self.init_pose.expand(batch_size, -1, -1, -1)
        init_pose2 = self.init_pose2.expand(batch_size, -1, -1, -1)
        init_pose3 = self.init_pose3.expand(batch_size, -1, -1, -1)

        x = x + init_pose
        x = x + init_pose2
        x = x + init_pose3
        
        out = self.model(x)
        return out

device = torch.device('cuda')
# model = Model()
model = resnet.resnet50(pretrained=True)
model = l2l.algorithms.MAML(model, lr=0.001, first_order=False).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001)

X = torch.randn(16,3,256,256).to(device)
Y = torch.randn(16,1000).to(device)
criterion = nn.MSELoss().to(device)

warmup_iters = 10
nb_iters = 20
# traning
torch.cuda.synchronize()
for i in tqdm(range(200), total=200):
    model.train()
    model = freeze_dropout(model)
    X = torch.randn(16,3,256,256).to(device)
    Y = torch.randn(16,1000).to(device)
    learner = model.clone()
    optimizer.zero_grad()

    # start profiling after 10 warmup iterations
    if i == warmup_iters: torch.cuda.cudart().cudaProfilerStart()

    # push range for current iteration
    if i &gt;= warmup_iters: torch.cuda.nvtx.range_push("iteration{}".format(i))

    # push range for forward
    if i &gt;= warmup_iters: torch.cuda.nvtx.range_push("forward")
    # inner update
    loss1 = criterion(learner(X), Y)
    learner.adapt(loss1)
    # outer update
    loss2 = criterion(learner(X), Y)
    if i &gt;= warmup_iters: torch.cuda.nvtx.range_pop()
    
    if i &gt;= warmup_iters: torch.cuda.nvtx.range_push("backward")
    loss2.backward()
    if i &gt;= warmup_iters: torch.cuda.nvtx.range_pop()

    if i &gt;= warmup_iters: torch.cuda.nvtx.range_push("opt.step()")
    optimizer.step()
    if i &gt;= warmup_iters: torch.cuda.nvtx.range_pop()

    torch.cuda.synchronize()

torch.cuda.cudart().cudaProfilerStop()
</code></pre>

Shanyan_Guan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, would you help me to figure it out? <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

ptrblck: <p>Sure! What is the new code snippet showing for each operation on both setups?</p>