In some condition, Conv3D outputs are very different according to Apex amp opt-level on V100 GPU

kkjh0723: <p>While training 3D CNN with Apex amp, I found very weird result (accuracy drops around 6%) specifically on pytorch version 1.4.0 , V100 gpu, and amp opt-level O1 and O2.<br>
On pytorch version 1.1.0, it worked correctly both on P40, V100 GPU with opt-level O2.<br>
On pytorch version 1.4.0, it worked correctly on P40 GPU with opt-level O2 (maybe also OK with O0).<br>
On pytorch version 1.4.0, it worked correctly on V100 GPU with opt-level O0.</p>
<p>So I compare every single module’s output in my model with opt-level O0 and O2 on cuda9.2, pytorch version 1.4.0, V100 GPU.<br>
And I found that Conv3D with kernel_size (3,1,1) outputs show large difference according to opt-level.<br>
So I wrote a code snippet that can observe the behavior as below.<br>
It seems that it happens when channel size is large.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import apex
from apex import amp

def init_model(in_channel=8, out_channel=4, kernel_size=(3, 1, 1), padding=(1, 0, 0), opt_level='O0'):
    model = nn.Conv3d(in_channel, out_channel, kernel_size=kernel_size, padding=padding, bias=False)
    model = model.cuda()
    parameters = model.parameters()
    optimizer = torch.optim.SGD(parameters,
                                    lr=0.1,
                                    )

    model, optimizer = amp.initialize(model, optimizer,
                                      opt_level=opt_level,
                                      keep_batchnorm_fp32=None if opt_level=='O1' else True,
                                      loss_scale=None
                                      )
    return model, optimizer

def compare_opt_level(in_channel, out_channel, kernel_size, padding, opt1='O0', opt2='O2'):
    input = torch.randn(size=(4, in_channel, 8, 16, 16)).cuda()

    n0, _ = init_model(in_channel, out_channel, kernel_size=kernel_size, padding=padding, opt_level=opt1)
    n2, _ = init_model(in_channel, out_channel, kernel_size=kernel_size, padding=padding, opt_level=opt2)
    init_param = torch.randn_like(n0.weight)
    #init_param = nn.init.kaiming_normal(n0.weight, mode='fan_out')
    n0.weight = nn.Parameter(init_param)
    n2.weight = nn.Parameter(init_param.half())

    v0 = n0(input)
    v2 = n2(input)

    print('/////////////////////////////')
    print('Compare {}/{}, in/out channel {}/{}, kernel size {}'.format(
        opt1, opt2, in_channel, out_channel, kernel_size))
    print(torch.sqrt((v0 - v2) ** 2).mean(dim=(1, 2, 3, 4)))

if __name__=='__main__':
    torch.random.manual_seed(1)
    ic, oc = 8, 4
    compare_opt_level(ic, oc, kernel_size=(3, 1, 1), padding=(1, 0, 0))
    compare_opt_level(ic, oc, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    compare_opt_level(ic, oc, kernel_size=(1, 1, 1), padding=(0, 0, 0))
    compare_opt_level(ic, oc, kernel_size=(1, 3, 3), padding=(0, 1, 1))
    
    ic, oc = 512, 128
    compare_opt_level(ic, oc, kernel_size=(3, 1, 1), padding=(1, 0, 0), opt1='O0', opt2='O2')
    compare_opt_level(ic, oc, kernel_size=(3, 3, 3), padding=(1, 1, 1), opt1='O0', opt2='O2')
    compare_opt_level(ic, oc, kernel_size=(1, 1, 1), padding=(0, 0, 0), opt1='O0', opt2='O2')
    compare_opt_level(ic, oc, kernel_size=(1, 3, 3), padding=(0, 1, 1), opt1='O0', opt2='O2')

    ic, oc = 512, 128
    compare_opt_level(ic, oc, kernel_size=(3, 1, 1), padding=(1, 0, 0), opt1='O0', opt2='O1')
    compare_opt_level(ic, oc, kernel_size=(3, 3, 3), padding=(1, 1, 1), opt1='O0', opt2='O1')
    compare_opt_level(ic, oc, kernel_size=(1, 1, 1), padding=(0, 0, 0), opt1='O0', opt2='O1')
    compare_opt_level(ic, oc, kernel_size=(1, 3, 3), padding=(0, 1, 1), opt1='O0', opt2='O1')
</code></pre>
<p>The printed MSE between O0 and O2 on V100 are,</p>
<pre><code class="lang-auto">Compare O0/O2, in/out channel 8/4, kernel size (3, 1, 1)
tensor([0.0016, 0.0017, 0.0016, 0.0016], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 8/4, kernel size (3, 3, 3)
tensor([0.0042, 0.0043, 0.0042, 0.0042], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 8/4, kernel size (1, 1, 1)
tensor([0.0007, 0.0007, 0.0007, 0.0007], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 8/4, kernel size (1, 3, 3)
tensor([0.0021, 0.0021, 0.0022, 0.0021], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (3, 1, 1)
tensor([34.6549, 34.6763, 34.7861, 34.7793], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (3, 3, 3)
tensor([0.0303, 0.0303, 0.0303, 0.0303], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (1, 1, 1)
tensor([0.0064, 0.0064, 0.0064, 0.0064], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (1, 3, 3)
tensor([0.0183, 0.0183, 0.0183, 0.0183], device='cuda:0',
       grad_fn=&lt;MeanBackward1&gt;)
</code></pre>
<p>You can see that the MSE is exceptionally large when kernel size is (3, 1, 1) with large channels</p>
<p>Unlike V100, the MSE is not so big on P40,</p>
<pre><code class="lang-auto">Compare O0/O2, in/out channel 8/4, kernel size (3, 1, 1)
tensor([0.0014, 0.0014, 0.0014, 0.0014], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 8/4, kernel size (3, 3, 3)
tensor([0.0038, 0.0037, 0.0037, 0.0037], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 8/4, kernel size (1, 1, 1)
tensor([0.0009, 0.0009, 0.0009, 0.0009], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 8/4, kernel size (1, 3, 3)
tensor([0.0023, 0.0023, 0.0023, 0.0023], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (3, 1, 1)
tensor([0.0106, 0.0106, 0.0106, 0.0106], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (3, 3, 3)
tensor([0.0305, 0.0303, 0.0303, 0.0304], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (1, 1, 1)
tensor([0.0064, 0.0064, 0.0064, 0.0064], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

Compare O0/O2, in/out channel 512/128, kernel size (1, 3, 3)
tensor([0.0184, 0.0184, 0.0183, 0.0183], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)

</code></pre>
<p>When I compare O0 and O1, MSEs are all zero.<br>
However the training was also not working properly with O1 on V100, so there might be some other issues.</p>
<p>I recently found that there is <code>torch.cuda.amp</code> in nightly version, but haven’t tested on it.<br>
I cannot move to <code>torch.cuda.amp</code> because it seems that O2 level is not provided yet.</p>

ptrblck: <p>Calling <code>amp.initialize</code> multiple times with different <code>opt_level</code>s is not supported, to you should split the test code into different runs using the same data (load the random tensors from disc and reinitialize the model with the same <code>state_dict</code>).<br>
Also, you shouldn’t call <code>half()</code> on any parameter, model, or data.</p>

kkjh0723: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, I tested as you suggested (on V100).<br>
Sorry that the modified codes below are not well organized.</p>
<p>I first save the <code>state_dict</code> and <code>input</code> in the disk as follows,</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import apex
from apex import amp

def init_model(in_channel=8, out_channel=4, kernel_size=(3, 1, 1), padding=(1, 0, 0), opt_level='O0'):
    model = nn.Conv3d(in_channel, out_channel, kernel_size=kernel_size, padding=padding, bias=False)
    model = model.cuda()
    parameters = model.parameters()
    optimizer = torch.optim.SGD(parameters,
                                    lr=0.1,
                                    )

    model, optimizer = amp.initialize(model, optimizer,
                                      opt_level=opt_level,
                                      keep_batchnorm_fp32=None if opt_level=='O1' else True,
                                      loss_scale=None
                                      )
    return model, optimizer

def compare_opt_level(in_channel, out_channel, kernel_size, padding, opt1='O0', opt2='O2'):
    input = torch.randn(size=(4, in_channel, 8, 16, 16)).cuda()

    n0, _ = init_model(in_channel, out_channel, kernel_size=kernel_size, padding=padding, opt_level=opt1)
    init_param = torch.randn_like(n0.weight)
    n0.weight = nn.Parameter(init_param)

    v0 = n0(input)

    save_dict = {'input': input.cpu(), 'state_dict':n0.state_dict()}
    torch.save(save_dict, 'ckpt_{}_{}_{}.pt'.format('_'.join(map(str, kernel_size)), in_channel, out_channel))

if __name__=='__main__':
    torch.random.manual_seed(1)
    ic, oc = 8, 4
    compare_opt_level(ic, oc, kernel_size=(3, 1, 1), padding=(1, 0, 0))
    compare_opt_level(ic, oc, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    compare_opt_level(ic, oc, kernel_size=(1, 1, 1), padding=(0, 0, 0))
    compare_opt_level(ic, oc, kernel_size=(1, 3, 3), padding=(0, 1, 1))
    
    ic, oc = 512, 128
    compare_opt_level(ic, oc, kernel_size=(3, 1, 1), padding=(1, 0, 0), opt1='O0', opt2='O2')
    compare_opt_level(ic, oc, kernel_size=(3, 3, 3), padding=(1, 1, 1), opt1='O0', opt2='O2')
    compare_opt_level(ic, oc, kernel_size=(1, 1, 1), padding=(0, 0, 0), opt1='O0', opt2='O2')
    compare_opt_level(ic, oc, kernel_size=(1, 3, 3), padding=(0, 1, 1), opt1='O0', opt2='O2')
</code></pre>
<p>Then, save the output of the model with different opt-level separately with following code,</p>
<pre><code class="lang-auto">import argparse
import torch
import torch.nn as nn
import apex
from apex import amp

def init_model(in_channel=8, out_channel=4, kernel_size=(3, 1, 1), padding=(1, 0, 0), opt_level='O0', ckpt=None):
    model = nn.Conv3d(in_channel, out_channel, kernel_size=kernel_size, padding=padding, bias=False)
    model.load_state_dict(ckpt['state_dict'])
    model = model.cuda()
    parameters = model.parameters()
    optimizer = torch.optim.SGD(parameters,
                                    lr=0.1,
                                    )

    model, optimizer = amp.initialize(model, optimizer,
                                      opt_level=opt_level,
                                      keep_batchnorm_fp32=None if opt_level=='O1' else True,
                                      loss_scale=None
                                      )
    return model, optimizer

def get_result(in_channel, out_channel, kernel_size, padding, opt_level='O0'):
    checkpoint = torch.load('ckpt_{}_{}_{}.pt'.format('_'.join(map(str, kernel_size)), in_channel, out_channel))
    input = checkpoint['input'].cuda()

    n0, _ = init_model(in_channel, out_channel, kernel_size=kernel_size, padding=padding, opt_level=opt_level,
                       ckpt = checkpoint)
    v0 = n0(input)

    return v0


if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('--opt_level', type=str)
    args = parser.parse_args()

    result_dict = {}
    ic, oc = 8, 4
    ks = [[3, 1, 1], [3, 3, 3], [1, 1, 1], [1, 3, 3]]
    for k in ks:
        r = get_result(ic, oc, kernel_size=k, padding=[(i-1)//2 for i in k], opt_level=args.opt_level)
        result_dict['{}_{}_{}'.format('_'.join(map(str, k)), ic, oc)] = r

    ic, oc = 512, 128
    for k in ks:
        r = get_result(ic, oc, kernel_size=k, padding=[(i-1)//2 for i in k], opt_level=args.opt_level)
        result_dict['{}_{}_{}'.format('_'.join(map(str, k)), ic, oc)] = r

    torch.save(result_dict, 'result_{}'.format(args.opt_level))
</code></pre>
<p>Then compare the result files as follows,</p>
<pre><code class="lang-auto">import torch

if __name__ == '__main__':
    opt1 = 'O0'
    opt2 = 'O2'
    dict_O0 = torch.load('./result_'+opt1)
    dict_O2 = torch.load('./result_'+opt2)

    for k in sorted(dict_O0.keys()):

        v0 = dict_O0[k]
        v2 = dict_O2[k]
        k1, k2, k3, in_channel, out_channel = k.split('_')
        kernel_size = (k1, k2, k3)
        print('/////////////////////////////')
        print('Compare {}/{}, in/out channel {}/{}, kernel size {}'.format(
            opt1, opt2, in_channel, out_channel, kernel_size))
        print(torch.sqrt((v0 - v2) ** 2).mean(dim=(1, 2, 3, 4)))
</code></pre>
<p>And I got the similar result as before.</p>
<pre><code class="lang-auto">Compare O0/O2, in/out channel 512/128, kernel size ('1', '1', '1')
tensor([0.0064, 0.0064, 0.0064, 0.0064], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 8/4, kernel size ('1', '1', '1')
tensor([0.0008, 0.0009, 0.0009, 0.0009], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 512/128, kernel size ('1', '3', '3')
tensor([0.0184, 0.0184, 0.0184, 0.0183], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 8/4, kernel size ('1', '3', '3')
tensor([0.0023, 0.0023, 0.0023, 0.0023], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 512/128, kernel size ('3', '1', '1')
tensor([34.9032, 34.8360, 34.9023, 34.8901], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 8/4, kernel size ('3', '1', '1')
tensor([0.0016, 0.0016, 0.0016, 0.0016], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 512/128, kernel size ('3', '3', '3')
tensor([0.0303, 0.0303, 0.0304, 0.0303], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
/////////////////////////////
Compare O0/O2, in/out channel 8/4, kernel size ('3', '3', '3')
tensor([0.0043, 0.0042, 0.0043, 0.0042], device='cuda:0',
       grad_fn=&lt;MeanBackward2&gt;)
</code></pre>
<p>By the way, if I shouldn’t call <code>half()</code>, then how can I use some <code>torch</code> functions such as <code>torch.randn()</code> inside a module? I got an type error without <code>half()</code> in pytorch v1.1. Will it be different in v1.4?</p>

ptrblck: <p>Thanks for splitting up the code!<br>
We’ll try to reproduce this issue locally.</p>
<aside class="quote no-group" data-username="kkjh0723" data-post="3" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kkjh0723/48/3571_2.png" class="avatar"> kkjh0723:</div>
<blockquote>
<p>By the way, if I shouldn’t call  <code>half()</code> , then how can I use some  <code>torch</code>  functions such as  <code>torch.randn()</code>  inside a module? I got an type error without <code>half()</code> in pytorch v1.1.</p>
</blockquote>
</aside>
<p>I assume you are creating these tensors in the <code>forward</code> method? While <code>O1</code> and native amp should work out of the box, I’m unsure if <code>O2</code> would work and we would need to see a code snippet for it.</p>
<p>EDIT: I was able to reproduce and narrow down this issue to a faulty kernel in cudnn.<br>
I’ve also tested the next release candidate and the issue is gone, so my only advice for you is not to use this particular convolution in mixed-precision training. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

kkjh0723: <p>Is there any possible workaround? If I cannot used the kernel, then many recent 3D ResNets cannot be used with amp.<br>
And what do you mean next release candidate? next release of cudnn?  Do you have a schedule for the release?</p>
<p>About the example code snippet of using <code>torch.randn()</code>, there is an old github issue that I asked a couple of month ago. I found that you answered.<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/NVIDIA/apex/issues/483#issuecomment-530227518" target="_blank" rel="nofollow noopener">github.com/NVIDIA/apex</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVIDIA/apex/issues/483#issuecomment-530227518" target="_blank" rel="nofollow noopener">How to handle torch.randn while using mixed precision training</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-09-09" data-time="08:47:42" data-timezone="UTC">08:47AM - 09 Sep 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/kkjh0723" target="_blank" rel="nofollow noopener">
          <img alt="kkjh0723" src="https://avatars3.githubusercontent.com/u/10104262?v=4" class="onebox-avatar-inline" width="20" height="20">
          kkjh0723
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">My module uses torch.randn during fowrard and it induces error while using amp.
Currently, I just add x=x.half() when run the module...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Please check if you can reproduce with the modified code.

ptrblck: <aside class="quote no-group" data-username="kkjh0723" data-post="5" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kkjh0723/48/3571_2.png" class="avatar"> kkjh0723:</div>
<blockquote>
<p>And what do you mean next release candidate? next release of cudnn? Do you have a schedule for the release?</p>
</blockquote>
</aside>
<p>Yes, the next cudnn release. I don’t know exactly when it’s released.</p>
<aside class="quote no-group" data-username="kkjh0723" data-post="5" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kkjh0723/48/3571_2.png" class="avatar"> kkjh0723:</div>
<blockquote>
<p>If I cannot used the kernel, then many recent 3D ResNets cannot be used with amp.</p>
</blockquote>
</aside>
<p>That sounds bad. Do you have links to public repositories, which implement these models and could post them here, please?</p>
<aside class="quote no-group" data-username="kkjh0723" data-post="5" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kkjh0723/48/3571_2.png" class="avatar"> kkjh0723:</div>
<blockquote>
<p>Please check if you can reproduce with the modified code.</p>
</blockquote>
</aside>
<p>I answered in the issue and cannot reproduce the error with your provided code.<br>
Could you check, if you are still seeing this issue?</p>

kkjh0723: <p>So, only way is wait until the next cudnn is released (probably 7.7? or 8.0?) and pytorch supports the new release. Maybe I need to find some other model without using <code>(3,1,1)</code> kernel.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>That sounds bad. Do you have links to public repositories, which implement these models and could post them here, please?</p>
</blockquote>
</aside>
<p>There are several models that uses <code>conv3D</code> with (3, 1, 1) kernel.<br>
Most famous one might be SlowFast network from facebook and there are many follow up models.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/facebookresearch/SlowFast/blob/6bb1cd8a24da17252ad4450de3e61fa074b181ff/slowfast/models/resnet_helper.py#L185">
  <header class="source">

      <a href="https://github.com/facebookresearch/SlowFast/blob/6bb1cd8a24da17252ad4450de3e61fa074b181ff/slowfast/models/resnet_helper.py#L185" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/SlowFast/blob/6bb1cd8a24da17252ad4450de3e61fa074b181ff/slowfast/models/resnet_helper.py#L185" target="_blank" rel="noopener nofollow ugc">facebookresearch/SlowFast/blob/6bb1cd8a24da17252ad4450de3e61fa074b181ff/slowfast/models/resnet_helper.py#L185</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="175" style="counter-reset: li-counter 174 ;">
          <li>    num_groups,</li>
          <li>    dilation,</li>
          <li>    norm_module,</li>
          <li>):</li>
          <li>    (str1x1, str3x3) = (stride, 1) if self._stride_1x1 else (1, stride)</li>
          <li></li>
          <li>    # Tx1x1, BN, ReLU.</li>
          <li>    self.a = nn.Conv3d(</li>
          <li>        dim_in,</li>
          <li>        dim_inner,</li>
          <li class="selected">        kernel_size=[self.temp_kernel_size, 1, 1],</li>
          <li>        stride=[1, str1x1, str1x1],</li>
          <li>        padding=[int(self.temp_kernel_size // 2), 0, 0],</li>
          <li>        bias=False,</li>
          <li>    )</li>
          <li>    self.a_bn = norm_module(</li>
          <li>        num_features=dim_inner, eps=self._eps, momentum=self._bn_mmt</li>
          <li>    )</li>
          <li>    self.a_relu = nn.ReLU(inplace=self._inplace_relu)</li>
          <li></li>
          <li>    # 1x3x3, BN, ReLU.</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>R(2+1)D also uses the 3D conv with (3, 1, 1) kernel.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/facebookresearch/VMZ/blob/f4089e2164f67a98bc5bed4f97dc722bdbcd268e/lib/models/builder/video_model.py#L89">
  <header class="source">

      <a href="https://github.com/facebookresearch/VMZ/blob/f4089e2164f67a98bc5bed4f97dc722bdbcd268e/lib/models/builder/video_model.py#L89" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/VMZ/blob/f4089e2164f67a98bc5bed4f97dc722bdbcd268e/lib/models/builder/video_model.py#L89" target="_blank" rel="noopener nofollow ugc">facebookresearch/VMZ/blob/f4089e2164f67a98bc5bed4f97dc722bdbcd268e/lib/models/builder/video_model.py#L89</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="79" style="counter-reset: li-counter 78 ;">
          <li>        no_bias=self.no_bias,</li>
          <li>    )</li>
          <li>    self.add_spatial_bn(middle_filters, suffix='_middle')</li>
          <li>    self.add_relu()</li>
          <li>    self.prev_blob = self.model.ConvNd(</li>
          <li>        self.prev_blob,</li>
          <li>        '%scomp_%d_conv_%d' %</li>
          <li>        (self.prefix, self.comp_count, self.comp_idx),</li>
          <li>        middle_filters,</li>
          <li>        out_filters,</li>
          <li class="selected">        [kernels[0], 1, 1],</li>
          <li>        weight_init=("MSRAFill", {}),</li>
          <li>        strides=[strides[0], 1, 1] * 1,</li>
          <li>        pads=[pads[0], 0, 0] * 2,</li>
          <li>        no_bias=self.no_bias,</li>
          <li>    )</li>
          <li>elif block_type == '0.3d' or block_type == '0.3d+relu':</li>
          <li>    self.prev_blob = self.model.ConvNd(</li>
          <li>        self.prev_blob,</li>
          <li>        '%scomp_%d_conv_%d_middle' %</li>
          <li>        (self.prefix, self.comp_count, self.comp_idx),</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I answered in the issue and cannot reproduce the error with your provided code.<br>
Could you check, if you are still seeing this issue?</p>
</blockquote>
</aside>
<p>As for the last issue, It seems it related to the pytorch version. It works without the error in pytorch-v1.4.<br>
Now problem is I cannot use <code>amp</code> in pytorch-v1.4 because of the above issue.</p>

kkjh0723: <p>I tested 8 possible <code>kernel_size</code> from [1, 1, 1] to [3, 3, 3] and found that only [3, 1, 1] has the issue.<br>
So, as a workaround, I considered to use Conv3d with [1, 1, 3] kernel and transpose the input/output tensor before/after the convolution inside the forward method as below.</p>
<pre><code class="lang-auto">class Conv3d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True,
              padding_mode='zeros'):
        super(Conv3d, self).__init__()
        print('Use Conv3d Module')
        self.permute = False
        ks, sr, pad, dl = kernel_size, stride, padding, dilation
        if isinstance(kernel_size, (list, tuple)):
            if kernel_size == (3, 1, 1) or kernel_size == [3, 1, 1]:
                self.permute = True
                ks = ks[::-1]
                if isinstance(sr, (list, tuple)): sr = sr[::-1]
                if isinstance(pad, (list, tuple)): pad = pad[::-1]
                if isinstance(dl, (list, tuple)):  dl = dl[::-1]
        self.conv = nn.Conv3d(in_channels, out_channels, ks, stride=sr, padding=pad, dilation=dl,
                  groups=groups, bias=bias, padding_mode=padding_mode)

    def forward(self, x):
        if self.permute:
            return self.conv(x.permute(0,1,4,3,2)).permute(0,1,4,3,2)
        else:
            return self.conv(x)
</code></pre>
<p>It seems the MSE is reduced as normal in the previous test code.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>, do you think it can be a possible workaround? Will the permutation increase the computation time a lot?<br>
If the operation is functionally same as the original <code>Conv3d</code>, I’m considering testing it in my model, though the computation time will increase somewhat.</p>

ptrblck: <p>You approach should be valid and you could compare the current performance hit with a run without cudnn. If your current permutation is still faster than disabling cudnn, it’s a valid workaround.</p>

kkjh0723: <p>I compared several possible combinations of cuda, pytorch and gpus on my 3D ResNet model.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Cuda</th>
<th>pytorch</th>
<th>GPU</th>
<th>Etc</th>
<th>Top1</th>
<th>Time(s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>9.1</td>
<td>1.1</td>
<td>V100</td>
<td></td>
<td>68.7</td>
<td>215732</td>
</tr>
<tr>
<td>9.2</td>
<td>1.4</td>
<td>P40</td>
<td></td>
<td>68.1</td>
<td>305547</td>
</tr>
<tr>
<td>9.2</td>
<td>1.4</td>
<td>V100</td>
<td></td>
<td>61.7</td>
<td>155272</td>
</tr>
<tr>
<td>9.2</td>
<td>1.4</td>
<td>V100</td>
<td>Custom Conv</td>
<td>68.2</td>
<td>162994</td>
</tr>
<tr>
<td>9.2</td>
<td>1.4</td>
<td>V100</td>
<td>No cudnn</td>
<td>62.9</td>
<td>160497</td>
</tr>
<tr>
<td>10.2</td>
<td>1.5</td>
<td>V100</td>
<td></td>
<td>63.0</td>
<td>162676</td>
</tr>
<tr>
<td>10.2</td>
<td>1.5</td>
<td>V100</td>
<td>Custom Conv</td>
<td>67.6</td>
<td>186566</td>
</tr>
</tbody>
</table>
</div><p>When I use the custom <code>Conv3d</code> above, the top1 accuracies are increased compared to <code>nn.Conv3d</code>.<br>
However, it is still 0.5~1 % lower than pytoch1.1. I’m not sure it is caused by random initialization or some other issues. The wall-time decreased compared to cuda 9.1 (V100 machines are not exactly same but each has same number of CPU cores and GPUs).</p>
<p>When cudnn is turned off, the top1 accuracy is still lower. In order to turn off the cudnn, I make <code>cudnn.benchmark=False</code>. I’m not sure it is right way.</p>

ptrblck: <aside class="quote no-group" data-username="kkjh0723" data-post="10" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kkjh0723/48/3571_2.png" class="avatar"> kkjh0723:</div>
<blockquote>
<p>In order to turn off the cudnn, I make <code>cudnn.benchmark=False</code> . I’m not sure it is right way.</p>
</blockquote>
</aside>
<p>No, you should use <code>torch.backends.cudnn.enabled = False</code>.<br>
Your code will only disable the benchmark mode, which tries to find the fastest kernel.</p>
<p>The accuracy drop can be explained by the wrong answers of the current cudnn kernels, so stick to the custom conv approach or disable cudnn (you would have to profile it again unfortunately).</p>

Alpha: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The accuracy drop can be explained by the wrong answers of the current cudnn kernels</p>
</blockquote>
</aside>
<p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Would you mind giving some hints that why this issue happens on V100, while no problem on P100, etc.?</p>
<p>Thank you.</p>

ptrblck: <p>Different GPU architectures can use different kernel implementations.<br>
Based on</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I was able to reproduce and narrow down this issue to a faulty kernel in cudnn.<br>
I’ve also tested the next release candidate and the issue is gone, so my only advice for you is not to use this particular convolution in mixed-precision training. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>a particular kernel for this architecture was broken in cudnn 7.6.5.32 and fixed in cudnn&gt;=8.0.</p>

Alpha: <p>thank you very much for your quick reply.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="82265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>wrong answers of the current cudnn kernels</p>
</blockquote>
</aside>
<p>What does the <code>wrong answers</code> mean? <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I meant numerical mismatches, e.g. a <code>1.0</code> is expected at a particular output index, while the kernel returns <code>1.1</code>.</p>