Non-deterministic training (gradient update) in Resnet

username: harryZZZ
cooked: <p>Hello I am implementing a resnet 3d model in pytorch but constantly get non-deterministic result.<br>
I do have all seed set up:<br>
def seed_torch(seed=123):<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = True</p>
<p>def _init_fn(worker_id):<br>
seed_torch(SEED)</p>
<p>I also change the model to Alexnet then the result is deterministic, so there must be some functions inside Resnet that cause this issue.</p>
<p>I am using the resnet 3D from:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/kenshohara/3D-ResNets-PyTorch/blob/master/models/resnet.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/kenshohara/3D-ResNets-PyTorch/blob/master/models/resnet.py" target="_blank" rel="nofollow noopener">kenshohara/3D-ResNets-PyTorch/blob/master/models/resnet.py</a></h4>
<pre><code class="lang-py">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import math
from functools import partial

__all__ = [
    'ResNet', 'resnet10', 'resnet18', 'resnet34', 'resnet50', 'resnet101',
    'resnet152', 'resnet200'
]


def conv3x3x3(in_planes, out_planes, stride=1):
    # 3x3x3 convolution with padding
    return nn.Conv3d(
        in_planes,
        out_planes,
        kernel_size=3,
        stride=stride,
</code></pre>

  This file has been truncated. <a href="https://github.com/kenshohara/3D-ResNets-PyTorch/blob/master/models/resnet.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I tried both wide resnet and resnet, 18 and 50, all have the same issue</p>

username: tom
cooked: <p>Probably the backward of the downsampling:</p>
<p><a href="https://pytorch.org/docs/stable/notes/randomness.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a></p>
<p>Seems to be one of the things everyone wants to have but nobody takes the time to do.</p>
<p>Best regards</p>
<p>Thomas</p>

username: harryZZZ
cooked: <p>Do you mean in shortcut A, the down_sample_basic_block?<br>
I used shortcut B, which the downsample is basically a conv + batchnorm</p>

username: tom
cooked: <p>Did you click on the link?</p>
<blockquote>
<p>A number of operations have backwards that use <code>atomicAdd</code> , in particular <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.embedding_bag" rel="noopener nofollow ugc"> <code>torch.nn.functional.embedding_bag()</code> </a>, <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.ctc_loss" rel="noopener nofollow ugc"> <code>torch.nn.functional.ctc_loss()</code> </a> and many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions.</p>
</blockquote>
<p>From a cursory reading of the code, you use pooling.</p>
<p>Best regards</p>
<p>Thomas</p>

username: harryZZZ
cooked: <p>yes I did read that although it is very vague…<br>
but I also use max pooling/avg pooling 3d in Alexnet, which is fine, so I figure pooling is not the issue.</p>
<p>add more info:<br>
for 2d models I work with before, resnet has no such issue</p>

username: tom
cooked: <p>The implementations can differ, both for the networks on top of the ops and the ops themselves. For example 3d avg pooling seems to use atomicAdd for cases where the windows can overlap, so something like</p>
<pre><code class="lang-auto">self.avgpool = nn.AvgPool3d((last_duration, last_size, last_size), stride=1)
</code></pre>
<p>is suspect.</p>
<p>Best regards</p>
<p>Thomas</p>

username: harryZZZ
cooked: <p>Thank you Tom, I tried to substitute this with maxpool 3d but still nondeterministic…<br>
however maxpool 3d is deterministic in my alexnet<br>
I will continue to explore other layer’s behavior</p>