Non-reproducible results with BlurPool in timm

username: clck10
cooked: <p>Hello all,</p>
<p>Wanted to stop by and check if things are working as expected before making an issue on the timm Github page.</p>
<p>In short, I am getting non-reproducible results when networks have a BlurPool module from timm. Other networks (without BlurPool) are exactly reproducible after setting the seed across the needed libraries (numpy, random, torch, torch.cuda, etc).</p>
<p>It is the strangest thing, since there is nothing in the module that screams hidden randomness:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/blur_pool.py">
  <header class="source">

      <a href="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/blur_pool.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/blur_pool.py" target="_blank" rel="noopener nofollow ugc">rwightman/pytorch-image-models/blob/master/timm/models/layers/blur_pool.py</a></h4>


    <pre><code class="lang-py">"""
BlurPool layer inspired by
 - Kornia's Max_BlurPool2d
 - Making Convolutional Networks Shift-Invariant Again :cite:`zhang2019shiftinvar`

Hacked together by Chris Ha and Ross Wightman
"""

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from .padding import get_padding


class BlurPool2d(nn.Module):
    r"""Creates a module that computes blurs and downsample a given feature map.
    See :cite:`zhang2019shiftinvar` for more details.
    Corresponds to the Downsample class, which does blurring and subsampling

</code></pre>


  This file has been truncated. <a href="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/blur_pool.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’ve confirmed this on several machines with different environments and versions of CUDA.</p>
<p>Has anyone seen something similar, or have some suggestions about what is going on?<br>
Or is this expected behavior from BlurPool?</p>
<p>Thanks much for any help in advance, please let me know if more details are needed.</p>
<p>Ty!</p>

username: ptrblck
cooked: <p>I assume you’ve followed all steps described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a> to get deterministic results?<br>
In particular, since a convolution is used inside this layer you would have to make sure to use deterministic cuDNN algorithms.</p>

username: clck10
cooked: <p>Thank you for your response! Yes, I believe so. Here is my current seed setter that is called in the main training function:</p>
<pre><code class="lang-python">    np.random.seed(seed)
    random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<p>I am also seeding the DataLoader workers with the worker_init_function in the same way.</p>
<p>The strangest thing is I can make things non-reproducible just by swapping the pooling layer. In the ResNets I am using that is either in: the max-pool after the stem, or the anti-aliasing avg-pool in the downsampling paths.</p>
<p>As soon as I swap either of those for a BlurPool2d, things become irreproducible.</p>
<p>Thank you again!</p>