Batch dependent results

isaacg: <p>Hi All,</p>
<p>I was wondering if someone knows if this same bug exists in pytorch?</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/tensorflow/tensorflow/issues/32544" target="_blank" rel="nofollow noopener">github.com/tensorflow/tensorflow</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/isaacgerg" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/11971499?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/tensorflow/tensorflow/issues/32544" target="_blank" rel="nofollow noopener">Issue: tf.keras batch normalization is batch dependent at test time</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/isaacgerg" target="_blank" rel="nofollow noopener">isaacgerg</a>
	on <a href="https://github.com/tensorflow/tensorflow/issues/32544" target="_blank" rel="nofollow noopener">2019-09-15</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Both
OS Platform and Distribution...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">TF 1.13</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">comp:keras</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">stat:awaiting response</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">type:bug</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Essentially, function outputs are dependent on batch size/data.  For example, on a given input x, you get y=f(x). now add x to a batch of data z and send over the batch.  You would exist for x’s index in z that y_z would equal y, but this is not the case.</p>

ptrblck: <p>I have to admit it’s a strange question to ask for a specific bug. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
However, if you set your batch norm to <code>.eval()</code> in PyTorch, the result will not be depending on the batch and the running estimates will be used for each sample.</p>
<p>Let us know, if you encounter any strange behavior.</p>

isaacg: <p>I understand what you mean. The test example is very simple to implement and if pytorch has the same error, then it might be with cudnn.</p>

ptrblck: <p>You could try to run this dummy example on your machine and check the difference:</p>
<pre><code class="lang-python">x = torch.randn(10, 3, 24, 24)
bn = nn.BatchNorm2d(3)

# dummy forward passes to update running estimates
for _ in range(10):
    _ = bn(x)

# Set to eval
bn.eval()

output_all = bn(x)

output_manual = []
for i in range(x.size(0)):
    output_manual.append(bn(x[i:i+1]))
output_manual = torch.cat(output_manual)

print((output_all - output_manual).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

isaacg: <p>I know this is a strange request but i really want to run this issue to ground.  Thanks for the code!</p>