Same Input Different Output (Not Random)

username: shawnbzhang
cooked: <p>I have a model that I have trained, and I am testing it by running it (on  <code>.eval()</code>  mode).</p>
<p>Here are the exact lines and order that I’ve executed in debugger:</p>
<pre><code class="lang-auto">(Pdb) p feature
tensor([[[ -4.0563,  -3.8415,  -3.9542,  ..., -14.8424, -14.9201, -14.8960],
         [ -5.8481,  -2.0405,  -2.4438,  ..., -19.6938, -19.4901, -19.9180],
         [ -5.2424,  -1.2804,  -1.5109,  ..., -19.3892, -19.4397, -19.5012],
         ...,
         [ -6.4756,  -2.0376,  -2.0894,  ..., -20.0942, -19.9635, -19.8762],
         [ -6.5087,  -2.0452,  -1.9018,  ..., -19.7127, -19.8574, -20.0103],
         [ -7.0725,  -4.2817,  -3.3231,  ..., -16.7170, -16.9004, -17.0333]]])
(Pdb) p feature2
tensor([[[ -4.0563,  -3.8415,  -3.9542,  ..., -14.8424, -14.9201, -14.8960],
         [ -5.8481,  -2.0405,  -2.4438,  ..., -19.6938, -19.4901, -19.9180],
         [ -5.2424,  -1.2804,  -1.5109,  ..., -19.3892, -19.4397, -19.5012],
         ...,
         [ -6.4756,  -2.0376,  -2.0894,  ..., -20.0942, -19.9635, -19.8762],
         [ -6.5087,  -2.0452,  -1.9018,  ..., -19.7127, -19.8574, -20.0103],
         [ -7.0725,  -4.2817,  -3.3231,  ..., -16.7170, -16.9004, -17.0333]]])
(Pdb) torch.all(feature == feature2)
tensor(True)
(Pdb) prediction_tag, prediction_time = model(feature)
(Pdb) prediction_tag2, prediction_time2 = model(feature2)
(Pdb) prediction_time 
tensor([[[9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.3993e-06, 3.7754e-05, 3.9786e-06,  ..., 1.7362e-04,
          1.0243e-02, 4.2382e-06],
         ...,
         [7.8885e-06, 1.1077e-05, 3.8594e-06,  ..., 1.9443e-04,
          3.8032e-03, 6.6878e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06]]])
(Pdb) p prediction_time2
tensor([[[8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [7.6509e-07, 1.9805e-05, 2.4918e-05,  ..., 3.2385e-04,
          4.3618e-03, 7.9963e-06],
         ...,
         [7.3927e-07, 8.7688e-06, 1.8454e-05,  ..., 1.9831e-04,
          1.9305e-03, 6.2879e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06]]])
(Pdb) torch.all(prediction_time == prediction_time2)
tensor(False)
</code></pre>
<p>As you can see, even though  <code>feature</code>  and  <code>feature2</code>  are seemingly the same inputs, the outputs by the model do not match. This isn’t random either, as after I have executed those lines above and running these lines below:</p>
<pre><code class="lang-auto">(Pdb) prediction_tag, prediction_time = model(feature)
(Pdb) prediction_time
tensor([[[9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.3993e-06, 3.7754e-05, 3.9786e-06,  ..., 1.7362e-04,
          1.0243e-02, 4.2382e-06],
         ...,
         [7.8885e-06, 1.1077e-05, 3.8594e-06,  ..., 1.9443e-04,
          3.8032e-03, 6.6878e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06]]])
(Pdb) prediction_tag2, prediction_time2 = model(feature2)
(Pdb) prediction_time2
tensor([[[8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [7.6509e-07, 1.9805e-05, 2.4918e-05,  ..., 3.2385e-04,
          4.3618e-03, 7.9963e-06],
         ...,
         [7.3927e-07, 8.7688e-06, 1.8454e-05,  ..., 1.9831e-04,
          1.9305e-03, 6.2879e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06]]])
</code></pre>
<p>I get the same, different outputs. Why am I experiencing this issue? I am totally confused.</p>
<p>Notes: I have checked that both  <code>feature</code>  and  <code>feature2</code>  have dtypes of  <code>torch.float32</code> .  <code>feature</code>  was extracted from a torch DataLoader that was set up, while  <code>feature2</code>  was directly obtained from reading a file.</p>
<p>Looking at my forward function, I see that these inputs change when they enter into the following function in the forward pass: <code>x = self.features(x)</code>, where:</p>
<pre><code class="lang-auto">self.features = nn.Sequential(
            Block2D(1, 32),
            nn.LPPool2d(4, (2, 4)),
            Block2D(32, 128),
            Block2D(128, 128),
            nn.LPPool2d(4, (2, 4)),
            Block2D(128, 128),
            Block2D(128, 128),
            nn.LPPool2d(4, (1, 4)),
            nn.Dropout(0.3),
)
</code></pre>
<p>where <code>Block2D</code> is:</p>
<pre><code class="lang-auto">class Block2D(nn.Module):
    def __init__(self, cin, cout, kernel_size=3, padding=1):
        super().__init__()
        self.block = nn.Sequential(
            nn.BatchNorm2d(cin),
            nn.Conv2d(cin,
                      cout,
                      kernel_size=kernel_size,
                      padding=padding,
                      bias=False),
            nn.LeakyReLU(inplace=True, negative_slope=0.1))

    def forward(self, x):
        return self.block(x)
</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce the issue using your model and random input:</p>
<pre><code class="lang-python">features.eval()
x = torch.randn(32, 1, 100, 100)
out0 = features(x).clone()
out1 = features(x).clone()

print((out0 - out1).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Could you post an executable code snippet to reproduce this issue?</p>

username: shawnbzhang
cooked: <p>So I actually figured out where the error was happening. I was getting different outputs between the following two ways I formatted the input:<br>
<code>feature1 = torch.as_tensor([feature]).to(DEVICE)</code><br>
<code>feature2 = torch.as_tensor(feature).unsqueeze(0).to(DEVICE)</code></p>
<p>Although <code>torch.all(feature1 == feature2)</code> would return true, the outputs would differ quite a bit. Do you have an idea of why this would be the case? The shapes all seemed to match.</p>

username: pfloat
cooked: <blockquote>
<p>Although <code>torch.all(feature1 == feature2)</code> would return true, the outputs would differ quite a bit. Do you have an idea of why this would be the case? The shapes all seemed to match.</p>
</blockquote>
<p>That may be the float precision, the absolute value <code>|feature1 - feature2|</code> might be smaller that a threshold value <code>t</code>, with <code>t</code> very small like <code>10^(-6)</code>. If <code>|feature1 - feature2| &lt; t</code>, then  <code>torch.all</code> will return <code>True</code>. It is similar to <code>numpy.allclose</code>.</p>
<pre><code class="lang-auto">torch.tensor([1,2]) == torch.tensor([1,2.0000001]) # returns tensor([True, True])
torch.tensor([1,2]) == torch.tensor([1,2.000001]) # returns tensor([ True, False])
</code></pre>

username: shawnbzhang
cooked: <p>I see. Do you know why <code>torch.as_tensor([feature])</code> and <code>torch.as_tensor(feature).unsqueeze(0)</code> would cause changes in float precision?</p>