Issue Report: Problem Encountered with 1x1 Convolutional Layer

username: vutya
cooked: <p>Dear PyTorch Community,</p>
<p>I am encountering an issue related to the usage of a 1x1 convolutional layer in my script. The problem is comprehensively described in the attached screenshot:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64eb5f36b78afc3c3a3e9324b9352a02fce85701.png" data-download-href="https://discuss.pytorch.org/uploads/default/64eb5f36b78afc3c3a3e9324b9352a02fce85701" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64eb5f36b78afc3c3a3e9324b9352a02fce85701_2_348x500.png" alt="image" data-base62-sha1="eoM1NLHQIeqVLkiZ1HHO4z3xrs5" width="348" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64eb5f36b78afc3c3a3e9324b9352a02fce85701_2_348x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64eb5f36b78afc3c3a3e9324b9352a02fce85701_2_522x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64eb5f36b78afc3c3a3e9324b9352a02fce85701_2_696x1000.png 2x" data-dominant-color="EEEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">926×1328 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I appreciate any assistance or insights the community can provide to help resolve this issue.</p>
<p>Thank you,<br>
V.</p>

username: vutya
cooked: <p>I can also provide all the pickle files loaded in the script, but I am currently couldn’t find an option to attach files. Please advise on the preferred method for sharing files to facilitate a full reproduction of the problem.</p>

username: ptrblck
cooked: <p>Could you describe what exactly the issue is?</p>

username: vutya
cooked: <p>I conduct inference on my validation dataset using the CNN. Initially, the results were anticipated to be independent of the batch size. However, upon experimenting with different batch sizes, I observed inconsistent outputs. Upon careful debugging, I identified that the first occurrence of variance appeared after the first 1x1 convolution layer. I localized the problem to this specific layer and conducted inference solely on this 1x1 convolution, not the entire CNN.</p>
<p>In the attached screenshot, the feature map “x” has four dimensions: (batch=2, channels=144, spatial_x=1, spatial_y=1). I expected that applying a 1x1 convolution to “x” would yield identical results to applying the same layer to “x[0]” and “x[1],” followed by concatenating them. It means that in the provided screenshot, “out0” and “out1” tensors should ideally be identical. Regrettably, a small discrepancy (~1e-7) exists, although even torch/np.isclose returns True for these tensors. This variance accumulates throughout the subsequent layers of the CNN, reaching approximately ~1e-4 by the inference’s conclusion.</p>
<p>In the second part of code of the screenshot, I demonstrate that replacing the 1x1 convolution with a linear layer resolves the issue so I have implemented a workaround for myself, but I believe this discrepancy in the 1x1 convolution’s output is indicative of a potential bug that merits attention and resolution.</p>
<p>I appreciate your time and consideration in addressing this matter. If further clarification or additional information is required, please do not hesitate to reach out.</p>
<p>Thank you for your assistance.</p>

username: vutya
cooked: <p>I wish to extend my apologies for an error I previously made in my code related to the comparison of out0 and out1 tensors. I have now rectified this mistake and am including a corrected screenshot for your reference. Upon reviewing the updated output, it appears that the issue also extends to a linear layer.</p>
<p>It is also worth mentioning that, despite the expectation of identical behavior from linear and conv 1x1 layers with equivalent weight data, there is a difference in the absolute values of variance observed in their outputs.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0dcdb2cd128215efce83ec79942f6bd60fdd77e.png" data-download-href="https://discuss.pytorch.org/uploads/default/b0dcdb2cd128215efce83ec79942f6bd60fdd77e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0dcdb2cd128215efce83ec79942f6bd60fdd77e_2_385x500.png" alt="image" data-base62-sha1="peBdrQZnLlHPdKwgCvdaBRAGP8O" width="385" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0dcdb2cd128215efce83ec79942f6bd60fdd77e_2_385x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0dcdb2cd128215efce83ec79942f6bd60fdd77e_2_577x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0dcdb2cd128215efce83ec79942f6bd60fdd77e_2_770x1000.png 2x" data-dominant-color="EEEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1024×1328 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>These small numerical mismatches are expected and caused by a different order of operations to calculate the result and the limited floating point precision. Neither of the two outputs is “more correct” and should show a similar error to a wider <code>dtype</code>.</p>

username: vutya
cooked: <p>OK, I see, thanks for the response anyway</p>