Convert Conv3d to Conv2d but outputs different

fengpings: <p>I tried to convert a Conv3d to Conv2d by splitting 3D kernels by the temporal axis, but outputs from the two differ:<br>
<a href="https://private-user-images.githubusercontent.com/71132578/380730915-b7d6bc4e-a0c2-40be-b4de-851800ab5f96.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzAxODQzOTUsIm5iZiI6MTczMDE4NDA5NSwicGF0aCI6Ii83MTEzMjU3OC8zODA3MzA5MTUtYjdkNmJjNGUtYTBjMi00MGJlLWI0ZGUtODUxODAwYWI1Zjk2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEwMjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMDI5VDA2NDEzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEzZWViMjU0NTY1ZDJjZjk1NzEyNWQ3MGNkOWRjMzVjZjliNGJhOWFlZDAwYjIzZmJiNDVkZmEwZjI5MTVlZDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.vVyLQNyS6_JRIrKWUQhqi00kQTwK18DP0uRojwCgPlM" rel="noopener nofollow ugc"><img src="https://private-user-images.githubusercontent.com/71132578/380730915-b7d6bc4e-a0c2-40be-b4de-851800ab5f96.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzAxODQzOTUsIm5iZiI6MTczMDE4NDA5NSwicGF0aCI6Ii83MTEzMjU3OC8zODA3MzA5MTUtYjdkNmJjNGUtYTBjMi00MGJlLWI0ZGUtODUxODAwYWI1Zjk2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEwMjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMDI5VDA2NDEzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEzZWViMjU0NTY1ZDJjZjk1NzEyNWQ3MGNkOWRjMzVjZjliNGJhOWFlZDAwYjIzZmJiNDVkZmEwZjI5MTVlZDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.vVyLQNyS6_JRIrKWUQhqi00kQTwK18DP0uRojwCgPlM" alt="image" width="690" height="460"></a><br>
I expect output1 and output2 should be the same. Any details I’ve overlooked?</p>
<p>code:</p>
<pre><code class="lang-auto">t = torch.randn(1, 1, 4, 4, dtype=torch.float32)
conv3d = torch.nn.Conv3d(out_channels=1, in_channels=1, kernel_size=(2,2,2), stride=(2,2,2), bias=False, dtype=torch.float32)
# conv3d2 = torch.nn.Conv3d(out_channels=1, in_channels=1, kernel_size=(2,2,2), stride=(2,2,2), bias=False, dtype=torch.bfloat16)
# conv3d.weight.data = torch.ones(1, 1, 2, 2, 2, dtype=torch.bfloat16)
# conv3d2.weight.data = conv3d.weight.data
conv2d1 = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2,2), stride=(2,2), bias=False, dtype=torch.float32)
conv2d2 = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2,2), stride=(2,2), bias=False, dtype=torch.float32)
# conv2d3 = torch.nn.Conv2d(in_channels=3, out_channels=12, kernel_size=(14,14), stride=(14,14), bias=False)
# print(conv3d.weight.shape)
# conv2d1.weight = torch.nn.parameter.Parameter(conv3d.weight.detach().reshape(12, 6, 14, 14).contiguous())
# conv2d3.weight = torch.nn.parameter.Parameter(torch.mean(conv3d.weight.detach(), dim=2).squeeze(dim=2))

print(conv3d.weight.data.dtype)
conv2d1.weight.data = conv3d.weight.data[:,:, 0, :, :]
conv2d2.weight.data = conv3d.weight.data[:,:, 1, :, :]


input = torch.stack([t, t], dim=0).view(-1, 1, 2, 2, 2)
# print(input.shape)
output1 = conv3d(input).view(-1, 1)
output2 = (conv2d1(input[:, :, 0, :, :]) + conv2d2(input[:, :, 1, :, :])).view(-1, 1)
# output3 = conv3d2(input).view(-1, 1)
print(output1==output2)
</code></pre>
<h3><a name="p-458229-versions-1" class="anchor" href="#p-458229-versions-1"></a>Versions</h3>
<p>PyTorch version: 2.4.1<br>
Is debug build: False<br>
CUDA used to build PyTorch: None<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: macOS 15.0.1 (arm64)<br>
GCC version: Could not collect<br>
Clang version: 16.0.0 (clang-1600.0.26.3)<br>
CMake version: version 3.28.1<br>
Libc version: N/A</p>
<p>Python version: 3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ] (64-bit runtime)<br>
Python platform: macOS-15.0.1-arm64-arm-64bit<br>
Is CUDA available: False<br>
CUDA runtime version: No CUDA<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Apple M3 Max</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.26.3<br>
[pip3] torch==2.4.1<br>
[pip3] torchaudio==2.4.1<br>
[pip3] torchvision==0.19.1<br>
[conda] numpy 1.26.3 pypi_0 pypi<br>
[conda] torch 2.4.1 pypi_0 pypi<br>
[conda] torchaudio 2.4.1 pypi_0 pypi<br>
[conda] torchvision 0.19.1 pypi_0 pypi</p>

ptrblck: <p>Your image is not displayed for me, but keep in mind that small numerical mismatches would be expected if different kernels are selected for the different workloads.<br>
Thus check the <code>.abs().max()</code> error in your comparison instead of assuming equal results.</p>