RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 3, 3, 3], but got 5-dimensional input of size [16, 64, 3, 32, 32] instead

j.h.Lee: <p>Hello, I’m currently using a code to learn RGB images of 256x256 size using <a href="https://arxiv.org/abs/1610.02357" rel="noopener nofollow ugc">Xception</a>.</p>
<p>This time, I found and used the code to learn using Patch Processing. Thanks to <a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/7">this post</a>, I succeeded in dividing the sensor into patches, but as the number of patches parameter was used, the 4-dimensional sensor became 5-dimensional, making it difficult to learn. I searched, but there were many 4D-3D transformations, but I couldn’t find any 4D-5D transformations. I wonder how to use Xception as it is by maintaining the preprocessed patch information.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7dd5bc927ef3654cc12d8ce80d8928ee057a13b.png" alt="이미지" data-base62-sha1="uNCUHPhQyb53xdNqIhQuzs2K5lx" width="515" height="201"></p>
<p>I’m learning using Pytorch now, but any solution is fine. Please understand that my translation is not good. <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=12" title=":cold_sweat:" class="emoji" alt=":cold_sweat:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I’m not exactly sure how you would like to use these patches, as they can be used to apply a convolution via a matrix multiplication (<code>unfold</code> is also known as <code>im2col</code> which is used in the “matmul-conv” approach). Could you describe your use case a bit more and how the patches should be treated by the <code>XceptionNet</code>?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

j.h.Lee: <p>Thank you for your answers.    The Xception I use looks like this.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Depthwise Separable Convolution
class SeparableConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()

        self.seperable = nn.Sequential(
            nn.Conv2d(in_channels, in_channels, 3, stride=1, padding=1, bias=False),
            nn.Conv2d(in_channels, out_channels, 1, stride=1, padding=0, bias=False)
        )

    def forward(self, x):
        x = self.seperable(x)
        return x
    
# EnrtyFlow
class EntryFlow(nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 32, 3, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 64, 3, stride=1, padding=0, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU()
        )

        self.conv2_residual = nn.Sequential(
            SeparableConv(64, 128),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            SeparableConv(128, 128),
            nn.BatchNorm2d(128),
            nn.MaxPool2d(3, stride=2, padding=1)
        )

        self.conv2_shortcut = nn.Sequential(
            nn.Conv2d(64, 128, 1, stride=2, padding=0),
            nn.BatchNorm2d(128)
        )

        self.conv3_residual = nn.Sequential(
            nn.ReLU(),
            SeparableConv(128, 256),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            SeparableConv(256, 256),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(3, stride=2, padding=1)
        )

        self.conv3_shortcut = nn.Sequential(
            nn.Conv2d(128, 256, 1, stride=2, padding=0),
            nn.BatchNorm2d(256)
        )

        self.conv4_residual = nn.Sequential(
            nn.ReLU(),
            SeparableConv(256, 728),
            nn.BatchNorm2d(728),
            nn.ReLU(),
            SeparableConv(728, 728),
            nn.BatchNorm2d(728),
            nn.MaxPool2d(3, stride=2, padding=1)
        )

        self.conv4_shortcut = nn.Sequential(
            nn.Conv2d(256, 728, 1, stride=2, padding=0),
            nn.BatchNorm2d(728)
        )

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2_residual(x) + self.conv2_shortcut(x)
        x = self.conv3_residual(x) + self.conv3_shortcut(x)
        x = self.conv4_residual(x) + self.conv4_shortcut(x)
        return x

# MiddleFlow
class MiddleFlow(nn.Module):
    def __init__(self):
        super().__init__()

        self.conv_residual = nn.Sequential(
            nn.ReLU(),
            SeparableConv(728, 728),
            nn.BatchNorm2d(728),
            nn.ReLU(),
            SeparableConv(728, 728),
            nn.BatchNorm2d(728),
            nn.ReLU(),
            SeparableConv(728, 728),
            nn.BatchNorm2d(728)
        )

        self.conv_shortcut = nn.Sequential()

    def forward(self, x):
        return self.conv_shortcut(x) + self.conv_residual(x)
    
# ExitFlow
class ExitFlow(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()

        self.conv1_residual = nn.Sequential(
            nn.ReLU(),
            SeparableConv(728, 1024),
            nn.BatchNorm2d(1024),
            nn.ReLU(),
            SeparableConv(1024, 1024),
            nn.BatchNorm2d(1024),
            nn.MaxPool2d(3, stride=2, padding=1)
        )

        self.conv1_shortcut = nn.Sequential(
            nn.Conv2d(728, 1024, 1, stride=2, padding=0),
            nn.BatchNorm2d(1024)
        )

        self.conv2 = nn.Sequential(
            SeparableConv(1024, 1536),
            nn.BatchNorm2d(1536),
            nn.ReLU(),
            SeparableConv(1536, 2048),
            nn.BatchNorm2d(2048),
            nn.ReLU()
        )

        self.avg_pool = nn.AdaptiveAvgPool2d((1,1))
    
    def forward(self, x):
        x = self.conv1_residual(x) + self.conv1_shortcut(x)
        x = self.conv2(x)
        x = self.avg_pool(x)
        return x
    
# Xception
class Xception(nn.Module):
    def __init__(self, num_classes=10, init_weights=True):
        super().__init__()
        self.init_weights = init_weights

        self.entry = EntryFlow()
        self.middle = self._make_middle_flow()
        self.exit = ExitFlow()

        self.linear = nn.Linear(2048, num_classes)

        # weights initialization
        if self.init_weights:
            pass


    def forward(self, x):
        x = self.entry(x)
        x = self.middle(x)
        x = self.exit(x)
        x = x.view(x.size(0), -1)
        x = self.linear(x)
        return x

    def _make_middle_flow(self):
        middle = nn.Sequential()
        for i in range(8):
            middle.add_module('middle_block_{}'.format(i), MiddleFlow())
        return middle

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init_kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
                if m.bias is not None:
                    nn.init_constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init_constant_(m.weight, 1)
                nn.init_bias_(m.bias, 0)
            elif isinstance(m, nn.Linear):
                nn.init_normal_(m.weight, 0, 0.01)
                nn.init_constant_(m.bias, 0)

                
if __name__ == "__main__":
    x = torch.randn([1,3,256,256])
    x = x.to(device)
    net = Xception(num_classes=2).to(device)
    y = net(x)

</code></pre>
<p>The dataset used is 256×256 image of FFHQ. I would like to simply divide this 256x256 image into 16 chapters 32x32 to preprocess and learn. I don’t know if this will help, but I look forward to your kind cooperation again.</p>

ptrblck: <p>If I understand your use case correctly you would like to pass each <code>32x32</code> patch into the model and classify it independently? If so you could <code>unfold</code> the patches and permute and view the tensor to create a tensor of <code>[batch_size=num_samples*num_patches, channels, 32, 32]</code> and use it as the new input. Of course you would also need to repeat the target tensor accordingly, since each <code>256x256</code> input image is now creating 16 <code>32x32</code> patches assuming none of them overlap.</p>

j.h.Lee: <p>Thank you. As you said, I’ll multiply and reduce the dimension and try again. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"></p>