Issue with layer of convolutional blocks in GAN generator

username: adrian_chervinchuk
cooked: <p>Hi, I am trying to reproduce some Generator for a GAN based on the code from this repo (they use tensor flow) <a href="https://github.com/lisc55/InfoGAN/blob/master/MNIST-wangchang/model.py" class="inline-onebox" rel="noopener nofollow ugc">InfoGAN/model.py at master · lisc55/InfoGAN · GitHub</a></p>
<p>at some point I get the error of mismatching dimensionalities(the error itself is below). I would appreciate any help explaining what I did wrong. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>here is my model:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        #  opt.latent_dim = 62; opt.n_classes = 5
        input_dim = opt.latent_dim + opt.n_classes # is equal to 67
        
        self.l1 = nn.Sequential(nn.Linear(input_dim, 1024, bias=False))
        self.conv_blocks = nn.Sequential(
            nn.BatchNorm2d(1024),
            nn.ReLU(True),
            nn.Linear(1024, 7*7*128, bias=False),
            nn.BatchNorm2d(7*7*128),
            nn.ReLU(True),
            #nn.Unflatten(7*7*128, [-1, 7, 7, 128]),
            nn.ConvTranspose2d(7*7*128, 64, (4, 4), 2, 1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.ConvTranspose2d(7*7*128, 1, (4, 4), 2, 1),
            nn.Tanh()
        )
        

    def forward(self, noise, labels):
        # batch_size is 64 
        # noise shape is (64, 62); labels shape is (64, 5)
        gen_input = torch.cat((noise, labels), -1) # shape is (64, 67)
        out = self.l1(gen_input) # shape is (64, 1024)
        out = out.view(out.shape[0], 1024, 1, 1) # shape is (64, 1024, 1, 1)
        img = self.conv_blocks(out) # &lt;- error appears here
        return img
</code></pre>
<p>the error I get is: <strong>RuntimeError: mat1 and mat2 shapes cannot be multiplied (65536x1 and 1024x6272)</strong></p>
<p>Original code using tf is:</p>
<pre><code class="lang-auto">def Generator(shape):
    w_init = tf.random_normal_initializer(stddev=0.02)
    gamma_init = tf.random_normal_initializer(1.0, 0.02)
    ni = tl.layers.Input(shape)
    nn = tl.layers.Dense(n_units=1024, b_init=None, W_init=w_init)(ni)
    nn = tl.layers.BatchNorm(decay=0.9, act=tf.nn.relu,
                             gamma_init=gamma_init)(nn)
    nn = tl.layers.Dense(n_units=7*7*128, b_init=None, W_init=w_init)(nn)
    nn = tl.layers.BatchNorm(decay=0.9, act=tf.nn.relu,
                             gamma_init=gamma_init)(nn)
    nn = tl.layers.Reshape([-1, 7, 7, 128])(nn)
    nn = tl.layers.DeConv2d(64, (4, 4), strides=(
        2, 2), padding="SAME", W_init=w_init)(nn)
    nn = tl.layers.BatchNorm(decay=0.9, act=tf.nn.relu,
                             gamma_init=gamma_init)(nn)
    nn = tl.layers.DeConv2d(
        1, (4, 4), strides=(2, 2), padding="SAME", act=tf.nn.tanh, W_init=w_init)(nn)
    return tl.models.Model(inputs=ni, outputs=nn)
</code></pre>

username: ptrblck
cooked: <p>You are trying to stack <code>nn.BatchNorm2d</code> layers with <code>nn.Linear</code> layers, which might work but it at least unusual as the default input shapes do not match. While <code>nn.BatchNorm2d</code> layers expect an input of <code>[batch_size, channels, height, width]</code>, <code>nn.Linear</code> layers operate on inputs in the shape <code>[batch_size, *, in_features]</code> where <code>*</code> denotes additional dimensions the linear layer will be applied on (similar as if you would loop these dimensions).<br>
Could you explain how exactly the reference model treats the dimensions?</p>

username: adrian_chervinchuk
cooked: <p>First of all thank You a lot for the response. Well, I am trying to implement a generator for MIST digits generation from the InfoGAN paper (<a href="https://arxiv.org/abs/1606.03657" class="inline-onebox" rel="noopener nofollow ugc">[1606.03657] InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets</a>). The proposed architecture is following<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ab0a7249f3d2c399407b0fea49a3b8579cdb17f.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ab0a7249f3d2c399407b0fea49a3b8579cdb17f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3ab0a7249f3d2c399407b0fea49a3b8579cdb17f_2_690x298.png" alt="image" data-base62-sha1="8ncayXl2kKJxuBdD7DgZzYAm8RV" width="690" height="298" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3ab0a7249f3d2c399407b0fea49a3b8579cdb17f_2_690x298.png, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ab0a7249f3d2c399407b0fea49a3b8579cdb17f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ab0a7249f3d2c399407b0fea49a3b8579cdb17f.png 2x" data-dominant-color="EBE7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">915×396 61.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I thought that treating every FC entry as a Linear layer is a right thing to do.</p>
<p>Not sure whether I answered your question…</p>

username: ptrblck
cooked: <p>Yes, treating the <code>FC</code> layers as <code>nn.Linear</code> layers is correct, but the more interesting question would be what the shape of the inputs is and how the layer is actually applied on them.<br>
From the screenshot I guess the first <code>FC</code> as well as the batchnorm layer would work on a 2D input while you are then reshaping the activation to pass it to an <code>nn.BatchNorm2d</code> layer.<br>
However, the screenshot misses these details and doesn’t show the activation shapes.</p>

username: adrian_chervinchuk
cooked: <p>Well, the input for the first FC was random noise of size 62 concatenated with 1 ten-dimensional vector which is dedicated to provide the generator with some useful info.</p>
<p>Now I use another architecture, namely:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/662996f9f97e3eb73b4b73585e437bd746e52516.png" data-download-href="https://discuss.pytorch.org/uploads/default/662996f9f97e3eb73b4b73585e437bd746e52516" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/662996f9f97e3eb73b4b73585e437bd746e52516_2_690x312.png" alt="image" data-base62-sha1="ezLO4u0t5qWOFz7kBoB9rSBBg0K" width="690" height="312" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/662996f9f97e3eb73b4b73585e437bd746e52516_2_690x312.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/662996f9f97e3eb73b4b73585e437bd746e52516_2_1035x468.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/662996f9f97e3eb73b4b73585e437bd746e52516.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1216×550 94.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>implementing it this way (seems to work better):</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.input_size = 190
        
        self.l1 = nn.Sequential(nn.Linear(self.input_size, 448*2*2))

        self.bn1 = nn.BatchNorm2d(448)

        self.tconv2 = nn.ConvTranspose2d(448, 256, 4, 2, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(256)

        self.tconv3 = nn.ConvTranspose2d(256, 128, 4, 2, padding=1, bias=False)

        self.tconv4 = nn.ConvTranspose2d(128, 64, 4, 2, padding=1, bias=False)

        self.tconv5 = nn.ConvTranspose2d(64, 3, 4, 2, padding=1, bias=False)

    def forward(self, x):
        x = x.view(x.shape[0], self.input_size)
        x = self.l1(x)
        x = x.view(x.shape[0], 448, 2, 2)
        x = F.relu(self.bn1(x))
        x = F.relu(self.bn2(self.tconv2(x)))
        x = F.relu(self.tconv3(x))
        x = F.relu(self.tconv4(x))

        img = torch.tanh(self.tconv5(x))

        return img
</code></pre>