How to make output of CNN to be input of RNN Layer?

sornpraram: <p>Hi,<br>
I am new to CNN, RNN and deep learning.<br>
I am trying to make architecture that will combine CNN and RNN.<br>
input image size = [20,3,48,48]<br>
a CNN output size = [20,64,48,48]<br>
and now i want cnn ouput to be RNN input<br>
but as I know the input of RNN must be 3-dimension only which is [seq_len, batch, input_size]<br>
How can I make 4-dimensional [20,64,48,48] tensor into 3-dimensional for RNN input?</p>

ptrblck: <p>You would have to decide which dimension(s) should be the temporal dimension (<code>seq_len</code>) and which the features (<code>input_size</code>).<br>
E.g. you could treat the output channels as the features and the spatial dimensions (height and width) as the temporal dimension.<br>
To do so, you could first flatten the spatial dimensions via:</p>
<pre><code class="lang-python">output = output.view(output.size(0), output.size(1), -1)
</code></pre>
<p>and then permute the dimensions via:</p>
<pre><code class="lang-python">output = output.permute(2, 0, 1)
</code></pre>

cskarthik7: <p>But what if we use batch_first=True in the RNN layer? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>In that case, the permutation should be:</p>
<pre><code class="lang-python">output = output.permute(0, 2, 1)
</code></pre>
<p>since the RNN expects the input as <code>[batch_size, seq_len, features]</code>.</p>

sornpraram: <p>Many thank <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
After your solution my Input size change to<code>[2304, 20, 64]</code>  after though RNN layer my output is the same size. So I try to reshape back to <code>[20, 64, 48, 48]</code> because my Network is CNN &gt; RNN &gt; CNN.<br>
But somehow after thought RNN layer. some data in tensor have changed.<br>
While I’m trying take the output back in CNN I got this error</p>
<pre><code class="lang-auto">TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple
</code></pre>
<p>Do you have any idea about this problem?</p>

ptrblck: <p><code>nn.RNN</code> outputs a tuple as <code>output, hidden_state</code> as described in the <a href="https://pytorch.org/docs/stable/nn.html?highlight=rnn#torch.nn.RNN">docs</a>, so you might want to use only one of these outputs for further processing.</p>

Sara176: <p>hi<br>
i am trying to feed output features of my 3DCNN to gru<br>
inputs to my 3DCNN are videos processed and stored in numpy arrays of the shape<br>
[70, 1, 29, 88, 88]where each dimension corresponds to  [batch size , num of channels , number of frames, Hight , width]</p>
<p>here is my 3DCNN<br>
class CNN3D(nn.Module):                                                                         <span class="hashtag">#classes_num</span><br>
def <strong>init</strong>(self, t_dim=29, img_x=88, img_y=88, drop_p=0.2, fc_hidden1=256, fc_hidden2=128, num_classes=2):<br>
super(CNN3D, self).<strong>init</strong>()</p>
<pre><code>    # set video dimension
    self.t_dim = t_dim
    self.img_x = img_x
    self.img_y = img_y
    # fully connected layer hidden nodes
    self.fc_hidden1, self.fc_hidden2 = fc_hidden1, fc_hidden2
    self.drop_p = drop_p
    self.num_classes = num_classes
    self.ch1, self.ch2 = 32, 48
    self.k1, self.k2 = (5, 5, 5), (3, 3, 3)  # 3d kernel size
    self.s1, self.s2 = (2, 2, 2), (2, 2, 2)  # 3d strides
    self.pd1, self.pd2 = (0, 0, 0), (0, 0, 0)  # 3d padding

    # compute conv1 &amp; conv2 output shape
    self.conv1_outshape = conv3D_output_size((self.t_dim, self.img_x, self.img_y), self.pd1, self.k1, self.s1)
    self.conv2_outshape = conv3D_output_size(self.conv1_outshape, self.pd2, self.k2, self.s2)

    self.conv1 = nn.Conv3d(in_channels=1, out_channels=self.ch1, kernel_size=self.k1, stride=self.s1,
                           padding=self.pd1)
    self.bn1 = nn.BatchNorm3d(self.ch1)
    self.conv2 = nn.Conv3d(in_channels=self.ch1, out_channels=self.ch2, kernel_size=self.k2, stride=self.s2,
                           padding=self.pd2)
    self.bn2 = nn.BatchNorm3d(self.ch2)
    self.relu = nn.ReLU(inplace=True)
    self.drop = nn.Dropout3d(self.drop_p)
    self.pool = nn.MaxPool3d(2)
    self.fc1 = nn.Linear(self.ch2 * self.conv2_outshape[0] * self.conv2_outshape[1] * self.conv2_outshape[2],#self.ch2 * self.conv2_outshape[0] * self.conv2_outshape[1] * self.conv2_outshape[2]
                         self.fc_hidden1)  # fully connected hidden layer
    self.fc2 = nn.Linear(self.fc_hidden1, self.fc_hidden2)
    self.fc3 = nn.Linear(self.fc_hidden2, self.num_classes)  # fully connected layer, output = multi-classes

def forward(self, x_3d):
    #print(x_3d.shape) #[70, 1, 29, 88, 88]
    # Conv 1
    x = self.conv1(x_3d)
    x = self.bn1(x)
    x = self.relu(x)
    x = self.drop(x)
    print(x.shape) #[70, 32, 13, 42, 42] 
    # Conv 2
    x = self.conv2(x)
    x = self.bn2(x)
    x = self.relu(x)
    x = self.drop(x)
    #print(x.shape) #[70, 48, 6, 20, 20]
    # FC 1 and 2
    x = x.view(x.size(0), -1)
    #print(x.shape) #[70, 115200]
    x = F.relu(self.fc1(x))
    #print(x.shape) #[70, 256]
    x = F.relu(self.fc2(x))
    #print(x.shape) #[70, 128]
    x = F.dropout(x, p=self.drop_p, training=self.training)
    x = self.fc3(x)
    #print(x.shape) #[70, 2]

    return x
</code></pre>
<p>i tried to add gru layer before linear layers but i could not as gru expects input of shape (batch size , sequence length , input size)  and my 3DCNN is reducing the number of frames 29</p>
<p>how to feed 3DCNN extracted features  to gru ?</p>