CNN outputs the same result even with different inputs

username: icarosadero
cooked: <p>Hello,</p>
<p>I am a beginner at neural networks and I have a project where I want to process text via a CNN. The input is basically a one-hot encoded tensor and the output is meant to be a single neuron with values ranging from -1 to 1. Just to illustrate what I am sending to this network, if T is the input tensor, a side by side composition of its layers in the form T[k,:,:] after the convolutions is in this link:</p>
<p>                    <a href="https://i.imgur.com/b9wkETZ.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/b9wkETZ.png" width="384" height="100">
          </a>

</p>
<p>The height stands for the input channels and the width is the product between the number of samples and the features.</p>
<p>Everything looks fine so far until it passes to the perceptron, the output always is a vector with equal valued entries. Even after changing the convolutional layers, loss function and optimization method over and over I get the same results. The activation function I am using for the perceptron is a sigmoid function. The code I made for the network architecture is as follows:</p>
<pre><code class="lang-auto">class model(nn.Module):
    def __init__(self,channels):
        super(model, self).__init__()
        self.convolutions = nn.Sequential(
            nn.Conv1d(channels, 500, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(500, 400, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(400, 300, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(300, 200, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(200, 100, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            )
        self.perceptron = nn.Sequential(
            nn.Linear(100,50),
            nn.Sigmoid(),
            nn.Linear(50,16),
            nn.Sigmoid(),
            nn.Linear(16,1),
            nn.Sigmoid()
        )
        self.transfer = lambda x,y: nn.Linear(x,y)
    def forward(self, x):
        x = self.convolutions(x)
        x = x.view(x.size(0), -1)
        l = self.transfer(x.shape[1],self.perceptron[0].in_features)
        x = self.perceptron(l(x))
        return x
</code></pre>
<p>With all this said, I would like to know if I am messing up something either about the theory or the implementation. Thanks in advance.</p>

username: phan_phan
cooked: <p>In the forward method:</p>
<pre><code class="lang-auto">l = self.transfer(x.shape[1], self.perceptron[0].in_features)
x = self.perceptron(l(x))
</code></pre>
<p>what you are doing is creating a new layer at each step … So this layer <code>l</code> cannot learn, since it is discarded and randomly reinitialized at each step.</p>
<p>The thing is, there is no variable-size <code>nn.Linear</code> layer!<br>
Because this layer has a fixed number of parameters which are learned, and it would not be logical to remove some, or add new parameters at each step during training.</p>
<p>Is your input tensor <code>x</code> of variable size ?</p>

username: icarosadero
cooked: <p>Thank you, I really didn’t know about that…</p>
<p>The input tensor x can vary from dataset to dataset but its dimensions stay constant on each run. Because of this, I managed to remove the <code>l</code> layer from the class entirely and instead set the input dimension on the first linear layer of the perceptron as the product of the last kernel size of the constitutional layer and the number of features. It seemed to still work but the network keeps getting stuck around the same values.</p>
<p>I also tried to run the same code in a different dataset and I still get the same results.</p>
<p>The code now is:</p>
<pre><code class="lang-auto">class model(nn.Module):
    def __init__(self,c_channels,char_size):
        super(model, self).__init__()
        self.last_kernel = 3
        self.p_channels = self.last_kernel*char_size
        self.convolutions = nn.Sequential(
            nn.Conv1d(c_channels, 500, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(500, 400, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(400, 300, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(300, 200, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(200, 100, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=self.last_kernel),
            )
        self.perceptron = nn.Sequential(
            nn.Linear(self.p_channels,50),
            nn.Sigmoid(),
            nn.Linear(50,16),
            nn.Sigmoid(),
            nn.Linear(16,1),
            nn.Sigmoid()
        )
        #self.transfer = lambda x,y: nn.Linear(x,y)
    def forward(self, x):
        x = self.convolutions(x)
        x = x.view(x.size(0), -1)
        print(x.shape)
        #l = self.transfer(x.shape[1],self.perceptron[0].in_features)
        #x = self.perceptron(l(x))
        x = self.perceptron(x)
        return x
</code></pre>

username: ChenK19
cooked: <p>Have you fix your problem? I face the same problem now, could you give me some advice?</p>