Multi-task regression problem using Pytorch (Problem: Same output for all test data)

Animesh_Kumar_Paul: <p>I am trying to multi-task regression problem.</p>
<p>Input shape: 200 * 60000, Output shape: 200*3 (here, 200 = Total number of data, 60000 = number of features)</p>
<p>So, for each data point, I have to predict 3 values (continuous).</p>
<p>Sample Code:</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self,input_nodes):
        super(Classifier, self).__init__()
        self.input_nodes = input_nodes

        self.sharedlayer = nn.Sequential(
            nn.Linear(input_nodes, 300),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(300, 100),
            nn.ReLU(),
            nn.Dropout(),
        )


        self.att1 = nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(20, 1)
        )
        self.att2 = nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(20, 1)
        )
        self.att3 = nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(20, 1)
        )

    def forward(self, x):

        h_shared = self.sharedlayer(x)
        out1 = self.att1(h_shared)
        out2 = self.att2(h_shared)
        out3 = self.att3(h_shared)

        return out1, out2, out3

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)
for epoch in range(n_epochs):
            running_loss = 0
            i = 0
            model.train()
            for data, label in trainloader:
                 i = i + 1
                 out1, out2, out3 = model(data)


                 l1 = criterion(out1, label[:,0].view(-1,1))    
                 l2 = criterion(out2, label[:,1].view(-1,1))
                 l3 = criterion(out3, label[:,2].view(-1,1))    

                 loss = (l1 + l2 + l3)
                 optimizer.zero_grad()
                 loss.backward()
                 optimizer.step()
</code></pre>
<p>Problem: The model always produces the same value for all test data.</p>
<p>Example: Suppose, 3 Test data:</p>
<p>For output1: 3.5 3.5 3.5<br>
For output2: 9.5 9.5 9.5<br>
For output3: 0.2 0.2 0.2</p>
<p>Can you please help me to figure out what is the problem here?</p>
<p>Why does it generate the same value for all test data?</p>

ptrblck: <p>Could you check the <code>bias</code> values of the last linear layers?<br>
Sometimes bad hyperparameters (e.g. high learning rate) throws the parameters out so that only the bias will be predicted (often the mean value of your target).</p>

Animesh_Kumar_Paul: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your suggestion.</p>
<p>Bias value is different than the output values.</p>
<p>For 11 test data —<br>
Generated Outputs for output3 are:</p>
<pre><code class="lang-auto">tensor([[0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806],
         [0.8806]], grad_fn=&lt;"AddmmBackward"&gt;),
</code></pre>
<p>Access the weights and bias using the below code:</p>
<pre><code class="lang-auto">a) model.att3[6].weight
Parameter containing:
tensor([[-0.0960,  0.1035, -0.0493,  0.4335, -0.2495,  0.1452,  0.1481, -0.4376,
         -0.0052,  0.0901,  0.0533, -0.0907,  0.0778,  0.1182, -0.4527,  0.2129,
         -0.2819, -0.0838,  0.0184, -0.0613]], requires_grad=True)

b) model.att3[6].bias
Parameter containing:
tensor([0.5688], requires_grad=True)
</code></pre>
<p>If you have any other idea, please let me know</p>