Training Makes Model Predict Everything as the Same Value

username: Rishav_Sen
cooked: <p>When I train my model and test it on the test data set, it returns all of the same values. The network is meant to classify objects with 2 float parameters into one of 9 classes. The data is imbalanced so I used a WeightedRandomSampler.</p>
<p>My neural network:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, D_in, H1, H2, D_out):
        super(Net, self).__init__()
        self.linear1 = nn.Linear(D_in, H1)
        self.linear2 = nn.Linear(H1, H2)
        self.linear4 = nn.Linear(H2, D_out)

    def forward(self, x):
        x = f.relu(self.linear1(x))
        x = f.relu(self.linear2(x))
        x = self.linear4(x)
        return x
</code></pre>
<p>What it returns when not trained and inputted test data:</p>
<pre><code class="lang-auto">tensor([[-300.7366,  142.0265, 1203.9167,  ..., -366.3187,  930.1630,
         -460.1914],
        [-118.0432,   55.4708,  467.9875,  ..., -142.0467,  360.8930,
         -178.8175],
        [-165.0909,   77.8327,  658.5565,  ..., -200.1997,  508.4938,
         -251.7095],
        ...,
        [-166.6815,   78.6092,  665.3017,  ..., -202.2803,  513.7719,
         -254.2981],
        [-197.1317,   93.1154,  789.1254,  ..., -240.1015,  609.7628,
         -301.6740],
        [-130.0121,   61.4217,  520.3043,  ..., -158.2938,  402.0972,
         -198.9381]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>Training:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
    model.train(True)
    for i, xy in enumerate(trainloader):
        x, y = xy
        optimizer.zero_grad()
        z = model(x)
        loss = Loss(z, y)
        loss.backward()
        optimizer.step()
        loss_list.append(loss.data)
    
    print("Epoch is", epoch)
</code></pre>
<p>What it returns when trained and inputted test data:</p>
<pre><code class="lang-auto">tensor([[-0.0234,  0.0055,  0.0225,  ..., -0.0304, -0.0017,  0.0139],
        [-0.0234,  0.0055,  0.0225,  ..., -0.0304, -0.0017,  0.0139],
        [-0.0234,  0.0055,  0.0225,  ..., -0.0304, -0.0017,  0.0139],
        ...,
        [-0.0234,  0.0055,  0.0225,  ..., -0.0304, -0.0017,  0.0139],
        [-0.0234,  0.0055,  0.0225,  ..., -0.0304, -0.0017,  0.0139],
        [-0.0234,  0.0055,  0.0225,  ..., -0.0304, -0.0017,  0.0139]],
       grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>Any help or advice would be appreciated.</p>
<p>Thanks,<br>
Rishav</p>

username: tom
cooked: <p>Did you check that the dataloader returns class-balanced (and random) samples (by running a for loop over it and collecting statistics)?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Rishav_Sen
cooked: <p>I confirmed that the dataloader returns class-balanced and random samples.</p>
<p>Thanks,<br>
Rishav</p>

username: tom
cooked: <p>Hi Rishav,</p>
<p>What does the loss do?<br>
What did you try to reduce your your learning rate?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Rishav_Sen
cooked: <p>Hello Tom,</p>
<p>Reducing my learning rate did not solve my problem. The loss variable is used for optimization.</p>
<p>Thanks,<br>
Rishav</p>

username: tom
cooked: <p>Hi Rishav,</p>
<p>but does the loss go up or down or stay the same?<br>
If you set the lr to 0, you should keep the same weights, so I’m tempted to say that it would be good to find where it stops to do funny things.<br>
What optimizer are you using and which parameters? Sometimes when you do funny things with momentum or somesuch, it’ll get the strangest results.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Rishav_Sen
cooked: <p>Hello Tom,</p>
<p>The loss value seems to go down until it reaches around 2, where it stays around. When the learning rate is 0, nothing changes after training. I am using a SGD optimizer. Learning rate = 0.01 and momentum = 0.9. No matter how low I set the learning rate to, the problem persists.</p>
<p>Thanks,<br>
Rishav</p>

username: tom
cooked: <p>You could see if you can overfit a single batch to be sure your loss function is working as expected.</p>

username: Rishav_Sen
cooked: <p>Hello Tom,</p>
<p>I determined that I either put in the data incorrectly or there is an error in the training training. Could you look at my code to help find the error</p>
<p>Dataloader:</p>
<pre><code class="lang-auto">class Data(Dataset):
    def __init__(self):
        self.x = TrainData
        self.y = TrainOutput
        self.len = self.x.shape[0]

    def __getitem__(self, index):
        return self.x[index], self.y[index]

    def __len__(self):
        return self.len

trainloader = DataLoader(dataset = Data(), batch_size = 1, sampler = sampler)
</code></pre>
<p>Training:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
    model.train(True)
    


    running_loss = 0.
    last_loss = 0.
    for i, xy in enumerate(trainloader):
        x, y = xy
        optimizer.zero_grad()
        z = model(x)
        loss = Loss(z, y)
        loss.backward()
        optimizer.step()
        loss_list.append(loss.data)
        running_loss += loss.item()
        test[y] += 1
        if i % 1000 == 999:
            last_loss = running_loss / 1000 # loss per batch
            running_loss = 0.
    
    print("Epoch is", epoch)
    epoch_number += 1
</code></pre>
<p>Model:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, D_in, H1, H2, D_out):
        super(Net, self).__init__()
        self.linear1 = nn.Linear(D_in, H1)
        self.linear2 = nn.Linear(H1, H2)
        self.linear4 = nn.Linear(H2, D_out)

    def forward(self, x):
        x = f.relu(self.linear1(x))
        x = f.relu(self.linear2(x))
        x = self.linear4(x)
        return x
</code></pre>
<p>Here is what the variables are.</p>
<p>TrainData:</p>
<pre><code class="lang-auto">Shape: torch.Size([50316, 2])
Value: tensor([[ 6.5819e+03, 3.5400e+00], [ 3.4742e+03, 9.2000e-02], [ 5.3910e+03, 3.6600e+00], ..., [ 3.6822e+03, -3.1800e-01], [ 6.2419e+03, 3.2160e+00], [ 5.6387e+03, 1.4360e+01]])
</code></pre>
<p>TrainOutput:</p>
<pre><code class="lang-auto">Shape: torch.Size([50316])
Value: tensor([6, 4, 5, ..., 4, 6, 8])
</code></pre>
<p>Thanks,<br>
Rishav</p>

username: Rishav_Sen
cooked: <p>To add on, using activation functions from torch.nn seems to cause the same problem before any training occurs. When torch or torch.nn.functional activation functions are used, the results displayed are before training are random.</p>

username: tom
cooked: <p>To me, it looks as if the trouble is more likely to be in your loss or model than in the bits you posted.</p>