Why regression model built using torch wont work well?

ganesh_shinde: <p>I am quite new with torch.<br>
And I have mostly difficulty with fitting simple feed forward network with some random data</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   
        self.predict = torch.nn.Linear(n_hidden, n_output)   

    def forward(self, x):
        x = F.relu(self.hidden(x))     
        x = self.predict(x)            
        return x
</code></pre>
<pre><code class="lang-auto">import sklearn.datasets as datasets
X_,y_=datasets.make_regression(n_samples=500,n_features=2,noise=0.1)
X = torch.from_numpy(X_).float()
y = torch.from_numpy(y_).float()
net = Net(n_feature=2, n_hidden=10, n_output=1)     
print(net)  

optimizer = torch.optim.SGD(net.parameters(), lr=0.02)
loss_func = torch.nn.MSELoss()

for t in range(2000):
    prediction = net(X)    

    loss = loss_func(prediction, y)     

    optimizer.zero_grad()   
    loss.backward()         
    optimizer.step()        

    if t % 5 == 0:
        print(loss.data.numpy())
</code></pre>
<p>I tried to tune hyper parameters like number of nodes and learning rate as well as tried changing optimiser and loss function but looks like not much improvement. Is there any issue in above code or someone also had bad time with torchs regression?</p>

Deeply: <aside class="quote no-group" data-username="ganesh_shinde" data-post="1" data-topic="45037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ganesh_shinde/48/12042_2.png" class="avatar"> ganesh_shinde:</div>
<blockquote>
<p>optimizer.zero_grad()</p>
</blockquote>
</aside>
<p><code>optimizer.zero_grad()</code>  should be calculated before  <code>loss = loss_func(prediction, y)</code></p>

ganesh_shinde: <p>I tried rearranging zero grad before loss but but seems like loss not going much down though.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/be98504d3707de1590d480c8d022a5a4fb21e3cc.png" alt="index" data-base62-sha1="rc535X70SCnZyRUI9kCRmJKFiTW" width="400" height="252"></p>

Deeply: <p>In fact, <code>optimizer.zero_grad()</code> should be before <code>prediction = net(X)</code></p>

Deeply: <p>Also, try to reduce the learning rate, maybe using learning rate decay via <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.StepLR" rel="nofollow noopener">scheduler</a></p>

ganesh_shinde: <p>Yes that is what I tried so now</p>
<pre><code class="lang-auto">for t in range(2000):
    prediction = net(X)    

    loss = loss_func(prediction, y)       
    loss.backward()         
    optimizer.step()        

    if t % 5 == 0:
        print(loss.data.numpy())
  optimizer.zero_grad()
</code></pre>
<p>yes it is before prediction<br>
and I just compared results from model with training data its still looks like not learning anything</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/82af4fa554bf00ac4aa57cf1e8a1d8912991499c.png" alt="index2" data-base62-sha1="iE5FKUi8G96MP8pXos0JAijDrZO" width="386" height="252"></p>
<p>hence loss is very high</p>

Deeply: <p>Try this:</p>
<pre><code class="lang-auto">for t in range(2000):
    optimizer.zero_grad()
    prediction = net(X)    
    loss = loss_func(prediction, y)       
    loss.backward()         
    optimizer.step()        

    if t % 5 == 0:
        print(loss.data.numpy())
</code></pre>
<p>Also, you might need to lower the learning rate as I described above</p>

ganesh_shinde: <pre><code class="lang-auto">losses=[]
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
for t in range(2000):
    optimizer.zero_grad()   # clear gradients for next train
    prediction = net(X)     # input x and predict based on x
    

    loss = loss_func(prediction, y)   

    
    loss.backward()         # backpropagation, compute gradients
    optimizer.step()        # apply gradients
    scheduler.step()      # change learning rate on every 30th iteration 

    if t % 5 == 0:
        losses.append(loss.data.numpy())
</code></pre>
<p>I already tried both ways but but still loss and prediction doesn’t change .<br>
I think need some transformation over data.</p>

Deeply: <aside class="quote no-group" data-username="ganesh_shinde" data-post="8" data-topic="45037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ganesh_shinde/48/12042_2.png" class="avatar"> ganesh_shinde:</div>
<blockquote>
<p>ink need some transformation over dat</p>
</blockquote>
</aside>
<p>You need thus to normalize your data to have zero mean and unit variance.</p>

ganesh_shinde: <p>I am trying normalisation but I am still worried about why torch fails to learn anything? as most of prediction from model is around zero</p>

ptrblck: <p>Might be unrelated, but could you check if the model output and target have the same shape?<br>
If your target is missing the feature dimension (<code>[batch_size]</code> instead of <code>[batch_size, 1]</code>), an unwanted broadcast might be applied.<br>
In the latest version this should throw a warning.</p>

ganesh_shinde: <p>This was real issue I was using Torch 1.0.0 hence didn’t got any warning</p>

lapayo: <p><a class="mention" href="/u/deeply">@Deeply</a> Do you have any references for your statement? As far as I know, that is not necessary.</p>

Deeply: <p>I think ptrblck provided the solution to your problem, but, it would be a good idea to normalize your data, especially if it has entries of different numerical ranges/scales. For example, assume having a vector with these two values x=[.05, 12]. Working with this vector may make your regression biased toward the highest value (<em>i.e.</em> 12). Here’s a <a href="https://towardsdatascience.com/how-when-and-why-should-you-normalize-standardize-rescale-your-data-3f083def38ff" rel="nofollow noopener">link</a> that describes this issue in some detail.</p>

lapayo: <p><a class="mention" href="/u/deeply">@Deeply</a> I did not mean the normalization, meant the part about the position of zero_grad <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

MariosOreo: <p>I think the method based on ptrblck’s comment with a lower learning_rate (i.e.  1e-3) makes sense.</p>