Simple Model not learning

Saswatm123: <p>I am making a model to learn the Sine function in domain x = [0, 2 * pi]. Very simple, and I have done it using C++ frameworks as a test to see if I know how to use the framework.</p>
<p>My model always returns a straight line at y = 0 after training, and a straight line with y between -1 and 1 before training. The Model’s structure is:</p>
<p>Linear(1, 20) -&gt; Sigmoid -&gt; Linear(20, 50) -&gt; Sigmoid -&gt; Linear(50, 50) -&gt; Sigmoid -&gt; Linear(50, 1).</p>
<p>This structure has worked for me in different frameworks, so I know using Sigmoid isnt the problem, and neither is the number of parameters. I believe the problem is in my usage of autograd.</p>
<p>Here is the code for my Net:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.model = nn.Sequential(
      nn.Linear(1, 20),
      nn.Sigmoid(),
      nn.Linear(20, 50),
      nn.Sigmoid(),
      nn.Linear(50, 50),
      nn.Sigmoid(),
      nn.Linear(50, 1)
      )
            
    def forward(self, x):
        x = self.model(x)
        return x
</code></pre>
<p>Here is my training loop:</p>
<pre><code class="lang-auto">def train(net, trainloader, valloader, learningrate, n_epochs):
  net = net.train()
  loss = nn.MSELoss()
  optimizer = torch.optim.SGD(net.parameters(), lr = learningrate)
  
  for epoch in range(n_epochs):
    
    for X, y in trainloader:
        X = X.reshape(-1, 1)
        optimizer.zero_grad()
      
        outputs = net.forward(X)
        error   = loss(outputs, y)
        error.backward()
        optimizer.step()
      
    total_loss = 0
    for X, y in valloader:
        X = X.reshape(-1, 1).float()
        outputs = net(X)
        error   = loss(outputs, y.float() )
        total_loss += error.data
      
    print('Val loss for epoch', epoch, 'is', total_loss / len(valloader) )
</code></pre>
<p>Where trainloader and valloader are the data loaders for the training and validation data.</p>
<p>Here is my data creation:</p>
<pre><code class="lang-auto">data = np.repeat( np.arange(0, 2 * np.pi, step = .001), 7)

from sklearn.model_selection import train_test_split
train_X, test_X, train_y, test_y = train_test_split(data, np.sin(data), test_size = 1/5, random_state = 9122019, shuffle = True)
from torch.utils.data import DataLoader

train_X, val_X, train_y, val_y = train_test_split(train_X, train_y, train_size = .8, shuffle = False)

train_data  = Dataset(train_X, train_y)
val_data    = Dataset(val_X, val_y)
test_data   = Dataset(test_X, test_y)

trainloader = DataLoader(train_data, batch_size = 64, shuffle = True)
valloader   = DataLoader(val_data, batch_size = 64, shuffle = True)
testloader  = DataLoader(test_data, batch_size = 64, shuffle = True)
</code></pre>
<p>The final result after a large amount of epochs is a line at y = 0. The final result overall is always a straight line at the mean of the output data, so if i set Y = X, the final result is a horizontal line at ( X.max() - X.min() )/ 2</p>
<p>Can someone tell me where my code is wrong? Thank you.</p>
<p>Edit: My train is called as such:</p>
<pre><code class="lang-auto">net = Net()
train(net, trainloader, valloader, .0001, n_epochs = 4)
</code></pre>

ptrblck: <p>You are accidentally broadcasting the output and target tensor inside your training loop and this should also raise a warning in the latest stable PyTorch version.<br>
Add <code>y = y.view(-1, 1)</code> to your training and validation loop and try to run the code again.<br>
PS: If you are using an older PyTorch version, I would recommend to update to the latest release. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Also, call the model directly as <code>model(data)</code> instead of <code>model.forward(data)</code>, as this will make sure to register hooks etc.</p>

Saswatm123: <p>Added, but still doesn’t change what’s happening. My new training/validation code is as follows (only added 2 lines saying y = y.view(-1, 1) ):</p>
<pre><code class="lang-auto">def train(net, trainloader, valloader, learningrate, n_epochs):
    net = net.train()
    loss = nn.MSELoss()
    optimizer = torch.optim.SGD(net.parameters(), lr = learningrate)

    for epoch in range(n_epochs):

        for X, y in trainloader:
            X = X.reshape(-1, 1)
            y = y.view(-1, 1) #Added line
            optimizer.zero_grad()

            outputs = net(X)

            error   = loss(outputs, y)
            error.backward()
            optimizer.step()
                
        total_loss = 0
        for X, y in valloader:
            X = X.reshape(-1, 1).float()
            y = y.view(-1, 1) #Added line
            outputs = net(X)
            error   = loss(outputs, y)
            total_loss += error.data

        print('Val loss for epoch', epoch, 'is', total_loss / len(valloader) )
  
</code></pre>