LSTM learns a horizontal line for periodic data

sahilkap07: <p>I am using LSTM network for a periodic data which includes training input sequence of angular data and outputs a value corresponding to this sequence. The output is not similar to the input but is related to the input sequence in some manner. The network learns a horizontal line which is just the average value for the entire output values. I have tried most of the things I could: varying the batch size, trying different learning rates, changing the hidden size, different optimizers, different sequence lengths(10-400) and scaling the data.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8ca7f5507f12a4267456720d03cab243bde668ef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8ca7f5507f12a4267456720d03cab243bde668ef" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8ca7f5507f12a4267456720d03cab243bde668ef_2_689x272.jpeg" alt="image" data-base62-sha1="k4iH4IauiSqD8sExYJvCFWo1bGv" width="689" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8ca7f5507f12a4267456720d03cab243bde668ef_2_689x272.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8ca7f5507f12a4267456720d03cab243bde668ef_2_1033x408.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8ca7f5507f12a4267456720d03cab243bde668ef.jpeg 2x" data-dominant-color="D4D3D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1181×467 117 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After printing the  weights (before and after training) and gradient values, I observed that the weights do change with some of the values remaining same and the gradient values become very small as the training proceeds ultimately leading to a flat line.<br>
Here the network architecture, it is very simple.</p>
<pre><code class="lang-auto"># Neural Net Class
class LSTM(nn.Module):

    def __init__(self, num_classes=1, input_size=1, hidden_size=2, num_layers=1, seq_length=40):
        super(LSTM, self).__init__()
        
        self.num_classes = num_classes
        self.num_layers = num_layers
        self.input_size = input_size  
        self.hidden_size = hidden_size
        self.seq_length = seq_length
        
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                            num_layers=num_layers, batch_first=True)
        
        self.fc = nn.Linear(hidden_size, num_classes)
 
    def forward(self, x):
        h_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size))
        
        c_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size))
        
        # Propagate input through LSTM
        ou, (h_out, _) = self.lstm(x, (h_0, c_0))
        
        h_out = h_out.view(-1, self.hidden_size)
        
        out = self.fc(h_out)
        
        return out

net = LSTM(num_classes, input_size, hidden_size, num_layers, seq_length) #  Neural Net

criterion = torch.nn.MSELoss() #    Loss Criterion
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate) #  Optimizer
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.6)   #   lr scheduler

#   Training function
def train(epoch):
    loss_batch = 0
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        # print(inputs.shape)
        # print(targets.shape)
        inputs = torch.squeeze(inputs,0)
        optimizer.zero_grad()
        out = net(inputs)
        loss = criterion(out, targets)
        loss.backward()
        # print(net.lstm.weight_hh_l0.grad)
        optimizer.step()
        scheduler.step()
        loss_batch += loss.item()
    return loss_batch

#   Training loop
for epoch in range(0, epochs):
    loss = train(epoch)
    if (epoch%2 == 0):
        print(loss)
</code></pre>
<p>This is the output which is just a flat line:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c35d642faa1ccb377e27ca08bee51a69712ef15.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3c35d642faa1ccb377e27ca08bee51a69712ef15" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c35d642faa1ccb377e27ca08bee51a69712ef15_2_690x270.jpeg" alt="image" data-base62-sha1="8ADZAyrGmmDdCrkbxF34vJBsDsx" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c35d642faa1ccb377e27ca08bee51a69712ef15_2_690x270.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c35d642faa1ccb377e27ca08bee51a69712ef15_2_1035x405.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c35d642faa1ccb377e27ca08bee51a69712ef15_2_1380x540.jpeg 2x" data-dominant-color="C0BDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1729×679 222 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The data-points have been collected at very high frequency and adjacent values in the data differ only by 0.01 utmost and consecutive values can also be repeated in the sequence.<br>
Is there anything that I am doing wrong with the network or the kind of architecture I am using? Should I try some other type of neural network or loss function?</p>