Terrible performance in simple linear regression from sequence

dayala: <p>Hello! I have been doing some research regarding plane trajectories prediction. I want to test the performance of different network architectures with points taken every 30/60/90/120 seconds. In particular, from a sequence of the last n increments in three features (longitude, latitude, and height), I try to predict the next increment. Since in most examples the plane is following a relatively stright line, I expected results to be good with the exception of some complex cases where the plane is turning or doing some maneuver. However, the results are terrible overall.<br>
Here is an example of input (sequence of 4 elements, each with 3 features):</p>
<pre><code class="lang-auto">xs_batch[0]
tensor([[-0.0156, -0.0226, -0.0750],
        [-0.0146, -0.0226, -0.0750],
        [-0.0161, -0.0226, -0.0750],
        [-0.0139, -0.0226, -0.0750]], device='cuda:0')
</code></pre>
<p>And an example of intended output (prediction of the next value for the 3 features):</p>
<pre><code class="lang-auto">ys_batch[0]
tensor([-0.0147, -0.0226, -0.0750], device='cuda:0')
</code></pre>
<p>I have tested first with a simple network that should at the very least learn that the predicted increment is very similar to the last one in the sequence:</p>
<pre><code class="lang-python"># Function for creating generic blocks of linear layers with activations and optional dropout
def create_dense_block(input_size, output_size, hidden_sizes:list, dropout_rate=0.1):
    layers = []
    input_sizes = [input_size,] + hidden_sizes
    output_sizes = hidden_sizes + [output_size,]

    for iz, oz in zip(input_sizes, output_sizes):
        if(dropout_rate &gt; 0):
            layers.append(nn.Dropout(dropout_rate))
        layers.append(nn.Linear(iz, oz))
        layers.append(nn.Tanh())
    return nn.Sequential(*layers)

# The feedforward architecture
class DenseNetwork(nn.Module): 
    def __init__(self, number_features, sequence_size):
        super(DenseNetwork, self).__init__()
        # block of linear layers that ends with an output of size 100
        self.fc1 = create_dense_block(number_features*sequence_size, 100, [500,500,400,200], dropout_rate=0)
        # Final linear layer without activation at the end to allow negative results
        self.fc2 = nn.Linear(100, number_features)

    def forward(self, x:torch.Tensor):
        shape = x.shape
        # reshaping to turn sequences of x elements with y features into x*y features
        x = x.reshape((shape[0], -1))
        x = self.fc1(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>The training code is here:</p>
<pre><code class="lang-python">    model = architectures.DenseNetwork(len(features_x), seq_size)
    loss_function = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), learning_rate)
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, verbose=True, threshold=1e-2)
    model.to(device)
    model.train()

    for epoch in range(num_epochs):
        for batch in loader_training:
            xs_batch:torch.Tensor = batch["inputs"]
            ys_batch:torch.Tensor = batch["labels"]
            xs_batch = xs_batch.to(device).float()
            ys_batch = ys_batch.to(device).float()
            model.zero_grad()
            out = model(xs_batch)
            loss = loss_function(out, ys_batch)
            loss.backward()
            optimizer.step()
</code></pre>
<p>The training data contains mora than 10.000 examples. I have tested with more and less layers, smaller and bigger, with higher and lesser learning rate, confirming that the shapes of the output and at every single step of the network are correct:</p>
<pre><code class="lang-auto">xs_batch.shape
torch.Size([128, 4, 3])

ys_batch.shape
torch.Size([128, 3])

out.shape
torch.Size([128, 3])
</code></pre>
<p>And yet the results are quite bad and bizarre at times, so I wonder if there is some detail related to parameters or similar that I’m missing.</p>

KFrank: <p>Hi Daniel!</p>
<aside class="quote no-group quote-modified" data-username="dayala" data-post="1" data-topic="142150" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ad7895/48.png" class="avatar"> dayala:</div>
<blockquote>
<p>However, the results are terrible overall.<br>
…</p>
<pre data-code-wrap="python"><code class="lang-python">def create_dense_block(input_size, output_size, hidden_sizes:list, dropout_rate=0.1):
    layers = []
    input_sizes = [input_size,] + hidden_sizes
    output_sizes = hidden_sizes + [output_size,]

    for iz, oz in zip(input_sizes, output_sizes):
        if(dropout_rate &gt; 0):
            layers.append(nn.Dropout(dropout_rate))
        layers.append(nn.Linear(iz, oz))
        layers.append(nn.Tanh())

# The feedforward architecture
</code></pre>
</blockquote>
</aside>
<p>As written, <code>create_dense_block()</code> doesn’t return anything.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">class DenseNetwork(nn.Module): 
        ...
        self.fc1 = create_dense_block(number_features*sequence_size, 100, [500,500,400,200], dropout_rate=0)
        ...

    def forward(self, x:torch.Tensor):
        ...
        x = self.fc1(x)
</code></pre>
</blockquote>
</aside>
<p>I’m surprised that your code runs rather than throwing an error.</p>
<p>You assign <code>self.fc1</code> to the “result” of a no-return function, so<br>
<code>self.fc1</code> should have <code>None</code> assigned to it.  So I would expect<br>
<code>x = self.fc1(x)</code> to throw an error because you can’t call <code>None</code><br>
as if it were a function.</p>
<p>Best.</p>
<p>K. Frank</p>

dayala: <p>It seems that i accidentally deleted it qhiñe formatting the code. There is a “return nn.Sequential(*layers)” at the end. Editing original mesaage to include it. Thanks for the notice.</p>

googlebot: <p>I haven’t noticed any problems. Try SGD optimizer with big initial lr and ExponentialLR scheduler (there is no scheduler.step() in your code btw). Or optimizing without mini-batches (Rprop or LBFGS optimizers).</p>

dayala: <p>No luck, the result is pretty much the same.</p>
<p>Also I will add: all features being used, as well as the predicted values, are normalised</p>

googlebot: <p>well, I’d first plot training loss to ensure it is decreasing epoch-to-epoch and maybe identify failure type from it (too slow or too fast learning, bad lr schedule, tiny gradients [not evident per se, need deeper inspection] or code problems)</p>

dayala: <p>Found the issue. Some samples had really anomalous values that had not been removed, caused a huge loss and influenced the training.</p>
<p>Thanks for all the replies!</p>