RNN for generating time series

AndreaCogliati: I’m trying to modify the world_language_model example to generate a time series. My naive approach was to replace the softmax output with a single linear output layer, and change the loss function to MSELoss. Unfortunately, my network seems to learn to output the current input, instead of predicting the next sample. So when I try to generate a new time series, the network is soon stuck at a fixed point. Any suggestions on how to improve my model? Here’s my code: <SCODE>#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Train a LSTM network to generate a time series.
'''

import argparse
import collections
import csv
import math
import pickle
import time
import torch
import torch.nn as nn
from torch.autograd import Variable


def read_arguments():
    parser = argparse.ArgumentParser(description='Train a recurrent network to generate a time series.')
    parser.add_argument('--data', type=str, default='data.txt',
                        help='data file to read (CSV)')
    parser.add_argument('--model', type=str, default='LSTM',
                        help='type of recurrent net (RNN_TANH, RNN_RELU, LSTM, GRU)')
    parser.add_argument('--nhid', type=int, default=100,
                        help='humber of hidden units per layer')
    parser.add_argument('--nlayers', type=int, default=2,
                        help='number of layers')
    parser.add_argument('--lr', type=float, default=.05,
                        help='initial learning rate')
    parser.add_argument('--clip', type=float, default=5,
                        help='gradient clipping')
    parser.add_argument('--epochs', type=int, default=10,
                        help='upper epoch limit')
    parser.add_argument('--batch-size', type=int, default=10, metavar='N',
                        help='batch size')
    parser.add_argument('--bptt', type=int, default=375,
                        help='sequence length')
    parser.add_argument('--checkpoint-interval', type=int, default=10, metavar='N',
                        help='interval to save intermediate models')
    parser.add_argument('--save', type=str,  default='model',
                        help='path to save the final model')
    args = parser.parse_args()
    return args


class RNNModel(nn.Module):
    """Container module with an encoder, a recurrent module, and a decoder."""

    def __init__(self, rnn_type, nhid, nlayers):
        super(RNNModel, self).__init__()
        self.rnn = getattr(nn, rnn_type)(1, nhid, nlayers)
        self.output = nn.Linear(nhid, 1)

        self.init_weights()

        self.rnn_type = rnn_type
        self.nhid = nhid
        self.nlayers = nlayers

    def init_weights(self):
        initrange = 0.1
        self.output.bias.data.fill_(1.0)
        self.output.weight.data.uniform_(-initrange, initrange)

    def forward(self, input, hidden):
        output_lstm, hidden = self.rnn(input, hidden)
        output = self.output(output_lstm.view(output_lstm.size(0)*output_lstm.size(1), output_lstm.size(2)))
        return output.view(output_lstm.size(0), output_lstm.size(1), output.size(1)), hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters()).data
        if self.rnn_type == 'LSTM':
            return (Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()),
                    Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()))
        else:
            return Variable(weight.new(self.nlayers, bsz, self.nhid).zero_())


def flatten(l):
    for el in l:
        if isinstance(el, collections.Iterable) and not isinstance(el, str):
            for sub in flatten(el):
                yield sub
        else:
            yield el
        
        
def batchify(data, bsz):
    nbatch = data.size(0) // bsz
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1).t().contiguous()
    if torch.cuda.is_available():
        data = data.cuda()
    return data            


def load_data(filename, batch_size):
    '''
    Load a training data sequence from a CSV file
    '''
    with open(filename) as csvfile:
        csvreader = csv.reader(csvfile)
        data = list(csvreader)
        data = torch.Tensor([float(x) for x in flatten(data)])

    train_length = math.ceil(len(data) * .7)
    val_length = math.ceil(len(data) * .2)
    train_data = data[:train_length]
    val_data = data[train_length:train_length+val_length]
    test_data = data[train_length+val_length:]
    return batchify(train_data, batch_size), batchify(val_data, batch_size), batchify(test_data, batch_size)
    
###############################################################################
# Training code
###############################################################################

def clip_gradient(model, clip):
    """Computes a gradient clipping coefficient based on gradient norm."""
    totalnorm = 0
    for p in model.parameters():
        modulenorm = p.grad.data.norm()
        totalnorm += modulenorm ** 2
    totalnorm = math.sqrt(totalnorm)
    return min(1, clip / (totalnorm + 1e-6))


def repackage_hidden(h):
    """Wraps hidden states in new Variables, to detach them from their history."""
    if type(h) == Variable:
        return Variable(h.data)
    else:
        return tuple(repackage_hidden(v) for v in h)


def get_batch(source, i, seq_length, evaluation=False):
    seq_len = min(seq_length, len(source) - 1 - i)
    data = Variable(source[i:i+seq_len].view(seq_len, -1, 1), volatile=evaluation)
    target = Variable(source[i+1:i+1+seq_len].view(-1))
    return data, target


def evaluate(data_source, model, criterion, batch_size, seq_length):
    total_loss = 0
    hidden = model.init_hidden(batch_size)
    for i in range(0, data_source.size(0) - 1, seq_length):
        data, targets = get_batch(data_source, i, seq_length, evaluation=True)
        output, hidden = model(data, hidden)
        total_loss += len(data) * criterion(output, targets).data
        hidden = repackage_hidden(hidden)
    return total_loss[0] / len(data_source)


def train(train_data, model, criterion, lr, batch_size, seq_length, grad_clip):
    total_loss = 0
    hidden = model.init_hidden(batch_size)
    for batch, i in enumerate(range(0, train_data.size(0) - 1, seq_length)):
        data, targets = get_batch(train_data, i, seq_length)
        hidden = repackage_hidden(hidden)
        model.zero_grad()
        output, hidden = model(data, hidden)
        loss = criterion(output, targets)
        loss.backward()

        clipped_lr = lr * clip_gradient(model, grad_clip)
        for p in model.parameters():
            p.data.add_(-clipped_lr, p.grad.data)

        print('.', end='', flush=True)
        total_loss += loss.data
    return total_loss[0] / batch

def save_model(model, name, checkpoint = ''):
    filename = name + str(checkpoint) + '.pt'
    print('Saving model to', filename)
    with open(filename, 'wb') as f:
        torch.save(model, f)
    

def main():
    args = read_arguments()
    print("Loading data...")
    ecg_data, val_data, test_data = load_data(args.data, args.batch_size)
    print("Building network ...")
    ###############################################################################
    # Build the model
    ###############################################################################
    model = RNNModel(args.model, args.nhid, args.nlayers)
    if torch.cuda.is_available():
        print('Using CUDA')
        model.cuda()

    criterion = nn.MSELoss()

    print("Training network ...")
    try:
        lr = args.lr
        ci = args.checkpoint_interval
        filename = args.save
        print('Learning rate {:.5f}'.format(lr))
        prev_loss = None
        for epoch in range(1, args.epochs+1):
            epoch_start_time = time.time()
            train_loss = train(ecg_data, model, criterion, lr, args.batch_size, args.bptt, args.clip)
            val_loss = evaluate(val_data, model, criterion, args.batch_size, args.bptt)
            print()
            print('-' * 89)
            print('| end of epoch {:3d} | time: {:5.2f}s | '
                'train loss {:5.2f} | val loss{:5.2f}'.format(epoch, (time.time() - epoch_start_time),
                                        train_loss, val_loss))
            print('-' * 89)
            if not (epoch % ci):
                save_model(model, filename, epoch)
            if prev_loss and val_loss > prev_loss:
                lr /= 4
                print('New learning rate {:.5f}'.format(lr))
            prev_loss = val_loss
    except KeyboardInterrupt:
        pass
    finally:
        print()
        test_loss = evaluate(test_data, model, criterion, args.batch_size, args.bptt)
        print('=' * 89)
        print('| End of training | test loss {:5.2f} |'.format(
            test_loss))
        print('=' * 89)
        save_model(model, filename)

if __name__ == '__main__':
    main()<ECODE>

apaszke: It’s hard to tell why it doesn’t learn anything. There are a lot of factors that can cause it, and they depend on the data, preprocessing, model, etc. Maybe it’s one of the things I mentioned below, maybe something else. If the expected output given the input is equal to the input, and the network can’t find any patterns in your data, it’s quite logical that the only thing it can do to optimize the loss is to simply return what it got. Another thing is that if the outputs barely differ from the inputs, the loss value will be very small even if the network returns the input.

drscotthawley: 

AndreaCogliati: <SCODE>#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Sample from an LSTM network to generate a time series.
'''

import argparse
import time
import math

import torch
import torch.nn as nn
from torch.autograd import Variable
from train import RNNModel

def read_arguments():
    parser = argparse.ArgumentParser(description='Sample from a recurrent network to generate a signal.')
    parser.add_argument('--checkpoint', type=str, default='./model.pt',
                        help='model checkpoint to use')
    parser.add_argument('--outf', type=str, default='generated.txt',
                        help='output file for generated signal')    
    parser.add_argument('--length', type=int,  default='12500',
                        help='length of the output signal')
    args = parser.parse_args()
    return args


def main():
    args = read_arguments()
    print("Loading model...")
    with open(args.checkpoint, 'rb') as f:
        model = torch.load(f)
    if torch.cuda.is_available():
        model.cuda()
    else:
        model.cpu()

    print('Sampling...')
    try:
        hidden = model.init_hidden(1)
        output = Variable(torch.zeros(1,1,1), volatile=True)
        # output.data[0] = 0.0
        if torch.cuda.is_available():
            output.data = output.data.cuda()
        with open(args.outf, 'w') as outf:
            for _ in range(args.length):
                output, hidden = model(output, hidden)
                outf.write('{:.5f}\n'.format(output.squeeze().data.cpu()[0]))
    except KeyboardInterrupt:
        pass

if __name__ == '__main__':
    main()<ECODE>

drscotthawley: I took the code that you posted and I found that, for my data, the predicted solution moves rapidly to zero.  I trained with a simple sine wave (black) and got the red line when I ran your prediction code… (this is zoomed in towards the beginning… the training data goes on for 50,000 timesteps)

drscotthawley: Perhaps that would be helpful for those more in-the-know, as far as making the PyTorch implementation match the Torch implementation.  I’m not at the stage where I can do that yet. I don’t see a Sequencer() layer defined in PyTorch though;  perhaps it’s not necessary.

AndreaCogliati: Thanks for your interest. Yes, that’s the very same behavior I’m observing, in general. With longer training, and longer backpropagation through time I was able to generate simple sinusoidal signals and even a square wave, but with more complex signals the network always ends up to a fixed point (not necessarily zero). And thanks for listing the post by Element Research. It looks like that my naive model is not completely off-track.

Russel_Russel: Can you please explain what is the purpose of your read_argumnets function? What does the argparser do here (I always see it in all pytorch examples? Also, what is an application of generating time series data (I mean how would that be useful)?  Please excuse my limited knowledge, as I am still new to deep_learning. Thanks for the help.

AndreaCogliati: 

csarofeen: What kind of “more complex signals” are you having trouble with?

AndreaCogliati: For instance a combination of 3 sinusoids.

osm3000: 

AndreaCogliati: No, I didn’t, sorry. However, I haven’t spent much on it lately, since I have been busy on other projects.

drscotthawley: Can someone comment on this?  It’s odd when even the example code doesn’t work.

tom: I get not perfect but reasonable predictions (the following image is after 25 steps) once the MSE is around 1e-5 to 3e-5. Sometimes the amplitudes vary more wildly, but that might be expected. What is the MSE you get? Best regards Thomas

drscotthawley: So, what is it that you’re doing that’s different from the rest of us? One thing seems to be that you’re running for more steps (25): the example stops at 15.  But I don’t think that would make a difference, because the loss seems to “flatten out” fairly early, e.g. by step 3.  Output is… …and all the predict*.pdf files from predict2.pdf onward look like the graph I posted.

tom: Hello Scott, Best regards Thomas

drscotthawley: 

tom: Best regards Thomas

drscotthawley: Thanks Thomas!  That’s it.  I confirm that your code yields the pictured result.  So glad to be able to move on. I’m also able to run the example code now that your pull request has been accepted. I’ll consider my part of this thread as “closed”.