Clarifying input size to RNN in word_language_model example?

mcskwayrd: Hi.  I’m trying to understand something… How is this possible?  If the model is expecting 20 inputs, shouldn’t it produce an error when you try to send it only 1? Furthermore, when I try to actually send the generation code a sequence of length 20 by creating… <SCODE>input = corpus.test[0:20]
print("input =",input)
<ECODE> Then I get… <SCODE>corpus = data.Corpus(args.data)
ntokens = len(corpus.dictionary)
hidden = model.init_hidden(1)

def batchify(data, bsz):       # breaks into parallel streams
    nbatch = data.size(0) // bsz
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1).t().contiguous()
    if args.cuda:
        data = data.cuda()
    return data

eval_batch_size = 10
test_data = batchify(corpus.test, eval_batch_size)
  
def get_batch(source, i, evaluation=False):
    bptt = 20
    seq_len = min(bptt, len(source) - 1 - i)
    data = Variable(source[i:i+seq_len], volatile=evaluation)
    target = Variable(source[i+1:i+1+seq_len].view(-1))
    return data, target

#input = Variable(torch.rand(1, 1).mul(ntokens).long(), volatile=True)
input, targets = get_batch(test_data, 0, evaluation=True)
<ECODE> Then I when I get to the prediction step (i.e., " output, hidden = model(input, hidden)" ), I get the error… PS- I see the documentation for torch.nn.RNN says input is supposed to be a Tensor, but that’s just what I’m sending.  It didn’t say anything about needing a Variable or other “matrix”: Thanks!

apaszke: The confusion comes from a dynamic vs static graph framework. PyTorch constructs the graphs every time, so it doesn’t care in advance what length of the sequence will you be using with the RNN. The only arguments that you have to pass in to the constructor of the RNN are how many features should the input have, and what’s the hidden layer size. Then, you can use sequences of different lengths at every iteration, and it should work just fine. The only note that can lower the memory usage is to forward a fake batch before the training, that’s of the size of the longest sequence. This will allow our CUDA allocator to preallocate memory that can be reused for all (smaller) batches.

mcskwayrd: Thanks for writing back Adam.  So this is the “flexible input size” feature I’ve been hearing so much about. Great! <SCODE>eval_batch_size = 10
test_data = batchify(corpus.test, eval_batch_size)

def get_batch(source, i, evaluation=False):
    bptt = 20
    seq_len = min(bptt, len(source) - 1 - i)
    data = Variable(source[i:i+seq_len], volatile=evaluation)
    target = Variable(source[i+1:i+1+seq_len].view(-1))
    return data, target

input, target = get_batch(test_data, 0, evaluation=True)
#input = Variable(torch.rand(1, 1).mul(ntokens).long(), volatile=True)

input = Variable(input, volatile=True)
<ECODE> But if it’s already a Variable, then I don’t understand why I can’t use it as an input to “model” further below.   (?) Because if I don’t include that extra “Variable” re-casting, then still I get “inconsistent tensor size”… What is the source of this inconsistency, if the input length isn’t supposed to matter?

apaszke: 

mcskwayrd: Ok, removed the rewrap. <SCODE>corpus = data.Corpus(args.data)
ntokens = len(corpus.dictionary)
hidden = model.init_hidden(1)
# input = Variable(torch.rand(1, 1).mul(ntokens).long(), volatile=True)
def batchify(data, bsz):       # breaks into parallel streams
    nbatch = data.size(0) // bsz
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1).t().contiguous()
    if args.cuda:
        data = data.cuda()
    return data
eval_batch_size = 10
test_data = batchify(corpus.test, eval_batch_size)
def get_batch(source, i, evaluation=False):
    bptt = 20
    seq_len = min(bptt, len(source) - 1 - i)
    data = Variable(source[i:i+seq_len], volatile=evaluation)
    target = Variable(source[i+1:i+1+seq_len].view(-1))
    return data, target
input, target = get_batch(test_data, 0, evaluation=True)
<ECODE> Running this version of the code produces an error about hidden size… It seems that it wants the hidden size to somehow follow the batch size, only with no "L"s…

apaszke: 

mcskwayrd: The only issue is that “output” then ends up being [20x1x10000] instead of [1x1x10000] like the remainder of the code expects.  So I grab only the last element of output via output = output[-1] <SCODE>###############################################################################
# Language Modeling on Penn Tree Bank
#
# This file generates new sentences sampled from the language model
#
###############################################################################

import argparse
import time
import math

import torch
import torch.nn as nn
from torch.autograd import Variable

import data

parser = argparse.ArgumentParser(description='PyTorch PTB Language Model')

# Model parameters.
parser.add_argument('--data', type=str, default='./data/penn',
                    help='location of the data corpus')
parser.add_argument('--checkpoint', type=str, default='./model.pt',
                    help='model checkpoint to use')
parser.add_argument('--outf', type=str, default='generated.txt',
                    help='output file for generated text')
parser.add_argument('--words', type=int, default='1000',
                    help='number of words to generate')
parser.add_argument('--seed', type=int, default=1111,
                    help='random seed')
parser.add_argument('--cuda', action='store_true',
                    help='use CUDA')
parser.add_argument('--temperature', type=float, default=1.0,
                    help='temperature - higher will increase diversity')
parser.add_argument('--log-interval', type=int, default=100,
                    help='reporting interval')
args = parser.parse_args()

# Set the random seed manually for reproducibility.
torch.manual_seed(args.seed)
if torch.cuda.is_available():
    if not args.cuda:
        print("WARNING: You have a CUDA device, so you should probably run with --cuda")
    else:
        torch.cuda.manual_seed(args.seed)

if args.temperature < 1e-3:
    parser.error("--temperature has to be greater or equal 1e-3")

with open(args.checkpoint, 'rb') as f:
    model = torch.load(f)

if args.cuda:
    model.cuda()
else:
    model.cpu()


def batchify(data, bsz):       # breaks into parallel streams
    nbatch = data.size(0) // bsz
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1).t().contiguous()
    if args.cuda:
        data = data.cuda()
    return data

corpus = data.Corpus(args.data)
ntokens = len(corpus.dictionary)
def batchify(data, bsz):       # breaks into parallel streams
    nbatch = data.size(0) // bsz
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1).t().contiguous()
    if args.cuda:
        data = data.cuda()
    return data
eval_batch_size = 1
test_data = batchify(corpus.test, eval_batch_size)
hidden = model.init_hidden(1)
def get_batch(source, i, evaluation=False):
    bptt = 20
    seq_len = min(bptt, len(source) - 1 - i)
    data = Variable(source[i:i+seq_len], volatile=evaluation)
    target = Variable(source[i+1:i+1+seq_len].view(-1))
    return data, target

input, target = get_batch(test_data, 0, evaluation=True)
#input = Variable(torch.rand(1, 1).mul(ntokens).long(), volatile=True)
#print("input = ",input)

if args.cuda:
    input.data = input.data.cuda()

with open(args.outf, 'w') as outf:
    for i in range(args.words):
        output, hidden = model(input, hidden)
        output = output[-1]
#        print("output = ",output)
        word_weights = output.squeeze().data.div(args.temperature).exp().cpu()
        word_idx = torch.multinomial(word_weights, 1)[0]
        input.data.fill_(word_idx)
        word = corpus.dictionary.idx2word[word_idx]

        outf.write(word + ('\n' if i % 20 == 19 else ' '))

        if i % args.log_interval == 0:
            print('| Generated {}/{} words'.format(i, args.words))

print(" ")<ECODE>

apaszke: Actually that might not be what you want. You want to pass the large input only once, to initialize the network, and then do the steps one by one. In this example you’ll forward a sequence of 20 words from the data, and then you’ll be feeding each output for 20 steps, and taking the last one as the next input (that will be applied 20 times). You should forward the batch through the network only once and slice off the last hidden state. Then, use that slice with an input of length 1 to generate the data.

mcskwayrd: