How can i compute seq2seq loss using mask?

yunjey: In TensorFlow, i can do this as below. <SCODE># targets is an int64 tensor of shape (batch_size, padded_length) which contains word indices.    
# masks is a tensor of shape (batch_size, padded_length) which contains 0 or 1 (0 if pad otherwise 1).

outputs = decoder(...)  # unnormalized scores of shape (batch_size, padded_length, vocab_size) 
outputs = tf.reshape(outputs, (-1, vocab_size))
targets = tf.reshape(targets, (-1))
losses = tf.nn.sparse_softmax_cross_entropy_loss(outputs, targets)    # loss of shape (batch_size*padded_length)

masks = tf.reshape(masks, (-1))
loss = losses * masks    
<ECODE> <SCODE>criterion = nn.CrossEntropyLoss()
outputs = decoder(features, inputs)   # (batch_size, padded_length, vocab_size)
loss = criterion(outputs.view(-1, vocab_size), targets.view(-1))  # this gives a scalar not tensor
<ECODE> How can i solve this problem?

jekbradbury: 

yunjey: <SCODE>cap = CocoCaptions(root = './data/train2014resized',
                   annFile = './data/annotations/captions_train2014.json',
                   vocab = vocab,
                   transform=transform,
                   target_transform=transforms.ToTensor())

data_loader = torch.utils.data.DataLoader(
    cap, batch_size=16, shuffle=True, num_workers=2)

for i, (images, input_seqs, target_seqs, masks) in enumerate(data_loader):
    # images: a tensor of shape (batch_size, 3, 256, 256).
    # input_seqs, target_seqs, masks: tensors of shape (batch_size, padded_length).<ECODE>

jekbradbury: <SCODE>import torch.nn.utils.rnn as rnn_utils  
lstm = nn.LSTM(in_size, hidden_size, bidirectional, num_layers)
packed = rnn_utils.pack_padded_sequence(padded, lengths)
packed_out, packed_hidden = lstm(packed)
unpacked, unpacked_len = rnn_utils.pad_packed_sequence(packed_out)<ECODE>

supakjk: 

apaszke: 

supakjk: 

jekbradbury: 

supakjk: Could you let me know how to deal with the following example case? Assume that I have an input, where maximum sequence length is 5, minibatch size is 4, and there are 3 possible labels. idxs has effective lengths of sequences in the input. <SCODE>input = Variable(torch.randn(5,4,3))
idxs = [5,3,3,2]
target = Variable(torch.LongTensor(5,4))
# assume the target labels are assigned corresponding to the input.
<ECODE> Thanks!

pranav: 

apaszke: 

Response777: Then the cross_entropy loss could calculated with <SCODE>def nll_loss(outputs, targets):
    return torch.gather(outputs, 1, targets.view(-1,1))
<ECODE>

vijendra_rana: Can you help me understand this torch.gather I have used the same tensors  with similar shape but this line torch.gather(outputs, 1, targets.view(-1,1)) gives me error.

volpato30: Thanks for sharing the code. I test the nll_loss function and get: RuntimeError: Input tensor must have same dimensions as output tensor at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:424 It seems that the input and index arguments of gather function should have same dimensions.

volpato30: currently I use this way to compute masked loss for seq2seq model: <SCODE>'''
target_sentences: [batch_size, max_length] int64
batch_target_len_array: [batch_size] int64, each element is the valid length of the corresponding target sentence. 
'''
for step in range(max_length):
    decoder_output, decoder_hidden = decoder(
        decoder_input, decoder_hidden, encoder_outputs)
    if np.sum(batch_target_len_array > step) == 0:
        break
    mask_vector = torch.from_numpy((batch_target_len_array > step).astype(np.int32)).byte()
    index_vector = Variable(torch.masked_select(torch.arange(0, batch_size), mask_vector).long()).cuda()
    valid_output = torch.index_select(decoder_output, 0, index_vector)
    valid_target = torch.index_select(target_sentences[:, step], 0, index_vector)

    step_loss = criterion(valid_output, valid_target)
    loss += step_loss
    decoder_input = target_sentences[:, step]  # Teacher forcing<ECODE>

cbaziotis: <SCODE>loss_function = nn.CrossEntropyLoss(ignore_index=0)
<ECODE> since i have zero padded targets.

Madotto_Andrea: I have tried, doesn’t work in my case. Could you post your code with a bit more details?

Littleye: It is clear. I’ll try! The newest version 0.3 updated the cost function with a new Parameter “reduce”, and it’s easy to mask the loss

himat: Can you explain how to mask the loss now?

jpzhou: With the parameter “reduce”, we can get the loss per batch element, but how can we use the mask on it? For example, if I have a minibatch whose valid sequence lengths is [3,1,2], and with “reduce” we can get three loss values [L1, L2, L3]. But we need to mask the last two values in calculating L2 and last one value in calculating L3. How could it be achieved? ------------------------------edited---------------------------- Oh I see, we only need to use ignore_index=0 parameter in loss(), but don’t need the “reduce” parameter to achieve this. Is it right?