Do I need to pad non-fixed examples or does run_language_modeling.py already takes care of that? 
Author: timsoraro
Pull request: no
Labels: name: wontfix

I use a custom TextDataset in [examples/run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py). I can't figure if I need to pad the examples to `bucket_size` myself, or is it already been taken care of in [L221-L223](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py#L221-L223)?

Thanks!

Comments:
I believe that the `collate` function should take care of it. However, you will need to create the `attention_mask` variable when you have inputs of variable length, so that the model does not attend to the padded indices.

@Genius1237 I'm a real noob (I come from a programming background and no ML), but I'm using the GPT2 model, not a masked-language modeling like BERT. Do I still need to do such a thing? I didn't see any `attention_mask` reference in TextDataset loaders in the example file.

Also, how does the `collate` function knows to pad the examples to `bucket_size` if no such variable is passed to it?

The `TextDataset` class takes text and converts it into blocks of size `block_size` (512), concatenating consecutive blocks if needed. My guess is that `LineByLineTextDataset` exists to cater to those who would like to have examples being limited to one sentence each, and thus the max size in one batch would be determined by the longest sentence in that batch.

`attention_mask` is definitely needed when you have sequences of different length. Have a look at Have a look at https://github.com/huggingface/transformers/issues/1408 . Something like this should do on top of the existing version of `LineByLineTextDataset`.
```
def collate(examples: List[torch.Tensor]):
        padding_value = 0 if tokenizer._pad_token is None else tokenizer.pad_token_id
        input_ids = pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)

        max_length = input_ids.shape[1]
        attention_mask = torch.stack([torch.cat([torch.ones(len(t), dtype=torch.long), torch.zeros(max_length - len(t), dtype=torch.long)]) for t in examples])

        return input_ids, attention_mask
```

Gotcha, thank you so much @Genius1237 ! I hope it would work.
Do I need to unpack `attention_mask` somehow or the collate function in the DataLoader will take care of that?

And in your code you probably meant to wrote this instead, right?

```python
input_ids = pad_sequence(examples, batch_first=True, padding_value=padding_value)
```

Also, the same code should be pasted in ```def evaluate():``` right?

Same code in evaluate. The collate function is called by the dataloader. The dataloader calls the __getitem__ on the dataset `batch_size` times and sends that output to the collate function. The output of the collate function is what you will get when you do `for batch in train_dataloader`. The batch in this case will be a 2 tuple, with `batch[0]` having the input_ids and `batch[1]` having the attention_mask.

I really appreciate your help, thanks a lot. I'll ping the maintainers to change the example code so others can benefit. @thomwolf @LysandreJik @patrickvonplaten Thanks!

Hey, @Genius1237, I'm getting this error using your code:

```
Traceback (most recent call last):
  File "run_language_modeling.py", line 974, in <module>
    main()
  File "run_language_modeling.py", line 924, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 508, in train
    inputs = inputs.to(args.device)
AttributeError: 'tuple' object has no attribute 'to'
```
Using the previous code I'm not getting any error.

When you do `for batch in train_dataloader`, batch is basically whatever is returned by the `collate` function, which is this cause becomes a 2-tuple. In your case `input_ids` is a 2-tuple containing 2 tensors. You'll have to split that into 2, i.e `input_ids, attention_mask = inputs`, and move forward with that, pushing both those tensors to the required device (`input_ids = input_ids.to(args.device); attention_mask = attention_mask.to(args.device)`).

Can you please help me? I'm really clueless...

I tried:

```python
for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):

            # Skip past any already trained steps if resuming training
            if steps_trained_in_current_epoch > 0:
                steps_trained_in_current_epoch -= 1
                continue

            inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
            input_ids, attention_mask = inputs
            inputs = input_ids.to(args.device)
            labels = labels.to(args.device)
            attention_mask = attention_mask.to(args.device)
            model.train()
            outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
            loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
```

But I get this error:
```
Traceback (most recent call last): 
  File "run_language_modeling.py", line 976, in <module>
    main()
  File "run_language_modeling.py", line 926, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 510, in train
    labels = labels.to(args.device)
AttributeError: 'tuple' object has no attribute 'to'
```

Also, the rest of the code doesn't seem to use `attention_mask` variable, wouldn't it be redundant?

```
for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):

            # Skip past any already trained steps if resuming training
            if steps_trained_in_current_epoch > 0:
                steps_trained_in_current_epoch -= 1
                continue

            input_ids, attention_mask = batch
            inputs, labels = mask_tokens(input_ids, tokenizer, args) if args.mlm else (input_ids, input_ids)
            inputs = inputs.to(args.device)
            labels = labels.to(args.device)
            attention_mask = attention_mask.to(args.device)
            model.train()
            outputs = model(inputs, masked_lm_labels=labels, attention_mask=attention_mask) if args.mlm else model(inputs, labels=labels, attention_mask=attention_mask)
            loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
```

Thank you, it works!

Btw, I don't see a difference in output between training using the attention mask and the original code. Does it mean something?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


 In training a GPT-2 i have the same question too. I don't see any significant change between training using the attention mask or without. Did u have any answer to this @timsoraro ?
Also my perplexity score i can say is too low, any idea for this behaviour?



@niklaras I didn't see much difference either after many experiments with or without, I got the same quality of generation.