loss.backward() being called twice in Trainer._training_step()
Author: aclifton314
Pull request: no
Labels: name: wontfix

**Setup**
pytorch: 1.5.1
huggingface transformers: 3.0.2
python: 3.7.6
OS: Pop!_OS 20.04 on VM

**Sample Code**
```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel, TrainingArguments, Trainer
import torch
from torch.utils.data import Dataset
import sys
import pandas as pd

ZERO = sys.float_info.min
ZERO_PT = torch.tensor(ZERO)

class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
        self.tokenizer.pad_token = self.tokenizer.eos_token

    def eval_sentence(self, sent: str):
        vec = torch.tensor(sentence_vec(sent), dtype=torch.float, requires_grad=True) # remove punct, lower case, split on space, prepend "
", postpend "
" start and stop tokens. Returns tensor of ints of vocab.
        last_idx = min(max_ngram, len(vec)) #max_ngram is an int
        probs = [max(ZERO_PT, pkatz(vec[0:i])) for i in range(2, last_idx + 1)] #pkatz is katz backoff probability and returns a tensor with grad function set.
        for i in range(1, len(vec) - last_idx + 1):
            j = i + last_idx
            probs.append(max(ZERO_PT, pkatz(vec[i:j])))

        probs = torch.stack(probs)
        log_probs = torch.log(probs)
        log_prob = torch.sum(log_probs)
        len_tensor = torch.tensor(len(vec), dtype=float, requires_grad=True)
        final_prob = torch.true_divide(-log_prob, len_tensor)
        return final_prob

    def sentence_loss(self, sent: str):
        p, l = self.eval_sentence(sent)
        return -p

    def generate_text_while_finetuning(self,
                                       input_ids=None,
                                       past=None,
                                       attention_mask=None,
                                       token_type_ids=None,
                                       position_ids=None,
                                       head_mask=None,
                                       inputs_embeds=None,
                                       labels=None,
                                       use_cache=None,
                                       output_attentions=None,
                                       output_hidden_states=None, ):
        transformer_outputs = self.transformer(
            input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
        )
        hidden_states = transformer_outputs[0]
        lm_logits = self.lm_head(hidden_states)
        outputs = (lm_logits,) + transformer_outputs[1:]
        return outputs  # (loss), lm_logits, presents, (all hidden_states), (attentions)

    def forward(
            self,
            input_ids=None,
            past=None,
            attention_mask=None,
            token_type_ids=None,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
            labels=None,
            use_cache=True,
    ):

        max_length = input_ids.shape[1] + 50
        full_generated_gpt2_ids = self.generate(input_ids=input_ids,
                                                max_length=max_length,
                                                is_finetuning_current_model=True,
                                                attention_mask=attention_mask,
                                                pad_token_id=50256,
                                                do_sample=True,
                                                top_k=50,
                                                top_p=0.95)

        decoded_gen_samples = self.tokenizer.batch_decode(full_generated_gpt2_ids, skip_special_tokens=True)
        tmp_losses = [self.sentence_loss(decoded_sample) for decoded_sample in decoded_gen_samples]
        losses = torch.stack(tmp_losses)
        loss = losses.mean()
        return (loss,)


##The code below is the run script.
class MyDataset(Dataset):
    def __init__(self, csv_file: str):
        self.df = pd.read_csv(csv_file, encoding='ISO-8859-1')

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        text = self.df.iloc[idx, 1]
        return text

def my_data_collator(dataset_samples_list):
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
    tokenizer.pad_token = tokenizer.eos_token

    encoded_results = tokenizer(dataset_samples_list, padding=True, truncation=True, return_tensors='pt', return_attention_mask=True)

    batch = {}
    batch['input_ids'] = torch.stack([result for result in encoded_results['input_ids']])
    batch['past'] = None
    batch['attention_mask'] = torch.stack([result for result in encoded_results['attention_mask']])
    batch['position_ids'] = None
    batch['head_mask'] = None
    batch['inputs_embeds'] = None
    batch['labels'] = None
    batch['use_cache'] = True
    return batch

dataset_train = MyDataset('/path/to/train_dataset.csv')

training_args = TrainingArguments(
    output_dir='/path/to/out',
    do_train=True,
    per_device_train_batch_size=64,
    logging_dir='/path/to/dir',
    max_steps=300000
)

model = GPT2FinetunedWithNgrams.from_pretrained('gpt2')

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=my_data_collator,
    train_dataset=dataset_train
)
trainer.train()
trainer.save_model('/path/to/model_save_dir')

```

**Issue**
The above code will produce the following error for some training examples:
```python
Traceback (most recent call last):
  File "/home/aclifton/ric-2020/textgen/run_finetune_gpt2.py", line 221, in <module>
    testfinetune()
  File "/home/aclifton/ric-2020/textgen/run_finetune_gpt2.py", line 215, in testfinetune
    trainer.train()
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 637, in _training_step
    loss.backward()
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

```

What I'm finding is that in `Trainer._training_step()`, some examples are causing `loss.backward()` to be called twice. So the error makes sense as the first call will compute the graph then clear it and the second call is what throws the error. I'm not sure what would cause this to happen and was wondering if others might have an idea?



Comments:
I'm not sure how we can expect this to work with the encoding and the usage of the `generate` method directly in the forward method.

@patrickvonplaten can chime in if I'm wrong, but I believe the `generate` method can not be back-propagated through as it is right now.

@LysandreJik I came across one issue (#6105) with `generate` being used in `forward`. My temporary workaround was to introduce `is_finetuning_current_model` into `generate` that will call `generate_text_while_finetuning` instead of `forward` again to avoid the recursion.

I'm still learning pytorch so I might be wrong on this, and correct me if I am. I checked the `grad_fn` for `input_ids`, `full_generated_gpt2_ids`,  and each of them were set to `None`. `tmp_losses`, `losses`, and `loss` all had their `grad_fn` set. My naive assumption is that backpropagation will run up to `tmp_losses`, skip over the `generate` part, and then continue on through the gpt2 model.

Another interesting point is that I get the error on different training examples. I set the batch size to 1 and it would produce the error on, say, example 5. Removing example 5 from the training set and rerunning would cause the error on example 3, etc.

yes, the `generate()` cannot be used for backpropagation at the moment . 

@patrickvonplaten Would that explain why I'm encountering the above error? Do you also mind elaborating on why `generate()` cannnot be usef for backpropagation? I'm interested to hear the details for the sake of my own knowledge.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.