Recursive error calling generate in forward
Author: aclifton314
Pull request: no
Labels: name: wontfix

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 3.0.2
Python: 3.7.6

## Question
Here is the training loop: 
```python
def sd_data_collator(dataset_samples_list):
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
    tokenizer.pad_token = tokenizer.eos_token

    encoded_results = tokenizer(dataset_samples_list, padding=True, truncation=True, return_tensors='pt', return_attention_mask=True)

    batch = {}
    batch['input_ids'] = torch.stack([result for result in encoded_results['input_ids']])
    batch['past'] = None
    batch['attention_mask'] = torch.stack([result for result in encoded_results['attention_mask']])
    batch['position_ids'] = None
    batch['head_mask'] = None
    batch['inputs_embeds'] = None
    batch['labels'] = None
    batch['use_cache'] = True
    return batch

sd_dataset = SDAbstractsDataset('/path/to/sd_samples_64.csv')

training_args = TrainingArguments(
    output_dir='/path/to/finetuned_gpt2',
    do_train=True,
    per_device_train_batch_size=4,
    learning_rate=1e-3,
    num_train_epochs=1
)

model = GPT2FinetunedWithNgrams.from_pretrained('gpt2')

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=sd_dataset,
    data_collator = sd_data_collator
)

trainer.train()
```
Here's the model class and its `forward` method:
```python
class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
        self.tokenizer.pad_token = self.tokenizer.eos_token

    def load_ngrams_model(self, ngrams_model_path):
        self.ngrams_model = NGrams(ngrams_model_path)

    def forward(
            self,
            input_ids=None,
            past=None,
            attention_mask=None,
            token_type_ids=None,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
            labels=None,
            use_cache=True,
    ):

        output = self.generate(input_ids=input_ids, max_length=474)
        decoded_output = self.tokenizer.decode(output[0], skip_special_tokens=True)
```
Here's the whole error. It's really lengthy and I cut out the repetitions:
```python
Some weights of GPT2FinetunedWithNgrams were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Iteration:   0%|          | 0/16 [00:00<?, ?it/s]Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
.
.
.
      File "/path/to/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 480, in generate
    model_specific_kwargs=model_specific_kwargs,
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 520, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/path/to/ric-2020/text_gen_w_transformers/finetune_gpt2.py", line 33, in forward
    
  File "/path/to/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 350, in generate
    "Setting `pad_token_id` to {} (first `eos_token_id`) to generate sequence".format(eos_token_id)
.
.
.
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1390, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1514, in _log
    self.handle(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1594, in callHandlers
    lastResort.handle(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 894, in handle
    self.emit(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1025, in emit
    msg = self.format(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 869, in format
    return fmt.format(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 608, in format
    record.message = record.getMessage()
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 360, in getMessage
    def getMessage(self):
RecursionError: maximum recursion depth exceeded while calling a Python object
```
My guess is the `self.generate()` being called within the model produces the recursion problem. I found this problematic because the `generate` method has some awesome functionality for beam search, greedy search, top-k, etc. To overcome this, I added a flag to `generate` called `is_finetuning_current_model`:
```python
@torch.no_grad()
    def generate(
        self,
        input_ids: Optional[torch.LongTensor] = None,
        max_length: Optional[int] = None,
        min_length: Optional[int] = None,
        do_sample: Optional[bool] = None,
        early_stopping: Optional[bool] = None,
        num_beams: Optional[int] = None,
        temperature: Optional[float] = None,
        top_k: Optional[int] = None,
        top_p: Optional[float] = None,
        repetition_penalty: Optional[float] = None,
        bad_words_ids: Optional[Iterable[int]] = None,
        bos_token_id: Optional[int] = None,
        pad_token_id: Optional[int] = None,
        eos_token_id: Optional[int] = None,
        length_penalty: Optional[float] = None,
        no_repeat_ngram_size: Optional[int] = None,
        num_return_sequences: Optional[int] = None,
        attention_mask: Optional[torch.LongTensor] = None,
        decoder_start_token_id: Optional[int] = None,
        use_cache: Optional[bool] = None,
        is_finetuning_current_model: Optional[bool] = None,
        **model_specific_kwargs
    ) -> torch.LongTensor:
```
propagated this down to the `num_beams` check:
```python
        if num_beams > 1:
            output = self._generate_beam_search(
                input_ids,
                cur_len=cur_len,
                max_length=max_length,
                min_length=min_length,
                do_sample=do_sample,
                early_stopping=early_stopping,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                repetition_penalty=repetition_penalty,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
                batch_size=effective_batch_size,
                num_return_sequences=num_return_sequences,
                length_penalty=length_penalty,
                num_beams=num_beams,
                vocab_size=vocab_size,
                encoder_outputs=encoder_outputs,
                attention_mask=attention_mask,
                use_cache=use_cache,
                is_finetuning_current_model=is_finetuning_current_model,
                model_specific_kwargs=model_specific_kwargs
            )
        else:
            output = self._generate_no_beam_search(
                input_ids,
                cur_len=cur_len,
                max_length=max_length,
                min_length=min_length,
                do_sample=do_sample,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                repetition_penalty=repetition_penalty,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
                batch_size=effective_batch_size,
                encoder_outputs=encoder_outputs,
                attention_mask=attention_mask,
                use_cache=use_cache,
                is_finetuning_current_model=is_finetuning_current_model,
                model_specific_kwargs=model_specific_kwargs
            )
```
updated `_generate_no_beam_search` and `_generate_beam_search` with the following:
```python
            if is_finetuning_current_model:
                outputs = self.generate_text_while_finetuning(**model_inputs)
            else:
                outputs = self(**model_inputs)
```
For my model class, I just added the `generate_text_while_finetuning` method and set the `is_finetuning_current_model`:
```python
class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
        self.tokenizer.pad_token = self.tokenizer.eos_token

    def load_ngrams_model(self, ngrams_model_path):
        self.ngrams_model = NGrams(ngrams_model_path)

    def generate_text_while_finetuning(self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        use_cache=None,
        output_attentions=None,
        output_hidden_states=None,):

        transformer_outputs = self.transformer(
            input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
        )
        hidden_states = transformer_outputs[0]

        lm_logits = self.lm_head(hidden_states)

        outputs = (lm_logits,) + transformer_outputs[1:]
        return outputs  # (loss), lm_logits, presents, (all hidden_states), (attentions)

    def forward(
            self,
            input_ids=None,
            past=None,
            attention_mask=None,
            token_type_ids=None,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
            labels=None,
            use_cache=True,
    ):

        output = self.generate(input_ids=input_ids, max_length=474, is_finetuning_current_model=True)
        decoded_output = self.tokenizer.decode(output[0], skip_special_tokens=True)
```
This seems to resolve the recursive error and produces an expected `decoded_output` for me. My usecase is using GPT2 with a different loss function to finetune it on a particular domain corpus. I imagine other people would be doing something similar for GPT2 and other models, so I tested this approach just using `GPT2LMHeadModel` and got the same expected results. 

My question is do contributors think I should open up a bug report for this?

Comments:
cool workaround! I think you might have a cleaner solution, potentially, if you compose instead of inheriting from `GPT2WithLMHead`. This is not worthy of a bug report, (what's the bug), but it could be an interesting proposal/project for examples/ if it works well on a task with a public dataset.

Could I hear more about your task?
Are you successfully backpropagating through beam search?

Hi @sshleifer, thanks for your reply!
I wasn't quite sure if it would warrant a bug report or feature suggestion (or neither). Thanks for clearing that up.

The task I am doing is text generation. I have a dataset of scientific abstracts that I want to finetune the GPT2 pretrained model on to generate similar abstracts. However, I wanted to replace the loss with a loss from a N-grams model I have. The procedure looks something like this:

- Feed sample abstract into Pre-trained GPT2.
- Generate a sequence of specified length based off that sample.
- Calculate the loss using the N-grams model I have and use that loss for backpropagation.

Basically I am replacing the loss function found in `GPT2LMHeadModel` with my own and utilizing the `generate` method in `GPT2Pretrained` to generate new abstracts. I was doing the generation one token at a time using a naive method, but the `generate` method is so handy for the generation that I really wanted to utilize it (and all the hard work the HF team has put in).

I have not tried to backpropagate yet. You'll notice most of the arguments that go into `Trainer` are pretty lousy. Right now, I just want to see if it will start training with no errors. I hope to try to do some more thoughtful training later this week.

Any further thoughts on this?

Nope. Excited to see what code modifications are required to get this working!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.