Big Bird generate() "local variable 'next_tokens' referenced before assignment"
Author: OscarGarciaF
Pull request: no

I am facing this problem when doing text summarization. I am using google/bigbird-roberta-base and I get the following error when calling model.generate(input, max_length = 4096, num_beams=4, early_stopping=True, length_penalty = 0.8):

```
Input length of input_ids is 4096, but ``max_length`` is set to 4096.This can lead to unexpected behavior. You should consider increasing ``config.max_length`` or ``max_length``.
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-13-90a633800ba7> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', ' \ni = 0\nsize = 1\nout = []\nend = False\nprint_iters = 100\nsave_iters = 5\n \nwhile True:\n  if (i+size) >= n:\n    last = n\n    end = True\n  else:\n    last = i + size   \n \n  result = make_gen(   model_sum, tokens[i:last, :].detach().clone()  )\n \n  for j in range(result.shape[0]):\n    out.append(result[j])\n \n  if last % (print_iters*size) == 0:\n    print(last)\n    gc.collect()\n    torch.cuda.empty_cache()\n    torch.cuda.synchronize()\n    if last % (print_iters*size*save_iters) == 0:\n      with open(path_output + name + ".pkl", \'wb\') as f:\n        pickle.dump(out, f)\n      print("Saved to disk")\n \n  if end:\n    break\n  i = last')

6 frames
<decorator-gen-53> in time(self, line, cell, local_ns)

<timed exec> in <module>()

/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py in beam_search(self, input_ids, beam_scorer, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, **model_kwargs)
   1808 
   1809         sequence_outputs = beam_scorer.finalize(
-> 1810             input_ids, beam_scores, next_tokens, next_indices, pad_token_id=pad_token_id, eos_token_id=eos_token_id
   1811         )
   1812 

UnboundLocalError: local variable 'next_tokens' referenced before assignment
```



Comments:
cc @vasudevgupta7 

@OscarGarciaF, i will need more details before i could see your issue. Are you using google/bigbird-roberta-base with EncoderDecoderModel for summarization? It would be great if you can share your code. 

@vasudevgupta7 I was using AutoModelForSeq2SeqLM (this is what you use for summarization right?)

I have now changed to EncoderDecoderModel  but now I face a new error

```
      1 input = tokens[0:1, :].to(device)
----> 2 generated = model_sum.generate(input, decoder_start_token_id = model_sum.config.decoder.pad_token_id, max_length = 512, num_beams = 4, early_stopping = True)

10 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/big_bird/modeling_big_bird.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)
    293 
    294         position_embeddings = self.position_embeddings(position_ids)
--> 295         embeddings += position_embeddings
    296 
    297         embeddings = self.dropout(embeddings)

RuntimeError: output with shape [4, 1, 768] doesn't match the broadcast shape [4, 0, 768]
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Any updates on this? I get the exact same error when running generate on EncoderDecoderModel. 

`RuntimeError: output with shape [1, 1, 768] doesn't match the broadcast shape [1, 0, 768]`

When I remove padding from the input_ids the error goes away but I think this is a bug of some sort.

I also got exact same error (`output with shape...`) when i generate on **custom** BigBird model. I fixed it by reducing `model_max_length` value from 4096 to 4094 and afterwards i can use pipeline for inference without any problem.

```
>>> tokenizer.max_len_single_sentence
4094
>>> tokenizer.model_max_length
4096
>>> tokenizer.model_max_length = 4094
```