Add decoding inputs to generate

# 🚀 Feature request

Add decoding inputs to generate

## Motivation

When generating with encoder-decoder, one may want to insert context for the decoder.
I'm currently working on summarization given that I know some parts of the gt. But other ideas can come to mind.

## Your contribution

```
    @torch.no_grad()
    def generate(
        self,
        input_ids: Optional[torch.LongTensor] = None,
        max_length: Optional[int] = None,
        min_length: Optional[int] = None,
        do_sample: Optional[bool] = None,
        early_stopping: Optional[bool] = None,
        num_beams: Optional[int] = None,
        temperature: Optional[float] = None,
        top_k: Optional[int] = None,
        top_p: Optional[float] = None,
        repetition_penalty: Optional[float] = None,
        bad_words_ids: Optional[Iterable[int]] = None,
        bos_token_id: Optional[int] = None,
        pad_token_id: Optional[int] = None,
        eos_token_id: Optional[int] = None,
        length_penalty: Optional[float] = None,
        no_repeat_ngram_size: Optional[int] = None,
        num_return_sequences: Optional[int] = None,
        attention_mask: Optional[torch.LongTensor] = None,
        decoder_start_token_id: Optional[int] = None,
        decoder_input_ids: Optional[torch.LongTensor] = None,
        decoder_attention_mask: Optional[torch.LongTensor] = None,
        use_cache: Optional[bool] = None,
        **model_specific_kwargs
    ) -> torch.LongTensor:
```
        r""" Generates sequences for models with a LM head. The method currently supports greedy decoding, beam-search decoding, sampling with temperature, sampling with top-k or nucleus sampling.

        Adapted in part from Facebook's XLM beam search code_.

        .. _Facebook's XLM beam search code:
           https://github.com/facebookresearch/XLM/blob/9e6f6814d17be4fe5b15f2e6c43eb2b2d76daeb4/src/model/transformer.py#L529

        Parameters:

            input_ids: (`optional`) `torch.LongTensor` of shape `(batch_size, sequence_length)`
                The sequence used as a prompt for the generation. If `None` the method initializes
                it as an empty `torch.LongTensor` of shape `(1,)`.

            max_length: (`optional`) int
                The max length of the sequence to be generated.  Between `min_length` and infinity. Default to 20.

            min_length: (`optional`) int
                The min length of the sequence to be generated.  Between 0 and infinity. Default to 0.

            do_sample: (`optional`) bool
                If set to `False` greedy decoding is used. Otherwise sampling is used. Defaults to `False` as defined in `configuration_utils.PretrainedConfig`.

            early_stopping: (`optional`) bool
                if set to `True` beam search is stopped when at least `num_beams` sentences finished per batch. Defaults to `False` as defined in `configuration_utils.PretrainedConfig`.

            num_beams: (`optional`) int
                Number of beams for beam search. Must be between 1 and infinity. 1 means no beam search. Default to 1.

            temperature: (`optional`) float
                The value used to module the next token probabilities. Must be strictly positive. Default to 1.0.

            top_k: (`optional`) int
                The number of highest probability vocabulary tokens to keep for top-k-filtering. Between 1 and infinity. Default to 50.

            top_p: (`optional`) float
                The cumulative probability of parameter highest probability vocabulary tokens to keep for nucleus sampling. Must be between 0 and 1. Default to 1.

            repetition_penalty: (`optional`) float
                The parameter for repetition penalty. Between 1.0 and infinity. 1.0 means no penalty. Default to 1.0.

            pad_token_id: (`optional`) int
                Padding token. Default to specicic model pad_token_id or None if it does not exist.

            bos_token_id: (`optional`) int
                BOS token. Defaults to `bos_token_id` as defined in the models config.

            eos_token_id: (`optional`) int
                EOS token. Defaults to `eos_token_id` as defined in the models config.

            length_penalty: (`optional`) float
                Exponential penalty to the length. Default to 1.

            no_repeat_ngram_size: (`optional`) int
                If set to int > 0, all ngrams of size `no_repeat_ngram_size` can only occur once.
            bad_words_ids: (`optional`) list of lists of int
                `bad_words_ids` contains tokens that are not allowed to be generated. In order to get the tokens of the words that should not appear in the generated text, use `tokenizer.encode(bad_word, add_prefix_space=True)`.

            num_return_sequences: (`optional`) int
                The number of independently computed returned sequences for each element in the batch. Default to 1.

            attention_mask (`optional`) obj: `torch.LongTensor` of same shape as `input_ids`
                Mask to avoid performing attention on padding token indices.
                Mask values selected in ``[0, 1]``:
                ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
                Defaults to `None`.

                `What are attention masks? <../glossary.html#attention-mask>`__

            decoder_start_token_id=None: (`optional`) int
                If an encoder-decoder model starts decoding with a different token than BOS.
                Defaults to `None` and is changed to `BOS` later.

            use_cache: (`optional`) bool
                If `use_cache` is True, past key values are used to speed up decoding if applicable to model. Defaults to `True`.

            model_specific_kwargs: (`optional`) dict
                Additional model specific kwargs will be forwarded to the `forward` function of the model.

        Return:

            output: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`
                sequence_length is either equal to max_length or shorter if all batches finished early due to the `eos_token_id`

        Examples::

            tokenizer = AutoTokenizer.from_pretrained('distilgpt2')   # Initialize tokenizer
            model = AutoModelWithLMHead.from_pretrained('distilgpt2')    # Download model and configuration from S3 and cache.
            outputs = model.generate(max_length=40)  # do greedy decoding
            print('Generated: {}'.format(tokenizer.decode(outputs[0], skip_special_tokens=True)))

            tokenizer = AutoTokenizer.from_pretrained('openai-gpt')   # Initialize tokenizer
            model = AutoModelWithLMHead.from_pretrained('openai-gpt')    # Download model and configuration from S3 and cache.
            input_context = 'The dog'
            input_ids = tokenizer.encode(input_context, return_tensors='pt')  # encode input context
            outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, temperature=1.5)  # generate 3 independent sequences using beam search decoding (5 beams) with sampling from initial context 'The dog'
            for i in range(3): #  3 output sequences were generated
                print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True)))

            tokenizer = AutoTokenizer.from_pretrained('distilgpt2')   # Initialize tokenizer
            model = AutoModelWithLMHead.from_pretrained('distilgpt2')    # Download model and configuration from S3 and cache.
            input_context = 'The dog'
            input_ids = tokenizer.encode(input_context, return_tensors='pt')  # encode input context
            outputs = model.generate(input_ids=input_ids, max_length=40, temperature=0.7, num_return_sequences=3)  # 3 generate sequences using by sampling
            for i in range(3): #  3 output sequences were generated
                print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True)))

            tokenizer = AutoTokenizer.from_pretrained('ctrl')   # Initialize tokenizer
            model = AutoModelWithLMHead.from_pretrained('ctrl')    # Download model and configuration from S3 and cache.
            input_context = 'Legal My neighbor is'  # "Legal" is one of the control codes for ctrl
            input_ids = tokenizer.encode(input_context, return_tensors='pt')  # encode input context
            outputs = model.generate(input_ids=input_ids, max_length=50, temperature=0.7, repetition_penalty=1.2)  # generate sequences
            print('Generated: {}'.format(tokenizer.decode(outputs[0], skip_special_tokens=True)))

            tokenizer = AutoTokenizer.from_pretrained('gpt2')   # Initialize tokenizer
            model = AutoModelWithLMHead.from_pretrained('gpt2')    # Download model and configuration from S3 and cache.
            input_context = 'My cute dog'  # "Legal" is one of the control codes for ctrl
            bad_words_ids = [tokenizer.encode(bad_word, add_prefix_space=True) for bad_word in ['idiot', 'stupid', 'shut up']]
            input_ids = tokenizer.encode(input_context, return_tensors='pt')  # encode input context
            outputs = model.generate(input_ids=input_ids, max_length=100, do_sample=True, bad_words_ids=bad_words_ids)  # generate sequences without allowing bad_words to be generated
        """

        # We cannot generate if the model does not have a LM head
        if self.get_output_embeddings() is None:
            raise AttributeError(
                "You tried to generate sequences with a model that does not have a LM Head."
                "Please use another model class (e.g. `OpenAIGPTLMHeadModel`, `XLNetLMHeadModel`, `GPT2LMHeadModel`, `CTRLLMHeadModel`, `T5WithLMHeadModel`, `TransfoXLLMHeadModel`, `XLMWithLMHeadModel`, `BartForConditionalGeneration` )"
            )

        max_length = max_length if max_length is not None else self.config.max_length
        min_length = min_length if min_length is not None else self.config.min_length
        do_sample = do_sample if do_sample is not None else self.config.do_sample
        early_stopping = early_stopping if early_stopping is not None else self.config.early_stopping
        use_cache = use_cache if use_cache is not None else self.config.use_cache
        num_beams = num_beams if num_beams is not None else self.config.num_beams
        temperature = temperature if temperature is not None else self.config.temperature
        top_k = top_k if top_k is not None else self.config.top_k
        top_p = top_p if top_p is not None else self.config.top_p
        repetition_penalty = repetition_penalty if repetition_penalty is not None else self.config.repetition_penalty
        bos_token_id = bos_token_id if bos_token_id is not None else self.config.bos_token_id
        pad_token_id = pad_token_id if pad_token_id is not None else self.config.pad_token_id
        eos_token_id = eos_token_id if eos_token_id is not None else self.config.eos_token_id
        length_penalty = length_penalty if length_penalty is not None else self.config.length_penalty
        no_repeat_ngram_size = (
            no_repeat_ngram_size if no_repeat_ngram_size is not None else self.config.no_repeat_ngram_size
        )
        bad_words_ids = bad_words_ids if bad_words_ids is not None else self.config.bad_words_ids
        num_return_sequences = (
            num_return_sequences if num_return_sequences is not None else self.config.num_return_sequences
        )
        decoder_start_token_id = (
            decoder_start_token_id if decoder_start_token_id is not None else self.config.decoder_start_token_id
        )

        if input_ids is not None:
            batch_size = input_ids.shape[0]  # overriden by the input batch_size
        else:
            batch_size = 1

        assert isinstance(max_length, int) and max_length > 0, "`max_length` should be a strictly positive integer."
        assert isinstance(min_length, int) and min_length >= 0, "`min_length` should be a positive integer."
        assert isinstance(do_sample, bool), "`do_sample` should be a boolean."
        assert isinstance(early_stopping, bool), "`early_stopping` should be a boolean."
        assert isinstance(use_cache, bool), "`use_cache` should be a boolean."
        assert isinstance(num_beams, int) and num_beams > 0, "`num_beams` should be a strictly positive integer."
        assert temperature > 0, "`temperature` should be strictly positive."
        assert isinstance(top_k, int) and top_k >= 0, "`top_k` should be a positive integer."
        assert 0 <= top_p <= 1, "`top_p` should be between 0 and 1."
        assert repetition_penalty >= 1.0, "`repetition_penalty` should be >= 1."
        assert input_ids is not None or (
            isinstance(bos_token_id, int) and bos_token_id >= 0
        ), "If input_ids is not defined, `bos_token_id` should be a positive integer."
        assert pad_token_id is None or (
            isinstance(pad_token_id, int) and (pad_token_id >= 0)
        ), "`pad_token_id` should be a positive integer."
        assert (eos_token_id is None) or (
            isinstance(eos_token_id, int) and (eos_token_id >= 0)
        ), "`eos_token_id` should be a positive integer."
        assert length_penalty > 0, "`length_penalty` should be strictly positive."
        assert (
            isinstance(no_repeat_ngram_size, int) and no_repeat_ngram_size >= 0
        ), "`no_repeat_ngram_size` should be a positive integer."
        assert (
            isinstance(num_return_sequences, int) and num_return_sequences > 0
        ), "`num_return_sequences` should be a strictly positive integer."
        assert (
            bad_words_ids is None or isinstance(bad_words_ids, list) and isinstance(bad_words_ids[0], list)
        ), "`bad_words_ids` is either `None` or a list of lists of tokens that should not be generated"

        if input_ids is None:
            assert isinstance(bos_token_id, int) and bos_token_id >= 0, (
                "you should either supply a context to complete as `input_ids` input "
                "or a `bos_token_id` (integer >= 0) as a first token to start the generation."
            )
            input_ids = torch.full(
                (batch_size, 1), bos_token_id, dtype=torch.long, device=next(self.parameters()).device,
            )
        else:
            assert input_ids.dim() == 2, "Input prompt should be of shape (batch_size, sequence length)."

        # not allow to duplicate outputs when greedy decoding
        if do_sample is False:
            if num_beams == 1:
                # no_beam_search greedy generation conditions
                assert (
                    num_return_sequences == 1
                ), "Greedy decoding will always produce the same output for num_beams == 1 and num_return_sequences > 1. Please set num_return_sequences = 1"

            else:
                # beam_search greedy generation conditions
                assert (
                    num_beams >= num_return_sequences
                ), "Greedy beam search decoding cannot return more sequences than it has beams. Please set num_beams >= num_return_sequences"

        # create attention mask if necessary
        # TODO (PVP): this should later be handled by the forward fn() in each model in the future see PR 3140
        if (attention_mask is None) and (pad_token_id is not None) and (pad_token_id in input_ids):
            attention_mask = input_ids.ne(pad_token_id).long()
        elif attention_mask is None:
            attention_mask = input_ids.new_ones(input_ids.shape)

        # set pad_token_id to eos_token_id if not set. Important that this is done after
        # attention_mask is created
        if pad_token_id is None and eos_token_id is not None:
            logger.warning(
                "Setting `pad_token_id` to {} (first `eos_token_id`) to generate sequence".format(eos_token_id)
            )
            pad_token_id = eos_token_id

        # current position and vocab size
        if hasattr(self.config, "vocab_size"):
            vocab_size = self.config.vocab_size
        elif (
            self.config.is_encoder_decoder
            and hasattr(self.config, "decoder")
            and hasattr(self.config.decoder, "vocab_size")
        ):
            vocab_size = self.config.decoder.vocab_size

        # set effective batch size and effective batch multiplier according to do_sample
        if do_sample:
            effective_batch_size = batch_size * num_return_sequences
            effective_batch_mult = num_return_sequences
        else:
            effective_batch_size = batch_size
            effective_batch_mult = 1

        if self.config.is_encoder_decoder:
            if decoder_start_token_id is None:
                decoder_start_token_id = bos_token_id

            assert (
                decoder_start_token_id is not None
            ), "decoder_start_token_id or bos_token_id has to be defined for encoder-decoder generation"
            assert hasattr(self, "get_encoder"), "{} should have a 'get_encoder' function defined".format(self)
            assert callable(self.get_encoder), "{} should be a method".format(self.get_encoder)

            # get encoder and store encoder outputs
            encoder = self.get_encoder()

            encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask)

        # Expand input ids if num_beams > 1 or num_return_sequences > 1

        if self.config.is_encoder_decoder:
            if decoder_input_ids is not None:
                input_ids = decoder_input_ids
            else:
                # create empty decoder_input_ids
                input_ids =  torch.full(
                (effective_batch_size * num_beams, 1),
                decoder_start_token_id,
                dtype=torch.long,
                device=next(self.parameters()).device,
            )
            cur_len = 1

            assert (
                batch_size == encoder_outputs[0].shape[0]
            ), f"expected encoder_outputs[0] to have 1st dimension bs={batch_size}, got {encoder_outputs[0].shape[0]} "

            # expand batch_idx to assign correct encoder output for expanded input_ids (due to num_beams > 1 and num_return_sequences > 1)
            expanded_batch_idxs = (
                torch.arange(batch_size)
                .view(-1, 1)
                .repeat(1, num_beams * effective_batch_mult)
                .view(-1)
                .to(input_ids.device)
            )
            # expand encoder_outputs
            encoder_outputs = (encoder_outputs[0].index_select(0, expanded_batch_idxs), *encoder_outputs[1:])

        else:
            encoder_outputs = None
            cur_len = input_ids.shape[-1]

        assert (
            cur_len < max_length
        ), f"The context has {cur_len} number of tokens, but `max_length` is only {max_length}. Please make sure that `max_length` is bigger than the number of tokens, by setting either `generate(max_length=...,...)` or `config.max_length = ...`"

        if num_return_sequences > 1 or num_beams > 1:
            input_ids_len = input_ids.shape[-1]
            input_ids = input_ids.unsqueeze(1).expand(batch_size, effective_batch_mult * num_beams, input_ids_len)
            attention_mask = attention_mask.unsqueeze(1).expand(
                batch_size, effective_batch_mult * num_beams, attention_mask.shape[-1]
            )

            input_ids = input_ids.contiguous().view(
                effective_batch_size * num_beams, input_ids_len
            )  # shape: (batch_size * num_return_sequences * num_beams, cur_len)
            attention_mask = attention_mask.contiguous().view(
                effective_batch_size * num_beams, attention_mask.shape[-1]
            )  # shape: (batch_size * num_return_sequences * num_beams, cur_len)

        if num_beams > 1:
            output = self._generate_beam_search(
                input_ids,
                cur_len=cur_len,
                max_length=max_length,
                min_length=min_length,
                do_sample=do_sample,
                early_stopping=early_stopping,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                repetition_penalty=repetition_penalty,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
                batch_size=effective_batch_size,
                num_return_sequences=num_return_sequences,
                length_penalty=length_penalty,
                num_beams=num_beams,
                vocab_size=vocab_size,
                encoder_outputs=encoder_outputs,
                attention_mask=attention_mask,
                use_cache=use_cache,
                decoder_attention_mask=decoder_attention_mask,
                model_specific_kwargs=model_specific_kwargs,
            )
        else:
            output = self._generate_no_beam_search(
                input_ids,
                cur_len=cur_len,
                max_length=max_length,
                min_length=min_length,
                do_sample=do_sample,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                repetition_penalty=repetition_penalty,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
                batch_size=effective_batch_size,
                encoder_outputs=encoder_outputs,
                attention_mask=attention_mask,
                use_cache=use_cache,
                decoder_attention_mask=decoder_attention_mask,
                model_specific_kwargs=model_specific_kwargs,
            )

        return output

    def _generate_no_beam_search(
        self,
        input_ids,
        cur_len,
        max_length,
        min_length,
        do_sample,
        temperature,
        top_k,
        top_p,
        repetition_penalty,
        no_repeat_ngram_size,
        bad_words_ids,
        pad_token_id,
        eos_token_id,
        batch_size,
        encoder_outputs,
        attention_mask,
        use_cache,
        decoder_attention_mask,
        model_specific_kwargs,
    ):
        """ Generate sequences for each example without beam search (num_beams == 1).
            All returned sequence are generated independantly.
        """
        # length of generated sentences / unfinished sentences
        unfinished_sents = input_ids.new(batch_size).fill_(1)
        sent_lengths = input_ids.new(batch_size).fill_(max_length)

        past = (encoder_outputs, None) if encoder_outputs is not None else None
        while cur_len < max_length:
            model_inputs = self.prepare_inputs_for_generation(
                input_ids, past=past, attention_mask=attention_mask, use_cache=use_cache, **model_specific_kwargs
            )

            model_inputs['decoder_attention_mask'] = decoder_attention_mask
            outputs = self(**model_inputs)

            next_token_logits = outputs[0][:, -1, :]

            scores = self.postprocess_next_token_scores(
                scores=next_token_logits,
                input_ids=input_ids,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                cur_len=cur_len,
                min_length=min_length,
                max_length=max_length,
                eos_token_id=eos_token_id,
                repetition_penalty=repetition_penalty,
                batch_size=batch_size,
                num_beams=1,
            )

            # if model has past, then set the past variable to speed up decoding
            if self._use_cache(outputs, use_cache):
                past = outputs[1]

            if do_sample:
                # Temperature (higher temperature => more likely to sample low probability tokens)
                if temperature != 1.0:
                    scores = scores / temperature
                # Top-p/top-k filtering
                next_token_logscores = top_k_top_p_filtering(scores, top_k=top_k, top_p=top_p)
                # Sample
                probs = F.softmax(next_token_logscores, dim=-1)
                next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
            else:
                # Greedy decoding
                next_token = torch.argmax(next_token_logits, dim=-1)

            # update generations and finished sentences
            if eos_token_id is not None:
                # pad finished sentences if eos_token_id exist
                tokens_to_add = next_token * unfinished_sents + (pad_token_id) * (1 - unfinished_sents)
            else:
                tokens_to_add = next_token

            # add token and increase length by one
            input_ids = torch.cat([input_ids, tokens_to_add.unsqueeze(-1)], dim=-1)
            cur_len = cur_len + 1

            if eos_token_id is not None:
                eos_in_sents = tokens_to_add == eos_token_id
                # if sentence is unfinished and the token to add is eos, sent_lengths is filled with current length
                is_sents_unfinished_and_token_to_add_is_eos = unfinished_sents.mul(eos_in_sents.long()).bool()
                sent_lengths.masked_fill_(is_sents_unfinished_and_token_to_add_is_eos, cur_len)
                # unfinished_sents is set to zero if eos in sentence
                unfinished_sents.mul_((~eos_in_sents).long())

            # stop when there is a </s> in each sentence, or if we exceed the maximul length
            if unfinished_sents.max() == 0:
                break

            # extend attention_mask for new generated input if only decoder
            if self.config.is_encoder_decoder is False:
                attention_mask = torch.cat(
                    [attention_mask, attention_mask.new_ones((attention_mask.shape[0], 1))], dim=-1
                )

        return input_ids

    def _generate_beam_search(
        self,
        input_ids,
        cur_len,
        max_length,
        min_length,
        do_sample,
        early_stopping,
        temperature,
        top_k,
        top_p,
        repetition_penalty,
        no_repeat_ngram_size,
        bad_words_ids,
        pad_token_id,
        eos_token_id,
        batch_size,
        num_return_sequences,
        length_penalty,
        num_beams,
        vocab_size,
        encoder_outputs,
        attention_mask,
        use_cache,
        decoder_attention_mask,
        model_specific_kwargs,
    ):
        """ Generate sequences for each example with beam search.
        """

        # generated hypotheses
        generated_hyps = [
            BeamHypotheses(num_beams, max_length, length_penalty, early_stopping=early_stopping)
            for _ in range(batch_size)
        ]

        # scores for each sentence in the beam
        beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=input_ids.device)

        # for greedy decoding it is made sure that only tokens of the first beam are considered to avoid sampling the exact same tokens three times
        if do_sample is False:
            beam_scores[:, 1:] = -1e9
        beam_scores = beam_scores.view(-1)  # shape (batch_size * num_beams,)

        # cache compute states
        past = (encoder_outputs, None) if encoder_outputs is not None else None

        # done sentences
        done = [False for _ in range(batch_size)]

        while cur_len < max_length:
            model_inputs = self.prepare_inputs_for_generation(
                input_ids, past=past, attention_mask=attention_mask, use_cache=use_cache, **model_specific_kwargs
            )
            model_inputs['decoder_attention_mask'] = decoder_attention_mask
            outputs = self(**model_inputs)  # (batch_size * num_beams, cur_len, vocab_size)
            next_token_logits = outputs[0][:, -1, :]  # (batch_size * num_beams, vocab_size)

            # if model has past, then set the past variable to speed up decoding
            if self._use_cache(outputs, use_cache):
                past = outputs[1]
            if self.config.is_encoder_decoder and do_sample is False:
                # TODO (PVP) still a bit hacky here - there might be a better solution
                next_token_logits = self.adjust_logits_during_generation(
                    next_token_logits, cur_len=cur_len, max_length=max_length
                )

            scores = F.log_softmax(next_token_logits, dim=-1)  # (batch_size * num_beams, vocab_size)

            scores = self.postprocess_next_token_scores(
                scores=scores,
                input_ids=input_ids,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                cur_len=cur_len,
                min_length=min_length,
                max_length=max_length,
                eos_token_id=eos_token_id,
                repetition_penalty=repetition_penalty,
                batch_size=batch_size,
                num_beams=num_beams,
            )

            assert scores.shape == (batch_size * num_beams, vocab_size), "Shapes of scores: {} != {}".format(
                scores.shape, (batch_size * num_beams, vocab_size)
            )

            if do_sample:
                _scores = scores + beam_scores[:, None].expand_as(scores)  # (batch_size * num_beams, vocab_size)
                # Temperature
                if temperature != 1.0:
                    _scores = _scores / temperature
                # Top-p/top-k filtering
                _scores = top_k_top_p_filtering(
                    _scores, top_k=top_k, top_p=top_p, min_tokens_to_keep=2
                )  # (batch_size * num_beams, vocab_size)
                # re-organize to group the beam together to sample from all beam_idxs
                _scores = _scores.contiguous().view(
                    batch_size, num_beams * vocab_size
                )  # (batch_size, num_beams * vocab_size)

                # Sample 2 next tokens for each beam (so we have some spare tokens and match output of greedy beam search)
                probs = F.softmax(_scores, dim=-1)
                next_tokens = torch.multinomial(probs, num_samples=2 * num_beams)  # (batch_size, num_beams * 2)
                # Compute next scores
                next_scores = torch.gather(_scores, -1, next_tokens)  # (batch_size, num_beams * 2)
                # sort the sampled vector to make sure that the first num_beams samples are the best
                next_scores, next_scores_indices = torch.sort(next_scores, descending=True, dim=1)
                next_tokens = torch.gather(next_tokens, -1, next_scores_indices)  # (batch_size, num_beams * 2)

            else:
                next_scores = scores + beam_scores[:, None].expand_as(scores)  # (batch_size * num_beams, vocab_size)

                # re-organize to group the beam together (we are keeping top hypothesis accross beams)
                next_scores = next_scores.view(
                    batch_size, num_beams * vocab_size
                )  # (batch_size, num_beams * vocab_size)

                next_scores, next_tokens = torch.topk(next_scores, 2 * num_beams, dim=1, largest=True, sorted=True)

            assert next_scores.size() == next_tokens.size() == (batch_size, 2 * num_beams)

            # next batch beam content
            next_batch_beam = []

            # for each sentence
            for batch_idx in range(batch_size):

                # if we are done with this sentence, add a pad token
                if done[batch_idx]:
                    assert (
                        len(generated_hyps[batch_idx]) >= num_beams
                    ), "Batch can only be done if at least {} beams have been generated".format(num_beams)
                    assert (
                        eos_token_id is not None and pad_token_id is not None
                    ), "generated beams >= num_beams -> eos_token_id and pad_token have to be defined"
                    next_batch_beam.extend([(0, pad_token_id, 0)] * num_beams)  # pad the batch
                    continue

                # next sentence beam content, this will get added to next_batch_beam
                next_sent_beam = []

                # next tokens for this sentence
                for beam_token_rank, (beam_token_id, beam_token_score) in enumerate(
                    zip(next_tokens[batch_idx], next_scores[batch_idx])
                ):
                    # get beam and token IDs
                    beam_id = beam_token_id // vocab_size
                    token_id = beam_token_id % vocab_size

                    effective_beam_id = batch_idx * num_beams + beam_id
                    # add to generated hypotheses if end of sentence
                    if (eos_token_id is not None) and (token_id.item() == eos_token_id):
                        # if beam_token does not belong to top num_beams tokens, it should not be added
                        is_beam_token_worse_than_top_num_beams = beam_token_rank >= num_beams
                        if is_beam_token_worse_than_top_num_beams:
                            continue
                        generated_hyps[batch_idx].add(
                            input_ids[effective_beam_id].clone(), beam_token_score.item(),
                        )
                    else:
                        # add next predicted token since it is not eos_token
                        next_sent_beam.append((beam_token_score, token_id, effective_beam_id))

                    # once the beam for next step is full, don't add more tokens to it.
                    if len(next_sent_beam) == num_beams:
                        break

                # Check if we are done so that we can save a pad step if all(done)
                done[batch_idx] = done[batch_idx] or generated_hyps[batch_idx].is_done(
                    next_scores[batch_idx].max().item(), cur_len
                )

                # update next beam content
                assert len(next_sent_beam) == num_beams, "Beam should always be full"
                next_batch_beam.extend(next_sent_beam)
                assert len(next_batch_beam) == num_beams * (batch_idx + 1), "We should have added num_beams each step"

            # stop when we are done with each sentence
            if all(done):
                break

            # sanity check / prepare next batch
            assert len(next_batch_beam) == batch_size * num_beams
            beam_scores = beam_scores.new([x[0] for x in next_batch_beam])
            beam_tokens = input_ids.new([x[1] for x in next_batch_beam])
            beam_idx = input_ids.new([x[2] for x in next_batch_beam])

            # re-order batch and update current length
            input_ids = input_ids[beam_idx, :]
            input_ids = torch.cat([input_ids, beam_tokens.unsqueeze(1)], dim=-1)
            cur_len = cur_len + 1

            # re-order internal states
            if past is not None:
                past = self._reorder_cache(past, beam_idx)

            # extend attention_mask for new generated input if only decoder
            if self.config.is_encoder_decoder is False:
                attention_mask = torch.cat(
                    [attention_mask, attention_mask.new_ones((attention_mask.shape[0], 1))], dim=-1
                )

        # finalize all open beam hypotheses and add to generated hypotheses
        for batch_idx in range(batch_size):
            if done[batch_idx]:
                continue

            # test that beam scores match previously calculated scores if not eos and batch_idx not done
            if eos_token_id is not None and all(
                (token_id % vocab_size).item() != eos_token_id for token_id in next_tokens[batch_idx]
            ):
                assert torch.all(
                    next_scores[batch_idx, :num_beams] == beam_scores.view(batch_size, num_beams)[batch_idx]
                ), "If batch_idx is not done, final next scores: {} have to equal to accumulated beam_scores: {}".format(
                    next_scores[:, :num_beams][batch_idx], beam_scores.view(batch_size, num_beams)[batch_idx],
                )

            # need to add best num_beams hypotheses to generated hyps
            for beam_id in range(num_beams):
                effective_beam_id = batch_idx * num_beams + beam_id
                final_score = beam_scores[effective_beam_id].item()
                final_tokens = input_ids[effective_beam_id]
                generated_hyps[batch_idx].add(final_tokens, final_score)

        # depending on whether greedy generation is wanted or not define different output_batch_size and output_num_return_sequences_per_batch
        output_batch_size = batch_size if do_sample else batch_size * num_return_sequences
        output_num_return_sequences_per_batch = 1 if do_sample else num_return_sequences

        # select the best hypotheses
        sent_lengths = input_ids.new(output_batch_size)
        best = []

        # retrieve best hypotheses
        for i, hypotheses in enumerate(generated_hyps):
            sorted_hyps = sorted(hypotheses.beams, key=lambda x: x[0])
            for j in range(output_num_return_sequences_per_batch):
                effective_batch_idx = output_num_return_sequences_per_batch * i + j
                best_hyp = sorted_hyps.pop()[1]
                sent_lengths[effective_batch_idx] = len(best_hyp)
                best.append(best_hyp)

        # shorter batches are padded
        if sent_lengths.min().item() != sent_lengths.max().item():
            assert pad_token_id is not None, "`Pad_token_id` has to be defined"
            sent_max_len = min(sent_lengths.max().item() + 1, max_length)
            decoded = input_ids.new(output_batch_size, sent_max_len).fill_(pad_token_id)

            # fill with hypothesis and eos_token_id if necessary
            for i, hypo in enumerate(best):
                decoded[i, : sent_lengths[i]] = hypo
                if sent_lengths[i] < max_length:
                    decoded[i, sent_lengths[i]] = eos_token_id
        else:
            # none of the hypotheses have an eos_token
            assert (len(hypo) == max_length for hypo in best)
            decoded = torch.stack(best).type(torch.long).to(next(self.parameters()).device)

        return decoded
`


problem about geting hidden_states using TFBertModel

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Pycharm
- Python version: 3.6
- PyTorch version (GPU?): None
- Tensorflow version (GPU?): 2.2.0 GPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu
documentation: @sgugger
 -->
 @LysandreJik @jplu
## Information

Model I am using Bert: I want to the outpts of **TFBertModel** class inclue **hidden_states**.  unfortunately， I use two method provided by transformers' document north can achieve this goal.

The first method:  **output_hidden_states=True** is passed to call(). code as follow:
```
class Model(tf.keras.Model):
    def __init__(self,init):
        super(Model,self).__init__()
        conf = transformers.BertConfig.from_json_file("model/chinese_L-12_H-768_A-12/config.json")
        self.bertmodel = transformers.TFBertModel.from_pretrained("bert-base-chinese")

    @tf.function
    def call(self, inputs, training=None, mask=None):
        out_bert = self.bertmodel(inputs,output_hidden_states=True)
        return out_bert

if __name__ == "__main__":
    tokenizer = transformers.BertTokenizer("model/chinese_L-12_H-768_A-12/vocab.txt")
    text_2 = tokenizer.batch_encode_plus(["你买啊，买了就是成都人", "你来啊，来了就是深圳人"], max_length=20, pad_to_max_length=True)
    print(text_2)
    model = Model()
    out = model([tf.convert_to_tensor(text_2["input_ids"]),tf.convert_to_tensor(text_2['attention_mask'])])
    print("out",out)
```
 the print in consol only two tensor :**last_hidden_state** and **pooler_output**

the second method： set **config.output_hidden_states=True**. code as follow:
config.json:
```
{
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 21128,
  "output_hidden_states": true
}
```
I set **"output_hidden_states": true** at last line.
code:
```
class Model(tf.keras.Model):
    def __init__(self,init):
        super(Model,self).__init__()
        conf = transformers.BertConfig.from_json_file("model/chinese_L-12_H-768_A-12/config.json")
        self.bertmodel = transformers.TFBertModel.from_pretrained("bert-base-chinese",config=conf)

    @tf.function
    def call(self, inputs, training=None, mask=None):
        out_bert = self.bertmodel(inputs)
        return out_bert

if __name__ == "__main__":
    tokenizer = transformers.BertTokenizer("model/chinese_L-12_H-768_A-12/vocab.txt")
    text_2 = tokenizer.batch_encode_plus(["你买啊，买了就是成都人", "你来啊，来了就是深圳人"], max_length=20, pad_to_max_length=True)
    print(text_2)
    model = Model()
    out = model([tf.convert_to_tensor(text_2["input_ids"]),tf.convert_to_tensor(text_2['attention_mask'])])
    print("out",out)
```
the print in consol is same as method one.

but if I cancel **@tf.function** in front of call(). the print is what I expect.

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


convert_pytorch_checkpoint_to_tf2.py AttributeError: cls.seq_relationship.weight not found in PyTorch model

Hi all. I'm trying to convert a pre-trained BERT model from PyTorch into TF2 format and facing a problem.

Model being converted: DeepPavlov/rubert-base-cased. I downloaded all files into a local folder.
File convert_pytorch_checkpoint_to_tf2.py from master branch - I downloaded it separately.

The command I'm running:
`python convert_pytorch_checkpoint_to_tf2.py --tf_dump_path ./rubert-base-cased_tf2/ --model_type bert --pytorch_checkpoint_path ./rubert-base-cased-pt/pytorch_model.bin --config_file ./rubert-base-cased-pt/config.json`

And the stacktrace:
```
Traceback (most recent call last):
  File "convert_pytorch_checkpoint_to_tf2.py", line 364, in <module>
    convert_all_pt_checkpoints_to_tf(
  File "convert_pytorch_checkpoint_to_tf2.py", line 298, in convert_all_pt_checkpoints_to_tf
    convert_pt_checkpoint_to_tf(
  File "convert_pytorch_checkpoint_to_tf2.py", line 209, in convert_pt_checkpoint_to_tf
    tf_model = load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path)
  File "/Users/denisstenyushkin/.virtualenvs/tf2_pytorch/lib/python3.8/site-packages/transformers/modeling_tf_pytorch_utils.py", line 92, in load_pytorch_checkpoint_in_tf2_model
    return load_pytorch_weights_in_tf2_model(
  File "/Users/denisstenyushkin/.virtualenvs/tf2_pytorch/lib/python3.8/site-packages/transformers/modeling_tf_pytorch_utils.py", line 166, in load_pytorch_weights_in_tf2_model
    raise AttributeError("{} not found in PyTorch model".format(name))
AttributeError: cls.seq_relationship.weight not found in PyTorch model
```

Versions:
- `transformers` version: 3.0.2
- Platform: macOS-10.15.5-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Create README.md

model card

Related to pretrained model cards


[T5Tokenizer] add prepare_seq2seq_batch method

This PR adds `prepare_seq2seq_batch` method to `T5Tokenizer` as per the proposal in #6080

@sshleifer


XLNet PLM Readme

Add information on XLNet and its PLM objective in the language-modeling README.


Don't see how to use correct padding with QA pipeline

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details

In some cases, when using a **QuestionAnsweringPipeline**, I get an error similar to the following:

    *** ValueError: expected sequence of length 384 at dim 1 (got 260)

I've traced the problem to line 1496 of pipelines.py, and the change to [this commit](https://github.com/huggingface/transformers/commit/896300177bf9f35feac4698370212a80a5ab6138).  The troubled line is:

    fw_args = {k: torch.tensor(v, device=self.device) for (k, v) in fw_args.items()}

Basically, when trying to convert this **v** to a tensor an error is thrown because not every array in **v** has the same length - because the padding strategy has been changed (if I comment out the padding strategy in the call to **squad_convert_examples_to_features** then the default value `"max_length"` takes effect and there is no problem).  I guess this was done as some sort of optimization, but I'm not really sure how to use it.  Every other argument to **squad_convert_examples_to_features** will use a *kwarg*, but this one does not.  Maybe it should use a *kwarg* like everything else, so if you need the padding (or don't want to have to deal with it) you can set the **padding_strategy** as you like?  Or am I missing something?

### Minimal code to reproduce:

    from transformers import QuestionAnsweringPipeline, AutoModelForQuestionAnswering, AutoTokenizer
    model = AutoModelForQuestionAnswering.from_pretrained('twmkn9/distilbert-base-uncased-squad2')
    tokenizer = AutoTokenizer.from_pretrained('twmkn9/distilbert-base-uncased-squad2')

    # I've omitted the context for brevity.  You can, for example, take the **plot** section from [the matrix](https://en.wikipedia.org/wiki/The_Matrix)
    context = """..."""

    pipeline({"question":"what was Neo's job?", "context": context}) # error as described above

<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


🐛 Empty TypeError on BartTokenizerFast.decode(tensor)

## Environment info
`transformers` `3.0.2`

### Who can help
 Summarization: @sshleifer

## To reproduce

```python
import torch
from transformers import BartTokenizerFast

t = BartTokenizerFast.from_pretrained('facebook/bart-large')

x = torch.tensor([0, 34, 45, 23, 54, 65, 765, 2])
t.decode(x)
```

will throw an empty `TypeError` :

```
File "/home/me/.venv/summarization/lib/python3.6/site-packages/tokenizers/implementations/base_tokenizer.py", line 267, in decode
    return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)
TypeError
```

To reproduce : [Colab Notebook](https://colab.research.google.com/drive/1bnP8TvmRrHrMD-7H2MOQQSE8QrhCb7SC?usp=sharing)

## Expected behavior

No error thrown, like with regular `Tokenizer` :

```python
import torch
from transformers import BartTokenizer

t = BartTokenizer.from_pretrained('facebook/bart-large')

x = torch.tensor([0, 34, 45, 23, 54, 65, 765, 2])
t.decode(x)
```

> `<s> has not at who one short</s>`


Is `guid` allowed to be None in `InputExample`?

## Environment
-`transformers` version: 3.0.2
Omitted the rest because they most likely don't affect this issue.

## To reproduce

```py
from transformers.data.processors.utils import SingleSentenceClassificationProcessor

processor = SingleSentenceClassificationProcessor(labels=["lbl1", "lbl2"])
processor.add_examples(texts_or_text_and_labels=["example1", "example2"]) # There's a default ids=None
print(processor[0])
```
prints
```InputExample(guid=None, text_a='example1', text_b=None, label=None)```

If `guid` is allowed to be None, that should be reflected in the type annotation(and documentation) of `InputExample`. If not, then `ids` should not be allowed to be `None`.

### Who can help
@sgugger because it's a documentation issue, @thomwolf because of `git blame`.


Using control codes for finetuning

Hi
I have a use case of style conditioned generation, where I ask the LM to generate a sentence based on the control code I provide. CTRL is pretty suitable for that task.
Can you tell me how to use control codes for fine-tuning as well as inference?  It should be the same as any CLM like GPT2 but I want to specifically know about the style and control code conditioning. How should be the data format and other stuffs


No button for creating new post at the forum.

![image](https://user-images.githubusercontent.com/4702353/88753699-76b01080-d18f-11ea-80c3-905a006ad31a.png)
discuss.huggingface.co


Usage of Pytorch Native AMP in place of apex (Pytorch 1.6) in Trainer

# 🚀 Feature request
It would be nice to remove the Apex dependency for `fp16` training and use the native [Pytorch's AMP methods](https://github.com/pytorch/pytorch/releases) in the `Trainer` method. Pytorch recommends Apex users to switch to it's native implementation, even [Apex does it so](https://github.com/NVIDIA/apex/issues/818). Moreover, it will eliminate the need for users to build apex by themselves.

## Your contribution
I am happy to submit a PR if you think it would be a good addition. Please let me know.


namespace object has no attribute to "enc_only"

# ❓ Questions & Help
when I running the distillation.py then
 File "E:/transformers-master/examples/seq2seq/distillation.py", line 370, in create_module
    elif args.enc_only:
AttributeError: 'Namespace' object has no attribute 'enc_only'
 how can I deal  with this problem??  thx a lot.

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


🌟 BigBird

# 🌟 New model addition

## Model description

Paper : https://arxiv.org/pdf/2007.14062.pdf

Abstract :
> Transformers-based models, such as BERT, have been one of the most successful deep learning
models for NLP. Unfortunately, one of their core limitations is the quadratic dependency
(mainly in terms of memory) on the sequence length due to their full attention mechanism.
To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this
quadratic dependency to linear. We show that BigBird is a universal approximator of
sequence functions and is Turing complete, thereby preserving these properties of the
quadratic, full attention model. Along the way, our theoretical analysis reveals some of the
benefits of having O(1) global tokens (such as CLS), that attend to the entire sequence
as part of the sparse attention mechanism. The proposed sparse attention can handle
sequences of length up to 8x of what was previously possible using similar hardware. As
a consequence of the capability to handle longer context, BigBird drastically improves
performance on various NLP tasks such as question answering and summarization. We also
propose novel applications to genomics data.

## Open source status

* [ ] the model implementation is available: *No*
* [ ] the model weights are available: *No*
* [ ] who are the authors: *?*


Is there any way that I can use the HuggingFace Transformers as Pyro models?

Hello,

`Pyro` is a Python module that allows users to convert a given (frequentist) neural network into a Bayesian neural network.
I can convert a HuggingFace Transformer into a Pyro model like below:
```python

import torch
from torch import distributions
from transformers import RobertaTokenizer, RobertaForMultipleChoice
import pyro
import pyro.infer
import pyro.optim
import pyro.distributions as dist
import pyro.nn.module as module
from torch import nn
from pyro.infer import SVI

# get the pre-trained HuggingFace RobertaForMultipleChoice and resize the token embeddings after adding the special token
model_RobertaForMultipleChoice = RobertaForMultipleChoice.from_pretrained('roberta-large', output_hidden_states = True)

module.to_pyro_module_(model_RobertaForMultipleChoice)

# Now we can attempt to be fully Bayesian:
for m in model_RobertaForMultipleChoice.modules():
      for name, value in list(m.named_parameters(recurse=False)):
                setattr(m, name, module.PyroSample(prior=dist.Normal(0, 1)
                                              .expand(value.shape)
                                              .to_event(value.dim())))

# define parameters for training
guide_delta = guides.AutoDelta(model_RobertaForMultipleChoice)
```

But when I try to compute the mc_loss from this Bayesian Transformer, Python generates an error:

```python
mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels = mc_labels)[0]

Traceback (most recent call last):
  File "STAT946_final_project_code_v4.py", line 625, in <module>
    success_rate_list_diag_normal = main_function_diag_normal('/home/ec2-user/test.txt', 'test_ans_num.txt', num_iter, log_interval)
  File "STAT946_final_project_code_v4.py", line 415, in main_function_diag_normal
    best_model_RobertaForMultipleChoice_diag_normal = train_loop(model_RobertaForMultipleChoice, tokenizer, optimizer_1, scheduler_1, log_interval, svi_diag_normal, guide_diag_normal, best_model_RobertaForMultipleChoice_diag_normal)
  File "STAT946_final_project_code_v4.py", line 342, in train_loop
    optimizer, scheduler, log_interval, svi, guide, epoch)
  File "STAT946_final_project_code_v4.py", line 237, in train_mc_head
    mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels = mc_labels)[0]
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 441, in forward
    output_hidden_states=output_hidden_states,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 732, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 228, in get_extended_attention_mask
    extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 159, in dtype
    first_tuple = next(gen)
StopIteration
```

Is there any way that I can compute the mc_loss in the regular way after converting HuggingFace Transformer into a Bayesian Transformer?

Thank you.,


Use FutureWarning to deprecate

As discussed, `DeprecationWarning` -> `FutureWarning`


Doc tokenizer

Improve the documentation of tokenizers, following what was done for the models last week, mainly:
- make sure all docstrings of public functions are properly formatted for sphinx
- make sure all args are properly documented
- add or fix type hints wherever necessary

The methods/classes that are not in the main `__init__` are all in the page `internal/tokenization_utils.html`. I had added `SpecialTokensMixin` in the __init__ of transformers a while ago to easily document it, but it can be removed from there now if we want.

I rewrote a few dosctrings here and there so pinging @n1t0 and @mfuntowicz to make sure I didn't write anything bad.

[Preview](https://65719-155220641-gh.circle-artifacts.com/0/docs/_build/html/main_classes/tokenizer.html) of the tokenization page.
[Preview](https://65719-155220641-gh.circle-artifacts.com/0/docs/_build/html/internal/tokenization_utils.html) of the tokenization utils page.


StopIteration error in RobertaForMultipleChoice

Hello,
I am trying to execute the line below for my `RobertaForMultipleChoice` model:
```python
# retrieve the resulting mc_loss
mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels =  mc_labels)[0]
```
but this generates the following error:

```python
Traceback (most recent call last):
  File "STAT946_final_project_code_v4.py", line 623, in <module>
    success_rate_list_diag_normal = main_function_diag_normal('/home/ec2-user/test.txt', 'test_ans_num.txt', num_iter, log_interval)
  File "STAT946_final_project_code_v4.py", line 414, in main_function_diag_normal
    best_model_RobertaForMultipleChoice_diag_normal = train_loop(model_RobertaForMultipleChoice, tokenizer, optimizer_1, scheduler_1, log_interval, svi_diag_normal, guide_diag_normal, best_model_RobertaForMultipleChoice_diag_normal)
  File "STAT946_final_project_code_v4.py", line 341, in train_loop
    optimizer, scheduler, log_interval, svi, guide, epoch)
  File "STAT946_final_project_code_v4.py", line 236, in train_mc_head
    mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels =  mc_labels)[0]
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 441, in forward
    output_hidden_states=output_hidden_states,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 732, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 228, in get_extended_attention_mask
    extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 159, in dtype
    first_tuple = next(gen)
StopIteration
```

How can I get around this type of error? Thank you,


allenai/longformer-large-4096 unavailable

For some reason, I'm unable to download allenai/longformer-large-4096. Everything was working an hour ago, but all of a sudden I get the error included below. It's still listed on https://huggingface.co/models?search=allenai%2Flongformer-large-4096. I'm not sure what's up. Any ideas?
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 242, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ptcc/run_glue.py", line 246, in <module>
    main()
  File "/ptcc/run_glue.py", line 123, in main
    cache_dir=model_args.cache_dir,
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py", line 203, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 251, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'allenai/longformer-large-4096'. Make sure that:

- 'allenai/longformer-large-4096' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'allenai/longformer-large-4096' is the correct path to a directory containing a config.json file

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 242, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ptcc/run_glue.py", line 246, in <module>
    main()
  File "/ptcc/run_glue.py", line 123, in main
    cache_dir=model_args.cache_dir,
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py", line 203, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 251, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'allenai/longformer-large-4096'. Make sure that:

- 'allenai/longformer-large-4096' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'allenai/longformer-large-4096' is the correct path to a directory containing a config.json file
```


Where do the Masked Language Model perform mask on the input data

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

I am trying to pre-train a bert from scratch with my own word set using only the Masked Language Model.
I have a trouble finding where exactly the code masks 15% of the words and replaces it with 80% mask, 10% random, and 10% original.

I noticed that the input "labels" kind of refers to the places where words are masked. Does it mean that when I preprocess the data, I need to masked it myself and then indicates the position of masks in the "labels" input? If so, is "labels" the only input that would be affect? Is there any other input variables, such as the "masked_lm_positions" and "masked_lm_ids" in google-bert that I need to take care of?

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Weird Behavior on XLNetTokenizer after new tokens added

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: ubuntu
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@mfuntowicz

## Information
I use XLNetTokenizer (pretrained) and added a new token in it. After that, the output from ```tokenizer.tokenize``` looks weird.
(I am not sure if the problem comes from ```transformers``` or ```tokenizers```, but I post here anyway.)

## To reproduce
```
from transformers import XLNetTokenizer
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

test = 'This is so awesome!!@username!'
out = tokenizer.tokenize(test)

# without new tokens added, @username is broken down as expected
out
>> ['▁This', '▁is', '▁so', '▁awesome', '!!', '@', 'user', 'name', '!']
```

```
from tokenizers import AddedToken

# introduce new tokens (with white-space on right only)
new_tokens = AddedToken(
    '@username', single_word = True,
    lstrip = False, rstrip = True
)
tokenizer.add_tokens(new_tokens)

out = tokenizer.tokenize(test)

# weird result about the white-space around new tokens
out
>> ['▁This', '▁is', '▁so', '▁awesome', '!!', '@username', '▁', '!']
```

In here, there are two places weird:
1. new tokens "@username" has a white-space on the left, so ""!!@username" should not be broken down into "!!", "@username" (I think it should be broken down into "!!", "@", "user", "name", "!")
2. I am a bit confused on why there is a white-space produced after "@username" token. (i.e. '@username', '▁', '!')

And oddly, when I encode and decode the sentence back, the white-space token after "@username" is not translated to actual whitespace. (Also note there is a white space added before "@username" which mean the new token is correctly identified to have a white-space on the left):
```
enc = tokenizer.encode(test, add_special_tokens = False)
dec = tokenizer.decode(enc)

# in encoding stage, the 2nd last token is whitespace
enc
>> [122, 27, 102, 8729, 7675, 32000, 17, 136]
# in decoding stage, the white-space disappear

dec
>> This is so awesome!! @username!
```


Recursive error calling generate in forward

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 3.0.2
Python: 3.7.6

## Question
Here is the training loop:
```python
def sd_data_collator(dataset_samples_list):
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
    tokenizer.pad_token = tokenizer.eos_token

    encoded_results = tokenizer(dataset_samples_list, padding=True, truncation=True, return_tensors='pt', return_attention_mask=True)

    batch = {}
    batch['input_ids'] = torch.stack([result for result in encoded_results['input_ids']])
    batch['past'] = None
    batch['attention_mask'] = torch.stack([result for result in encoded_results['attention_mask']])
    batch['position_ids'] = None
    batch['head_mask'] = None
    batch['inputs_embeds'] = None
    batch['labels'] = None
    batch['use_cache'] = True
    return batch

sd_dataset = SDAbstractsDataset('/path/to/sd_samples_64.csv')

training_args = TrainingArguments(
    output_dir='/path/to/finetuned_gpt2',
    do_train=True,
    per_device_train_batch_size=4,
    learning_rate=1e-3,
    num_train_epochs=1
)

model = GPT2FinetunedWithNgrams.from_pretrained('gpt2')

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=sd_dataset,
    data_collator = sd_data_collator
)

trainer.train()
```
Here's the model class and its `forward` method:
```python
class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
        self.tokenizer.pad_token = self.tokenizer.eos_token

    def load_ngrams_model(self, ngrams_model_path):
        self.ngrams_model = NGrams(ngrams_model_path)

    def forward(
            self,
            input_ids=None,
            past=None,
            attention_mask=None,
            token_type_ids=None,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
            labels=None,
            use_cache=True,
    ):

        output = self.generate(input_ids=input_ids, max_length=474)
        decoded_output = self.tokenizer.decode(output[0], skip_special_tokens=True)
```
Here's the whole error. It's really lengthy and I cut out the repetitions:
```python
Some weights of GPT2FinetunedWithNgrams were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Iteration:   0%|          | 0/16 [00:00<?, ?it/s]Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
.
.
.
      File "/path/to/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 480, in generate
    model_specific_kwargs=model_specific_kwargs,
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 520, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/path/to/ric-2020/text_gen_w_transformers/finetune_gpt2.py", line 33, in forward

  File "/path/to/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 350, in generate
    "Setting `pad_token_id` to {} (first `eos_token_id`) to generate sequence".format(eos_token_id)
.
.
.
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1390, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1514, in _log
    self.handle(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1594, in callHandlers
    lastResort.handle(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 894, in handle
    self.emit(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 1025, in emit
    msg = self.format(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 869, in format
    return fmt.format(record)
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 608, in format
    record.message = record.getMessage()
  File "/path/to/anaconda3/lib/python3.7/logging/__init__.py", line 360, in getMessage
    def getMessage(self):
RecursionError: maximum recursion depth exceeded while calling a Python object
```
My guess is the `self.generate()` being called within the model produces the recursion problem. I found this problematic because the `generate` method has some awesome functionality for beam search, greedy search, top-k, etc. To overcome this, I added a flag to `generate` called `is_finetuning_current_model`:
```python
@torch.no_grad()
    def generate(
        self,
        input_ids: Optional[torch.LongTensor] = None,
        max_length: Optional[int] = None,
        min_length: Optional[int] = None,
        do_sample: Optional[bool] = None,
        early_stopping: Optional[bool] = None,
        num_beams: Optional[int] = None,
        temperature: Optional[float] = None,
        top_k: Optional[int] = None,
        top_p: Optional[float] = None,
        repetition_penalty: Optional[float] = None,
        bad_words_ids: Optional[Iterable[int]] = None,
        bos_token_id: Optional[int] = None,
        pad_token_id: Optional[int] = None,
        eos_token_id: Optional[int] = None,
        length_penalty: Optional[float] = None,
        no_repeat_ngram_size: Optional[int] = None,
        num_return_sequences: Optional[int] = None,
        attention_mask: Optional[torch.LongTensor] = None,
        decoder_start_token_id: Optional[int] = None,
        use_cache: Optional[bool] = None,
        is_finetuning_current_model: Optional[bool] = None,
        **model_specific_kwargs
    ) -> torch.LongTensor:
```
propagated this down to the `num_beams` check:
```python
        if num_beams > 1:
            output = self._generate_beam_search(
                input_ids,
                cur_len=cur_len,
                max_length=max_length,
                min_length=min_length,
                do_sample=do_sample,
                early_stopping=early_stopping,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                repetition_penalty=repetition_penalty,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
                batch_size=effective_batch_size,
                num_return_sequences=num_return_sequences,
                length_penalty=length_penalty,
                num_beams=num_beams,
                vocab_size=vocab_size,
                encoder_outputs=encoder_outputs,
                attention_mask=attention_mask,
                use_cache=use_cache,
                is_finetuning_current_model=is_finetuning_current_model,
                model_specific_kwargs=model_specific_kwargs
            )
        else:
            output = self._generate_no_beam_search(
                input_ids,
                cur_len=cur_len,
                max_length=max_length,
                min_length=min_length,
                do_sample=do_sample,
                temperature=temperature,
                top_k=top_k,
                top_p=top_p,
                repetition_penalty=repetition_penalty,
                no_repeat_ngram_size=no_repeat_ngram_size,
                bad_words_ids=bad_words_ids,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
                batch_size=effective_batch_size,
                encoder_outputs=encoder_outputs,
                attention_mask=attention_mask,
                use_cache=use_cache,
                is_finetuning_current_model=is_finetuning_current_model,
                model_specific_kwargs=model_specific_kwargs
            )
```
updated `_generate_no_beam_search` and `_generate_beam_search` with the following:
```python
            if is_finetuning_current_model:
                outputs = self.generate_text_while_finetuning(**model_inputs)
            else:
                outputs = self(**model_inputs)
```
For my model class, I just added the `generate_text_while_finetuning` method and set the `is_finetuning_current_model`:
```python
class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
        self.tokenizer.pad_token = self.tokenizer.eos_token

    def load_ngrams_model(self, ngrams_model_path):
        self.ngrams_model = NGrams(ngrams_model_path)

    def generate_text_while_finetuning(self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        use_cache=None,
        output_attentions=None,
        output_hidden_states=None,):

        transformer_outputs = self.transformer(
            input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
        )
        hidden_states = transformer_outputs[0]

        lm_logits = self.lm_head(hidden_states)

        outputs = (lm_logits,) + transformer_outputs[1:]
        return outputs  # (loss), lm_logits, presents, (all hidden_states), (attentions)

    def forward(
            self,
            input_ids=None,
            past=None,
            attention_mask=None,
            token_type_ids=None,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
            labels=None,
            use_cache=True,
    ):

        output = self.generate(input_ids=input_ids, max_length=474, is_finetuning_current_model=True)
        decoded_output = self.tokenizer.decode(output[0], skip_special_tokens=True)
```
This seems to resolve the recursive error and produces an expected `decoded_output` for me. My usecase is using GPT2 with a different loss function to finetune it on a particular domain corpus. I imagine other people would be doing something similar for GPT2 and other models, so I tested this approach just using `GPT2LMHeadModel` and got the same expected results.

My question is do contributors think I should open up a bug report for this?


Fix zero-shot pipeline single seq output shape

Fixes zero shot pipelines bug that returns sequence as a list rather than a str when a single sequence is passed as a list.


rename prepare_translation_batch -> prepare_seq2seq_batch

cc @patil-suraj

Starts work on #6080 , which suggests that all seq2seq tokenizers expose a `prepare_seq2seq_batch` method.

TODO:
- add common test enforcing API consistency.


Fix deebert tests


Use HFArgParser instead of Fire


[Fix] position_ids tests again

Yesterday, in my haste, I added authorized_missing_keys to `BertModel`, rather than `BertPretrainedModel`. Since position_ids are allowed to be missing for all Bert variants, we want the latter, not the former.

I also improved the traceback for the failing test.


[fix] add bart to LM_MAPPING

This fixes 2/5 failing slow tests in #6094


Fix #6096: MBartTokenizer's mask token

add 3 regression tests.


Logs should not be hidden behind a logger.info

Currently the logs that are printed when the global step is a multiple of the `logging_steps` are printed using a `logger.info`. In the case where a user did not set its logging level to INFO, these logs are not shown, even if the user has set a `logging_steps > 0`. This PR fixes that by putting back a print statement.

Closes https://github.com/huggingface/transformers/issues/5901


mBART: incorrect <mask> token id

# 🐛 Bug

## Information

Model I am using: mBART

## To reproduce

```
from transformers import MBartTokenizer
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25')
print(tokenizer.convert_tokens_to_ids(['<mask>', 'ar_AR']))
```
The output for the above code is `[250001, 250001]`  - two different special tokens are mapped to the same id.

## Expected behavior

As far as I can tell, `<mask>` token should be mapped to id 250026.

I've checked [fairseq implementation](https://github.com/pytorch/fairseq/blob/master/fairseq/tasks/multilingual_denoising.py) and it seems that `<mask>` token is added after all the language codes, so it should be the last token in the vocab.

Currently, when I try to use mBART to denoise text with `<mask>` tokens, it mostly just ignores them, but if I replace mask ids with 250026, the model actually generates new text in place of `<mask>` tokens:

```
from transformers import MBartTokenizer, BartForConditionalGeneration

tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25')
model = BartForConditionalGeneration.from_pretrained('facebook/mbart-large-cc25')

text = 'I highly recommend <mask> - it is one of the best <mask> ever read!'
inputs = tokenizer.prepare_translation_batch([text], src_lang='en_XX')

outputs = model.generate(inputs['input_ids'], decoder_start_token_id=tokenizer.lang_code_to_id['en_XX'],
                         num_beams=5)
print(tokenizer.batch_decode(outputs)[0])
```

The output is:
```
en_XX<s> highly recommend - it is one of the best ever read!
```
Replacing mask ids:

```
where = (inputs['input_ids'] == 250001)
inputs['input_ids'][where] = 250026

outputs = model.generate(inputs['input_ids'], decoder_start_token_id=tokenizer.lang_code_to_id['en_XX'],
                         num_beams=5)
print(tokenizer.batch_decode(outputs)[0])
```

The output is:
```
en_XX<s> highly recommend this book - it is one of the best books I have ever read!
```

(In both cases, the model also skips the first input token when generating output, as discussed in #5755.)

I've also noticed that fairseq is using [language code tokens](https://github.com/pytorch/fairseq/blob/108bb2560b1ec01524ba723bc7c69186875afa0a/fairseq/tasks/multilingual_denoising.py#L62) of the form `[en_XX]` rather than just `en_XX`, which can lead to different tokenization if words like  `en_XX` appear in the text, but that's a rather contrived case.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2

@sshleifer


Add BERTweet and PhoBERT

Related to pretrained model cards

I'd like to add BERTweet and PhoBERT to the huggingface's transformers library.
Users can now use these models directly from transformers. E.g:

    tokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base")
    bertweet = BertweetModel.from_pretrained("vinai/bertweet-base")


5 Slow test failures

traceback: [here](https://github.com/huggingface/transformers/runs/916838508?check_suite_focus=true)

```bash
=========================== short test summary info ============================
FAILED tests/test_modeling_auto.py::AutoModelTest::test_model_for_pretraining_from_pretrained
FAILED tests/test_modeling_auto.py::AutoModelTest::test_model_from_pretrained
FAILED tests/test_modeling_bart.py::BartModelIntegrationTests::test_bart_base_mask_filling
FAILED tests/test_modeling_bart.py::BartModelIntegrationTests::test_bart_large_mask_filling
FAILED tests/test_modeling_common.py::ModelUtilsTest::test_model_from_pretrained
==== 5 failed, 1423 passed, 489 skipped, 384 warnings in 1609.33s (0:26:49) ====
```

I'm investigating


Fix #6092

`BatchEncoding` are not instances of dictionaries so we need the separate test.


i dont know what Tranier`s Dataset is.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
![image](https://user-images.githubusercontent.com/32102558/88669501-8ccdba80-d116-11ea-8b29-8dda1a61ce42.png)
![image](https://user-images.githubusercontent.com/32102558/88669549-9bb46d00-d116-11ea-840c-ff661b0a2b0d.png)
 i thought its my customer dataset goes wrong, i dont konw what dataset it should return. the Trainer receive what dataset.
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Fix local_files_only for TF

The `local_files_only` was not working for TF. The flag is to be used as such:

```py
from transformers import TFBertModel
model = TFBertModel.from_pretrained('bert-base-cased', local_files_only=True)
```
Setting it to `True` for any TF model would result in the following error:
```
Traceback (most recent call last):
  File "/Users/jik/Library/Application Support/JetBrains/PyCharm2020.1/scratches/scratch_1.py", line 7, in <module>
    model = TFBertModel.from_pretrained('bert-base-cased', local_files_only=True)
  File "/Users/jik/Workspaces/python/transformers/src/transformers/modeling_tf_utils.py", line 578, in from_pretrained
    local_files_only=local_files_only,
  File "/Users/jik/Workspaces/python/transformers/src/transformers/file_utils.py", line 663, in cached_path
    local_files_only=local_files_only,
  File "/Users/jik/Workspaces/python/transformers/src/transformers/file_utils.py", line 801, in get_from_cache
    "Cannot find the requested files in the cached path and outgoing traffic has been"
ValueError: Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.
```

This is because the `file_utils.get_from_cache` method obtains the filename from the `url_to_filename` method:

https://github.com/huggingface/transformers/blob/1246b20f6d81bcd949078d26cf5ab3d0f3acccc6/src/transformers/file_utils.py#L777

The issue is that this method adds `.h5` to TF models:

https://github.com/huggingface/transformers/blob/1246b20f6d81bcd949078d26cf5ab3d0f3acccc6/src/transformers/file_utils.py#L585-L586

This works when the file is already built using `{filename}.{etag}`, resulting in `{filename}.{etag}.h5`. However, since the etag is `None` with `local_files_only=True`, this results in `{filename}.h5`.

The method tries to find the saved files using the filename followed by `.*`:

https://github.com/huggingface/transformers/blob/1246b20f6d81bcd949078d26cf5ab3d0f3acccc6/src/transformers/file_utils.py#L788-L792

This doesn't work since it's looking for `{filename}.h5.*` which doesn't exist. It should instead be looking for `{filename}.*`, which is what it's doing now.

Fix https://github.com/huggingface/transformers/issues/5016


customize special tokens

# ❓ Questions & Help

## Details

I am trying to add more special tokens in order to train new language models without modifying the model architecture but simply modifying the input data. For example, adding more separators to involving more complex structure information.

Is there a simple, clear, consistent way to add/customize special tokens.
1. The function `add_special_tokens` can only add token string to token attribute specified in `SPECIAL_TOKENS_ATTRIBUTES`
2. I tried to extend this list and `add_special_tokens` works, but calling
`tokenizer.get_special_tokens_mask` report that _XXX does not exist, where XXX is the new customized special token attribute

finally I found a solution: modifying `tokenizer._additional_special_tokens.extend` directly makes `tokenizer.get_special_tokens_mask` work. But I don't know how could this work and if there is any other side effect.

Properties like `special_tokens_map` is also confusing, it seems like a @property implementation, which is not modifiable. And I don't know where this could be used to have what effect.

I cost much time to read the source code but still lack a overview about the special token architecture in tokenizers.


Added capability to quantize a model while exporting through ONNX.

Add quantization support as part of our collaboration with ONNX team.

Quantization is available through the new method `quantize` which takes the path to the initial ONNX model and operate the conversion.

From a CLI point of view, adding `--quantize` allows to seamlessly export the quantized model.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


Finetuning German BERT for QA on biomedical domain

Hello there and thank you very much for this wonderful work. I am relatively new to this field, so please bear with my amateur question. I want to perform question-answering on a German Biomedical text. From what I understand up to now, I need to fine-tune German BERT on biomedical QA datasets. Is there any script/pipeline that I should be using for this?
I have also posted in StackOverflow and the huggingface forum, but to no avail as of now.
Thank you very much in advance.


fixed typos, added example question.

Related to pretrained model cards

Added a new example question for the widget. Fixed some typos in the description.


Replace mecab-python3 with fugashi for Japanese tokenization

This replaces mecab-python3 with fugashi for Japanese tokenization. I am
the maintainer of both projects.

Both projects are MeCab wrappers, so the underlying C++ code is the
same. fugashi is the newer wrapper and doesn't use SWIG, so for basic
use of the MeCab API it's easier to use.

This code insures the use of a version of ipadic installed via pip,
which should make versioning and tracking down issues easier.

fugashi has wheels for Windows, OSX, and Linux, which will help with
issues with installing old versions of mecab-python3 on Windows.
Compared to mecab-python3, because fugashi doesn't use SWIG, it doesn't
require a C++ runtime to be installed on Windows.

In adding this change I removed some code dealing with `cursor`,
`token_start`, and `token_end` variables. These variables didn't seem to
be used for anything, it is unclear to me why they were there.

I ran the tests and they passed. For reference, since I had trouble figuring it out, this is needed to run the tests:

    RUN_CUSTOM_TOKENIZERS=yes RUN_SLOW=1 pytest -rs tests/test_tokenization_bert_japanese.py

This is a followup to #5375 .

It's not in this PR, but because installing MeCab separately is not required, it might be a good idea to have the docs changed to point directly to fugashi instead of MeCab.