noamaanMulla-03/transformers-issues huggingface dataset:
Where is the casual mask when using BertLMHeadModel and set config.is_decoder = True?
Author: Doragd
Pull request: no

I hope to use BERT for the task of causal language modeling.  

`BertLMHeadModel ` seems to meet my needs, but I did not find any code snippets about the causal mask, even if I set the `config.is_decoder=True`.

I only find the following related code in https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L968.

however, I do not have any values to pass into the argument `encoder_hidden_states` when doing causal language modeling.
So maybe the causal mask does not work?

```
if self.config.is_decoder and encoder_hidden_states is not None:
    encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.size()
    encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
    if encoder_attention_mask is None:
        encoder_attention_mask = torch.ones(encoder_hidden_shape, device=device)
    encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
else:
    encoder_extended_attention_mask = None
```

Comments:
Hi @Doragd, BERT is an encoder model, and is therefore ill-suited to the causal language modeling task. Is there a reason you would like to use that model specifically for causal language modeling?

Hi, @LysandreJik  I just apply causal language modeling as an auxiliary task to lead stable training of our model.  I should have implemented this process myself, but I found this class `BertLMHeadModel`. However, I did not find any code snippet to implement causal mask. I would like to know that if is_decoder=True is set in BERT, can causal language modeling be achieved correctly?

cc @patrickvonplaten 

Setting `is_decoder=True` automatically creates a causal mask in those lines of code: https://github.com/huggingface/transformers/blob/7fae5350528474c29b664ebb4df5bbc8104b48ec/src/transformers/modeling_utils.py#L266
T5 Generate from Encoder Output
Author: kevin3567
Pull request: no

Hi,

I am working with the T5ConditionalGeneration for sequence generation. I am wondering that for autoregressively decoding outputs with the _generate()_ function, is there a way to decode a sequence from by feeding an intermediate layer input (such as _encoder_outputs_ or _input_embeds_) as opposed to the _input_ids_. I have noticed that for _forward()_ function, this feature is supported, where we can input _encoder_outputs_ or _input_embeds_ instead of the _input_ids_. However, I have not yet figured out a way to decode through the following:

> \# model is t5 conditional generation
> out_sequence = model.generate(encoder_outputs=encoder_outputs, num_beams=args.num_beams, ...)

If this is feature not directly available, are there any alternative approaches recommended to allow sequence decoding directly from an encoder output?

Comments:
from [https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py](url)
row 369-374:
`
def prepare_inputs_for_generation(self, input_ids: torch.LongTensor, **kwargs) -&gt; Dict[str, Any]:       
          return {"input_ids": input_ids}
`
Maybe haven't a new way to add own args other than "input_ids".


This might help: https://github.com/huggingface/transformers/pull/10599

It should be possible to directly pass `encoder_outptus`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add support for RemBERT
Author: stefan-it
Pull request: no
Labels: name: New model

name: Feature request

# 🌟 New model addition

## Model description

Hi,

I just found this really interesting upcoming ICLR 2021 paper: "Rethinking Embedding Coupling in Pre-trained Language Models":

>  We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to significantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By reallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on standard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that allocating additional capacity to the output embedding provides benefits to the model that persist through the fine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger output embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage Transformer representations to be more general and more transferable to other tasks and languages. Harnessing these findings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the number of parameters at the fine-tuning stage. 

Paper can be found [here](https://openreview.net/forum?id=xpFFI_NtgpW).

Thus, the authors propose a new *Rebalanced mBERT (**RemBERT**) model* that outperforms XLM-R. An integration into Transformers would be awesome!

I would really like to help with the integration into Transformers, as soon as the model is out!

## Open source status

* [ ] the model implementation is available: authors plan to release model implementation
* [ ] the model weights are available: authors plan to release model checkpoint
* [ ] who are the authors: @hwchung27, @Iwontbecreative, Henry Tsai, Melvin Johnson and @sebastianruder


Comments:
Decided it would be easier for us to take care of this since we plan to directly release the model checkpoint in huggingface.

Started working on it over the week-end, will share PR once it is more polished. 

This is great news @Iwontbecreative! Let us know if you need help.
Can't load config for [community model]
Author: abedkhooli
Pull request: no

Although I can use a fine-tuned GPT2 model from code, the model page complains about the config file (which is already uploaded). 
at https://huggingface.co/akhooli/gpt2-small-arabic-poetry (for a prompt), I get: 
```
Can't load config for 'akhooli/gpt2-small-arabic-poetry'. Make sure that: - 'akhooli/gpt2-small-arabic-poetry' is a correct model identifier listed on 'https://huggingface.co/models' - or 'akhooli/gpt2-small-arabic-poetry' is the correct path to a directory containing a config.json file 
```

Comments:
Currently working on a fix. Will update here

Should be fixed now

Hey I'm facing the same issue with two models I uploaded today
https://huggingface.co/rohanrajpal/bert-base-en-hi-codemix-cased?text=I+like+you.+I+love+you
https://huggingface.co/rohanrajpal/bert-base-en-es-codemix-cased?text=I+like+you.+I+love+you

Here are the config files
https://s3.amazonaws.com/models.huggingface.co/bert/rohanrajpal/bert-base-en-es-codemix-cased/config.json
https://s3.amazonaws.com/models.huggingface.co/bert/rohanrajpal/bert-base-en-hi-codemix-cased/config.json

pinging @mfuntowicz on this

I am seeing the same issue with a new model I just uploaded (akhooli/xlm-r-large-arabic-sent). It works if called in code through HF pipeline.

Seeing the same issue, pinging @mfuntowicz, @julien-c

https://huggingface.co/donal/Pro_Berta?text=The+goal+of+life+is+%3Cmask%3E.

All the files seem to be in the right place.

Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/merges.txt
Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/special_tokens_map.json
Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/training_args.bin
Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/pytorch_model.bin
Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/config.json
Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/tokenizer_config.json
Your file now lives at:                                                         
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/vocab.json



You probably are already aware of this, but inference worked for a bit and broke again (Can't load config for [model]).

I had to revert the change I did because it was breaking one workflow we have on api-inference side. I'm working on having a stable patch by today, sorry for the inconvenience

Should be fixed now. Let us know 👍 

Yup works

Confirmed (existing model then updated). Thanks for the fix and for HF great work!

It works now. Thanks! @mfuntowicz
Model card updated/deleted
Author: tuner007
Pull request: no

Hi,

I can check that the Model "tuner007/t5_abs_qa" has been removed from model hub...Is there anything i need to update ?
@patrickvonplaten  [Refer](https://huggingface.co/tuner007/t5_abs_qa/commit/faf30925ced0f25d0d5d321fb0ada04caaf5568d)

/thanks

Comments:
Hey @tuner007,

Thanks a lot for your issue! I'm very sorry that this has happened - this was a bug from my side :-/ I corrected it and your model should work as before now :-)

> Hey @tuner007,
> 
> Thanks a lot for your issue! I'm very sorry that this has happened - this was a bug from my side :-/ I corrected it and your model should work as before now :-)

No worries ! thanks
[Question] When pretraining a language model, can I choose to mask specific words?
Author: wenting-zhao
Pull request: no

Hi there,

I apologize if this is answered anywhere.  I need to pretrain a language model with some specific words masked.  I was wondering if this is currently supported?  Since language models are trained in an unsupervised way, I saw in the examples that the provided datasets don't need any labels.  However, I was thinking if it would be possible to create my own (sentence_with_masks, masked_words) pairs.  If library isn't currently supporting that, may anyone point me to a file so that I can make my modifications?  Thanks in advance!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[WIP] tweak model repo saving
Author: julien-c
Pull request: yes


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This was incorporated by @sgugger and @LysandreJik in another PR
loss sudden increase
Author: hongjianyuan
Pull request: no

![image](https://user-images.githubusercontent.com/41334719/126919981-52d9f969-8a26-4fef-ac91-3528b2426942.png)
I tried this a few times

Comments:
Hi,

We like to keep Github issues for bugs/feature requests. For training related questions, please see the [forum](https://discuss.huggingface.co/). Also, make sure to make it possible for people to reproduce your issue, by providing code or a colab notebook.

Thanks!
add `classifier_dropout` to classification heads
Author: PhilipMay
Pull request: yes

PR for #12781 and maybe a fix for #12792

# Logbook
- added to Electra: `SequenceClassification` and `TokenClassification`
- added to BERT: `SequenceClassification` and `TokenClassification`
- added to RoBERTa: `SequenceClassification` and `TokenClassification`
- added to big_bird: `SequenceClassification` and `TokenClassification`
- added to mobilebert: `SequenceClassification` and `TokenClassification`
- added to reformer: `SequenceClassification` 
and `TokenClassification`

- added to ConvBERT: `SequenceClassification` and `TokenClassification`
- added to albert: `SequenceClassification` and `TokenClassification`

Comments:
@sgugger should `classifier_dropout` be only added for `<model_name>ForSequenceClassification`
or also to `<model_name>ForTokenClassification`?

It should be used for both, yes.

This is still WIP - I want to add more models. Please do not merge yet ...

I am done with this PR. It is ready for review.
See logbook above for changed model types.
Add possibility to ignore imports in test_fecther
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds the ability to ignore some imports in the internal mechanism of the tests fetcher. The issue is that a recent PR introduced a dependency on the auto modules for `tokenization_utils_base`, which is a module that often ends up in a dep. Since this is just to grab the tokenizer class and issue an error, the auto module (and then all the models) does not need to be tested if there is a change of tokenization_utils_base.

This is a short-term fix, but a longer term patch is to reduce the the cross-dependencies inside the library.


Comments:
Possibly wrong API documentation for BigBirdTokenizerFast 
Author: nabito
Pull request: no

- `transformers` version: v.4.9.0

### Who can help

Documentation: @sgugger 

## Information

At this URL: 
https://huggingface.co/transformers/model_doc/bigbird.html#transformers.BigBirdTokenizerFast

The doc says,

`bos_token (str, optional, defaults to "[CLS]")`
and
`eos_token (str, optional, defaults to "[SEP]")`

but the actual code is:

```
    def __init__(
        self,
        vocab_file=None,
        tokenizer_file=None,
        unk_token="<unk>",
        bos_token="
",
        eos_token="
",
        pad_token="<pad>",
```
May be the API document need a fix? the explanation seems already clear though, only default value of those params are wrong.


Comments:
Thanks for flagging, should be fixed by the PR mentioned above!
TPU training is stuck using T5 with PyTorch Lightning
Author: mozharovsky
Pull request: no

## Environment info

- `transformers` version: 4.6.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

@patrickvonplaten, @patil-suraj

## Information

I'm fine-tuning the `t5-small` model using PyTorch Lightning on TPU v3 (Google Colab) on the `imdb` dataset. The training is stuck using an 8-cores setup and works well using a 1-core setup. It seems super weird since the `roberta-base` model works just fine using all 8-cores. 

I've filed a similar issue https://github.com/PyTorchLightning/pytorch-lightning/issues/7984, but would be great to receive some feedback if `t5` models are proved to work on TPUs using Lightning trainer.

## To reproduce

Please, use this Google Colab Notebook:
https://colab.research.google.com/drive/1FbWfkho3Otfl19y5ybkrK5Jw_tWNqV-M?usp=sharing

## Expected behavior

`t5` models should work fine using TPU 8-cores training setup.


Comments:
Hello! We don't have CI running with pytorch lightning so we would recommend opening an issue on their repository.

Did you try a TPU training with the `Trainer` that comes with `transformers`? It should work fine on 8 cores.

Hello @LysandreJik, thanks for replying! 

Yes, `🤗/Trainer` works perfectly fine with 8-cores, `t5-small` gets fine-tuned in under 10 minutes on TPU-v3. Nevertheless, it's still unclear whether this issue happens due to some lightning trainer internals or T5 model – `roberta-base` works fine with lightning using 8 TPU cores. 

The latter makes me think that it might be some T5-specific issue, mightn't be?

P.S. Lightning Trainer reports, that `lm_head.weight` parameter isn't tied (it seems missing prior to moving the model to the XLA device). Just in case.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Asking for consent to publish `_LazyModule` as a standalone PyPI package on GitHub
Author: PhilipMay
Pull request: no

Hi,

I very much like your `_LazyModule` implementation.

https://github.com/huggingface/transformers/blob/e218249b02465ec8b6029f201f2503b9e3b61feb/src/transformers/file_utils.py#L1945

I would like to reuse it on different other projects. That is why I ask for your consent to publish it as a 
standalone PyPI package on GitHub while keeping the license. Are you ok with that?

Comments:
Tagging @LysandreJik and @sgugger 

Thanks for asking! You can definitely package this class in a module as long as it's on the same license as in this repo (Apache 2.0) and you are willing to maintain it.

Here you go. A release will follow the next days:
https://github.com/telekom/lazy-imports

Many thanks again. I will close the issue now.
run_mlm.py errors when running validation only
Author: david-waterworth
Pull request: no

If you run `mlm_train.py' for validation only (i.e. --do_eval but don't pass -do_train) the script only runs if you still pass --train_file

This is due to the inference of the train file being used to determine the file type (i.e. text)

https://github.com/huggingface/transformers/blob/9ff672fc4d84db3b077e03ea22e2dafbd5d99fa4/examples/pytorch/language-modeling/run_mlm.py#L281

despite `--train_file` being an option argument.

https://github.com/huggingface/transformers/blob/9ff672fc4d84db3b077e03ea22e2dafbd5d99fa4/examples/pytorch/language-modeling/run_mlm.py#L131

It's useful being able to run eval only, it's not a big deal passing --train_file despite it not being used but given it's option the code should probably use train_file only if it's not none else validation_file


Comments:
I'm not sure what you mean, could you check you have the latest version of the script? There is a test of whether the `train_file` is None or not at the line you mention, and then the `validation_file` is used instead if the `train_file` has not been set.

@sgugger You're right, it was fixed in this commit https://github.com/huggingface/transformers/commit/9490d668d2f59ad2e7a4db3dc7ed2f9684af369c#diff-5f4433e38787dd047b331ec822da660195a786ea9350ad611623cd03d468b102

I'm using version 4.8.0
I donnot want print trainer's logging info
Author: cdhx
Pull request: no

torch 1.18.0，tf 1.14.0

every thing works well yesterday

i didn't do any thing, but today when i use predict of trainer, it print this infomation before the output 

this situation didn't occur before

```
***** Running Prediction *****
  Num examples = 1
  Batch size = 256
```

now i want this info don't print

I check the document, but didn't find a parameter to control don't print logger

```
In predict_loop  function

        batch_size = dataloader.batch_size
        num_examples = self.num_examples(dataloader)
        logger.info(f"***** Running {description} *****")
        logger.info(f"  Num examples = {num_examples}")
        logger.info(f"  Batch size = {batch_size}")
        losses_host: torch.Tensor = None
        preds_host: Union[torch.Tensor, List[torch.Tensor]] = None
        labels_host: Union[torch.Tensor, List[torch.Tensor]] = None
```


thanks @sgugger 

Comments:
You can use the argument `log_level` to adjust the level of the logger. If you set it to "warning", it won't print this.

thanks, it works
How to achive character lvl tokenization? (cant convert from huggingface/tokenizers)
Author: hadaev8
Pull request: no

Initially, I thought that huggingface/tokenizers is the same thing as tokenization in this repo.
I made it like this:
```
from tokenizers import Tokenizer, models, pre_tokenizers
from tokenizers.processors import TemplateProcessing

tokenizer = Tokenizer(models.WordLevel(unk_token='[UNK]'))
tokenizer.pre_tokenizer = pre_tokenizers.Split("", "isolated")


trainer = tokenizer.model.get_trainer()
trainer.vocab_size = 100
trainer.special_tokens = ["[UNK]", "[PAD]", "[SOS]", "[SEP]", "[EOS]"]

tokenizer.train(files=["alchemist.txt"], trainer=trainer)
tokenizer.post_processor = TemplateProcessing(
    single="[SOS] $A [EOS]",
    pair="[SOS] $A [SEP] $B:1 [EOS]:1",
    special_tokens=[
        ("[SOS]", 2),
        ("[SEP]", 3),
        ("[EOS]", 4),
    ],
)
tokenizer.enable_padding(pad_id=1, pad_token="[PAD]")
```

Still, huggingface/tokenizers lack some features I want like returning tensors.
So I tried to convert it to transformers tokenizer as suggested here
https://github.com/huggingface/tokenizers/issues/669#issuecomment-828864108

But got error:
```
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1631                 FutureWarning,
   1632             )
-> 1633             file_id = list(cls.vocab_files_names.keys())[0]
   1634             vocab_files[file_id] = pretrained_model_name_or_path
   1635         else:

IndexError: list index out of range
```

So, is tokenizer from tokenizers + converting to transformer tokenizer is a best way to achieve what I want?
If so, how should I convert it?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I just can not figure out why two api can not be the same. Extra Burden for users
`Seq2SeqTrainer` set max_length and num_beams only when non None 
Author: cchen-dialpad
Pull request: yes

# What does this PR do?

This PR slightly modifies the logic of setting `self._max_length` and `self._num_beams` in `Seq2SeqTrainer`'s `evaluate()` and `predict()` methods, i.e., the two variables will be set only when they are provided non `None` values.

This is to address a potentially inconsistent evaluation configuration inside the Seq2Seq training loop. For example, if you create a `Seq2SeqTrainer` object and invoke its `train()` method with a by epoch evaluation strategy, this line will do the evaluation after each training epoch: https://github.com/huggingface/transformers/blob/ba15fe7995a02357ecea6e7024918f6915564c36/src/transformers/trainer.py#L1437

`Seq2SeqTrainer ` subclasses `Trainer`, so the actual `evaluate()` method is https://github.com/huggingface/transformers/blob/ba15fe7995a02357ecea6e7024918f6915564c36/src/transformers/trainer_seq2seq.py#L36-L43

Now the problem is that `max_length` and `num_beams` can only be the default value `None` inside the training loop, since the training method is not aware of parameters introduced from the subclass.  To avoid this issue, this PR basically says that we will set the two variables only when non `None` values are provided. This allows users to set them using `seq2seq_trainer._max_length = 100` and `seq2seq_trainer._num_beams = 4` before entering the training loop (and won't be reset to `None` during training).



## Who can review?
@patrickvonplaten @sgugger 


Comments: