noamaanMulla-03/transformers-issues huggingface dataset:
🚀 Feature request Multimodal BERT Models
Author: ecekt
Pull request: no
Labels: name: wontfix

Hello, it would be great if more **multimodal** BERT models are included in the library. I have noticed that MMBT from Facebook is provided; however, I was unable to find some guidelines about how to make it work with the help of 🤗 Transformers. 

Possible models can be [VilBERT](https://arxiv.org/abs/1908.02265), [VL-BERT](https://arxiv.org/abs/1908.08530), [VisualBERT](https://arxiv.org/abs/1908.03557), [VideoBERT](https://arxiv.org/abs/1904.01766) and so on.

Best regards.

Comments:
As for guidelines about making MMBT work, here is an example on the mm-imdb dataset: https://github.com/huggingface/transformers/blob/master/examples/mm-imdb/run_mmimdb.py.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi seems like above examples folder has been removed, is it because the multi modal experiment is in intermediate stage.

I believe it's available here: https://github.com/huggingface/transformers/tree/master/examples/contrib/mm-imdb

Hey is there any progress with it soon? 
I find only the mm-imdb example: https://github.com/huggingface/transformers/tree/master/examples/contrib/mm-imdb

Your LXMERT model receives only text features from what I see ("visual_feats - These are currently not provided by the transformers library.")

Thanks :) 

The example for mmbt on mm-imdb is also an invalid link now. 

Here is a correct link for now https://github.com/huggingface/transformers/tree/master/examples/research_projects/mm-imdb

> As for guidelines about making MMBT work, here is an example on the mm-imdb dataset: https://github.com/huggingface/transformers/blob/master/examples/mm-imdb/run_mmimdb.py.

The link is broken!

> The link is broken!

See the reply above you :) That seems to work
stop at load tokenizer_config.json when run barthez for mrpc 
Author: gongzheng0
Pull request: no

run code
```shell
python examples/text-classification/run_glue_tune.py --model_name_or_path /home2/zhenggo1/checkpoint/barthez_mrpc --task_name $TASK_NAME --do_eval --tune --max_seq_length 512 --output_dir /home2/zhenggo1/checkpoint/barthez_mrpc --tuned_checkpoint="/home2/zhenggo1/checkpoint/barthez_mrpc"
```
stop at here
```shell
[INFO|tokenization_utils_base.py:1618] 2021-05-13 10:11:25,859 >> Model name '/home2/zhenggo1/checkpoint/barthez_mrpc' not found in model shortcut name list (moussaKam/mbarthez, moussaKam/barthez, moussaKam/barthez-orangesum-title). Assuming '/home2/zhenggo1/checkpoint/barthez_mrpc' is a path, a model identifier, or url to a directory containing tokenizer files.
[INFO|tokenization_utils_base.py:1651] 2021-05-13 10:11:25,860 >> Didn't find file /home2/zhenggo1/checkpoint/barthez_mrpc/tokenizer.json. We won't load it.
[INFO|tokenization_utils_base.py:1651] 2021-05-13 10:11:25,860 >> Didn't find file /home2/zhenggo1/checkpoint/barthez_mrpc/added_tokens.json. We won't load it.
[INFO|tokenization_utils_base.py:1714] 2021-05-13 10:11:25,860 >> loading file /home2/zhenggo1/checkpoint/barthez_mrpc/sentencepiece.bpe.model
[INFO|tokenization_utils_base.py:1714] 2021-05-13 10:11:25,860 >> loading file None
[INFO|tokenization_utils_base.py:1714] 2021-05-13 10:11:25,860 >> loading file None
[INFO|tokenization_utils_base.py:1714] 2021-05-13 10:11:25,860 >> loading file /home2/zhenggo1/checkpoint/barthez_mrpc/special_tokens_map.json
[INFO|tokenization_utils_base.py:1714] 2021-05-13 10:11:25,860 >> loading file /home2/zhenggo1/checkpoint/barthez_mrpc/tokenizer_config.json
```

Comments:
No problem.It is just loading too long as 9 minute.
Loading Basic GPT-2  model gives warning that attention layers weren't loaded from pre-trained weights
Author: pgarz
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0
- Platform: Linux-4.19.0-16-cloud-amd64-x86_64-with-debian-10.9
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes. One GPU on Google Cloud Compute
- Using distributed or parallel set-up in script?: No

Tagging the following people for assistance
- gpt2: @patrickvonplaten, @LysandreJik


## Information

I'm using the GPT-2 vanilla model. Locally on my MacbookPro, my code runs as expected. When I try to run my Jupyter notebook on GCP, I however encounter an unseen warning. I provided the env details of said GCP instance above. I'm simply trying to load the vanilla GPT-2 model, but I keep getting the warning that the attention layers are not being initialized from the pre-trained weights as intended.

I get the following warning message:
`Some weights of GPT2Model were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.1.attn.masked_bias', 'h.8.attn.masked_bias', 'h.2.attn.masked_bias', 'h.10.attn.masked_bias', 'h.7.attn.masked_bias', 'h.4.attn.masked_bias', 'h.11.attn.masked_bias', 'h.9.attn.masked_bias', 'h.0.attn.masked_bias', 'h.3.attn.masked_bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.`

This happens when I attempt to execute the super simple model loading statement:

` pretrained_transformer = GPT2Model.from_pretrained('gpt2')`

I've seen similar issues people have posted, however, they tend to be loading a model and trying to load the weights to a different type of model. Here I am strictly trying to load a GPT2 model weight set and then configure the same GPT2 model with said weights.

I'm worried these warnings are real and it seems my experiments on GCP are not looking the same as the ones locally due to the weights not being loaded properly.

## To reproduce

Steps to reproduce the behavior:

1. Make GCP notebook
2. Try to load GPT-2 model

## Expected behavior

The behaviour I desire is no warning messages and being able to use all the fully trained pre-trained weights as I'm able to do locally. 



Comments:
These warnings mention that buffers are not loaded, which is normal - they're created during the model initialization.

There was an attribute missing on the `GPT2Model` which led the warnings to still be raised, I'm fixing this in #11709!

Installing from source should resolve the warnings issue :)
EncoderDecoderModel with different model dimensions
Author: LarsHill
Pull request: no
Labels: name: WIP

## Who can help
@patrickvonplaten, @patil-suraj

## Information

When instantiating an `EncoderDecoderModel` from two pretrained models whose model dimensions are different, a `RunTimeError` occurs at the `CrossAttention` calculation step. 

The reason is, that regardless of a potentially different encoder model dimension, the projection layers for key and value are initialized with the decoder model dimension. 

This leads to a dimensionality mismatch when performing the matrix multiplication of encoder outputs (encoder model dimension) in the key and value projection layers (decoder model dimension).

Looking a little bit deeper in the API I would suspect it should be easy to provide the correct encoder model dimension to the `Attention` module in most Model implementations and their key/value projection layers, if the `add_cross_attention=True` argument is set. Also, I think the encoder model dimension should be easily accessible via `self.encoder.config.d_model` or something along these lines.

Generally, I think there is no reason against using `EncoderDecoderModel` with `encoder='bert-large-cased'` (`d_model=1024`) and `decoder='gpt2'` (`d_model=768`), but currently this setup doesnt't work.

Thanks a lot for looking into it :) 

Best regards
Lars

Comments:
Hey @LarsHill, 

Yes, we should fix this indeed :-) I'll try to open a PR for this this week!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[T5] Add 3D attention mask to T5 model (2) (#9643)
Author: lexhuismans
Pull request: yes

# What does this PR do?
It allows for 3D attention mask in T5 model (modeling_t5.py) with an accompanying test. 

Fixes #9643

This is a clean version for an earlier PR #10903.

This is a solution for allowing the 3D attention mask in the T5 model by making it broadcastable. It is based on what is used in BERT.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Who can review?
@patrickvonplaten 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Great job @lexhuismans !

Error is unrelated -> merging
Flag to disable shuffling for data loader
Author: hasansalimkanmaz
Pull request: no

# 🚀 Feature request

Currently, Trainer is shuffling the train_dataset by default and there is no flag to enable/disable it. 

@sgugger  

## Motivation

Even if shuffling the dataset brings a lot of benefits like preventing overfitting, at some point, one can need to disable it for experimental motivation. It isn't possible to do it without overwriting the `_get_train_sampler` method of Trainer. :( 

## Your contribution

I can work on this issue (maybe next month) if this issue gets positive feedback.

Comments:
I don't think this is a suitable feature, so I would recommend you override and subclass `get_train_dataloader` to simply return a training dataloader without using a sampler or `shuffle=True`.
Problems with webbased editing of model cards 
Author: PhilipMay
Pull request: no

When I open a model in hugging face model repository - like here: https://huggingface.co/german-nlp-group/electra-base-german-uncased

And then click "Edit model card" the text in the webbased editor contains `\r` characters. When the webbased editor is now used to save the modelcard these characters are saved and shown.

See screenshot:
![image](https://user-images.githubusercontent.com/229382/115135797-88865c00-a01b-11eb-9fae-de9fc3b48908.png)

This is a bug.

Comments:
The probem only seems to happen on some models. This model here works ok: https://huggingface.co/T-Systems-onsite/cross-en-de-roberta-sentence-transformer

Tagging @Pierrci for visibility

Hi @PhilipMay, thank you for reporting this, we just deployed a fix for that, let us know if you still encounter the problem, otherwise feel free to close the issue :)

Works now. Thanks.
Add parallelize method to GPT-neo models
Author: anamtamais
Pull request: no

# 🚀 Feature request

Add `parallelize` method to GPT-neo models, so we can finetune them using model parallelism using less expensive GPUs.

## Motivation

I want to finetune a GPT-neo model using model parallelism in order to do it using less expensive GPUs. It is not yet implemented, and, as higher-end GPUs are too expensive, it would be better if we distributed the model along several less expensive GPUs  rather than using a very expensive one. It would also make it possible for us to iterate using larger batches, what can have big impact on the model fitting. 

I would be very glad if you people could do it and I think it would enable the finetuning of specific purpose GPT-neo language models.

Comments:
Hi @anamtamais,

We decided that it's not worth investing time into porting the naive MP solution to other models beyond t5+gpt2 since this solution doesn't scale well resource-wise. And given the 2 existing implementations of ZeRO (fairscale and DeepSpeed) this is by far more scalable solution, in particular now that ZeRO stage 3 has been released. You don't need high-end GPUs for ZeRO.

We have everything ready on our side https://github.com/huggingface/transformers/pull/10753, just waiting for the DeepSpeed team to merge several PRs and make a new release. If you want to try it right away, you can use the 2 branches posted here https://github.com/huggingface/transformers/issues/11044 

Also there are notes comparing the different scalability solutions here: https://github.com/huggingface/transformers/issues/9766

If you have any questions please let me know.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Training mask language model using multiple files
Author: lindan1128
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.5
- Platform:Linux
- Python version:3.7
- PyTorch version (GPU?):1.7.1
- Tensorflow version (GPU?):
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?:No

Hi all,

I am very new to transformer. Now I am training my own mask language models using big data (~500G). And these data are saved in several files. When I follow the tutorial (https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=YpvnFFmZJD-N), I find that they always train the model using only one training file. So, my question is that how to train the model using several training files? Hope you can help me! Thank you very very much!


Comments:
@lindan1128 the same code can be used for multiple files too, check this cell:-

```
%%time 
from pathlib import Path

from tokenizers import ByteLevelBPETokenizer

paths = [str(x) for x in Path(".").glob("**/*.txt")]

# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()

# Customize training
tokenizer.train(files=paths, vocab_size=52_000, min_frequency=2, special_tokens=[
    "
",
    "<pad>",
    "
",
    "<unk>",
    "<mask>",
])
```

Here paths will get all the txt files in the current directory and will read all of them.

@frankhart2018 It indeed works!!! Thank you very much!!!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add batching in TokenClassificationPipeline
Author: parakalan
Pull request: yes

# What does this PR do?

Currently, the NER pipeline in transformers iterates through the list of input sentences and processes them, sequentially.
This PR adds batching support in the pipeline to decrease latency and use GPU more efficiently.

Relevant Issue :-  #11244 

## Benchmark Report

### Without Batching (CPU)
Device: CPU
No. examples: 1000
Time taken: 283.27826976776123

Device: GPU
No. examples: 1000
Time taken: 17.89318561553955

Please check the benchmark gist [here](https://gist.github.com/parakalan/88b613ed4ca0001afb60448996f6b62a)

### Without Batching (CPU)
Device: CPU
No. examples: 1000
Batch Size: 512
Time taken: 121.81582999229431

Device: GPU
No. examples: 1000
Batch Size: 512
Time taken: 2.780881404876709

Please check the benchmark gist [here](https://gist.github.com/parakalan/f1fa25f25b8a70125145afbcbbeac85f)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. - https://github.com/huggingface/transformers/issues/11244
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
FYI there is also work done on this pipeline in https://github.com/huggingface/transformers/pull/10568 if you want to give it a look! It doesn't concern batching, however.

Thanks, let me check that out. 

Please review this  @LysandreJik , @Narsil , @joshdevins

Closing this PR based on @Narsil's review. Thanks
Fix loading the best model on the last stage of training
Author: vbyno
Pull request: yes

# What does this PR do?

It fixes the best model loading on the last stage of the training.

Fixes #11666

## Who can review?
@sgugger 



Comments:
Great! Lat thing is to run `make style` on your branch to make the CI pass. Let me know if you run into any issue, I can also push on your branch.

> Great! Lat thing is to run `make style` on your branch to make the CI pass. Let me know if you run into any issue, I can also push on your branch.

CI is already fixed

Thanks again for the fix!
Why run_translation.py automatically runs on CPU?
Author: ustcwhy
Pull request: no

I use examples/pytorch/translation/run_translation.py fine-tune mbart-large-cc25 on my datasets, it automatically runs on CPU.  I have 2 GPU, but only one is Nvidia.It is RTX 2080super. 

python main.py \
--model_name_or_path facebook/mbart-large-cc25 \
--do_train  \
--do_eval \
--source_lang en_XX \
--target_lang zh_CN \
--train_file /data/2WangHongyu/bioNMT_WHY/train.json \
 --validation_file /data/2WangHongyu/bioNMT_WHY/dev.json \
 --output_dir /output \
--per_device_train_batch_size=4 \
--per_device_eval_batch_size=4 \
--overwrite_output_dir \
--predict_with_generate \
 --cache_dir /model/2WangHongyu/mbart-large 




Comments:
Duplicate of https://github.com/huggingface/transformers/issues/11548#issuecomment-831159016

Could you please answer to 

> Hi! Is your CUDA environment correctly set up? What is the output of the following in your environment?
> 
> `python -c "import torch;print(torch.cuda.is_available())"`

and how do you identify that it's not running on GPU? Could you put the accompanying logs? At which point do you see it running on CPU when you think it should be running on GPU?

> Duplicate of [#11548 (comment)](https://github.com/huggingface/transformers/issues/11548#issuecomment-831159016)
> 
> Could you please answer to
> 
> > Hi! Is your CUDA environment correctly set up? What is the output of the following in your environment?
> > `python -c "import torch;print(torch.cuda.is_available())"`
> 
> and how do you identify that it's not running on GPU? Could you put the accompanying logs? At which point do you see it running on CPU when you think it should be running on GPU?

True…
While the program is running, the GPU utilization is close to 0%, and the CPU utilization is close to 10%. After loading weights, the cmd window keeps showing ? it/s