noamaanMulla-03/transformers-issues huggingface dataset:
Fix the wrong attention mask in TransformerXL by shifting `mask_shift_len` with 1
Author: tanchihpin0517
Pull request: yes

# What does this PR do?

In `forward` of TransformerXL, the attention mask `dec_attn_mask` is set up by:
```
dec_attn_mask = (torch.triu(all_ones, 1 + mlen) + torch.tril(all_ones, -mask_shift_len))[:, :,     None]  # -1
```
But I noticed if `mask_shift_len` is zero (while the default memory length equals to the input memory length, which is the most case), it will make `dec_attn_mask[0,:,:]` all 1, which make the model unable to get any attention from the first memory embedding.
I fix this problem by shift `mask_shift_len` with 1.

@patrickvonplaten

Comments:
Thanks a lot for your PR @tanchihpin0517. @TevenLeScao - could you maybe take a look here, I think you've added this part of the code a while back

Hey @tanchihpin0517 , if this is correct, I am surprised that it didn't show up in generation before. Have you noticed erroneous generation with TransformerXL in this setting?

No, it didn't generate an obvious error. It only makes the model ignore the first memory embedding. But in most of cases the length of memory is much longer (more than 64 or 128), so it just generate a tiny effect to the results.

hi @patrickvonplaten @TevenLeScao , is there any update here? Just remind myself. Thanks.

@TevenLeScao - could you take a look when you have a moment? Otherwise let me know and I'll try to allocate time

Sorry, I won't have the time in tge coming weeks, can you take it? I don't remember the code much better than you do anyway 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Traced models serialization and torchscripting fix
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

- Fixes the issue that was preventing traced models to be TorchScripted
- Fixes the issue that was preventing trace models serialization
- Fixes get_attr issues

Fixes #15974 

@jamesr66a Can you try on your end and validate that it solves your issues? 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

There seems to remain a few failures in `torch.fx` tests; the PR can be merged after those are solved!

@michaelbenayoun I'd like to propose some additional fixes that we discovered were needed to properly trace `T5ForConditionalGeneration`:

https://github.com/jamesr66a/transformers/commit/1a75148346cb471267b59fe7b473f304f6a02691

Can these be added?

Yes, for some reason the tests do not pass for torch 1.11 (I tested locally on torch 1.10).
I will add those changes too.

@michaelbenayoun Can I propose one final change to switch the graph surgery workaround to only trigger on older PyTorch versions where it's relevant?

https://github.com/jamesr66a/transformers/commit/5ac7bb737d9c9806051311a16f1799102d456fb8

Otherwise, when we're working on PyTorch nightly, this^ code breaks because it's trying to remove nodes that still have uses

@jamesr66a I added the gating, but only from version 1.12 as it was failing otherwise.

@michaelbenayoun Unfortunately, bumping the version check up to `1.12` breaks us. Actually, that was indirectly working around a semantic issue with deleting the concrete arg node. Do you mind augmenting the patch with this:

https://github.com/pbelevich/transformers/commit/e3fce52d1e14b1f75941fcaca7cd3029a6128016

@sgugger  Replaced the creation of a new flag by setting a special value for the `fx_compatible` flag for models that can be traced but not torchscipted (-1). 
This flag should take a boolean value 99% of the time anyways.

@michaelbenayoun This doesn't really work either. I'm not trying to be gratuitously painful here, but the common model tester is at the core of our test suite for the new model addition PRs. Those PRs are huge, and it only thanks to a robust CI that we can make sure the models added actually work with the whole API Transformers offers.

Adding a new flag, or a magic value for an existing flag, just because there is one model that needs different testing is not something we usually do or allow. In both cases, either the contributor or the reviewer will have no idea what your new flag/magic value does, especially since there is no documentation of it anywhere.

As I said before, in those instances where we need to adapt a common test to a specific model, we override it in the tester of said model. cc @LysandreJik and @patrickvonplaten
Add support for `device_map="auto"` to OPT
Author: sgugger
Pull request: yes

# What does this PR do?

I forgot to have OPT in the initial list of models supporting `device_map="auto"` (along side GPT-J, GPT-2 and T5). This PR takes care of it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17382). All of your documentation changes will be reflected on that endpoint.
Add FP16 Support for SageMaker Model Parallel
Author: haohanchen-aws
Pull request: yes

# What does this PR do?
(This PR is still pending some changes/fixes)
This PR adds support for SageMaker Model Parallel with FP16.

- To Enable fp16 with SMMP, user needs to add `fp16: True` to `SM_HP_MP_PARAMETERS`, when there's mismatch between `SM_HP_MP_PARAMETERS` and trainer args, a warning log would be printed and `SM_HP_MP_PARAMETERS` will be used as truth.
- Only do `uncale_` for `pp_rank` 0 in the beginning, as `scaler._scale `would be None at first for `pp_rank` > 0.
- Added `clip_master_grads` for grad clipping
- Some minor fixes




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Author: seanbenhur
Pull request: no
Labels: name: New model

### Model description

LayoutLMV3 is the successor of the LayoutLM models. the models are specialized in multimodal document analysis tasks and achieve SOTA results on them.

The current [code](https://github.com/microsoft/unilm/blob/master/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py) for the LayoutLMV3 is implemented in huggingface format.

Since the code is already more similar to the Huggingface format, I am not sure whether integrating this model into this repo is necessary, but there will be an ease of use if it is available in the library itself.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Model Implementation: https://github.com/microsoft/unilm/tree/master/layoutlmv3
Model Weights: https://huggingface.co/microsoft/layoutlmv3-base
Authors: Yupan Huang,Tengchao Lv, Lei Cui, Yutong Lu ,Furu Wei

Comments:
cc @NielsRogge :)
Adds support for OPT in Flax and TF. 
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Adds support for OPT in Flax and TF. 
Also clean Pytorch code a bit. 

## Who can review?
 @LysandreJik, @patrickvonplaten, @patil-suraj, @sgugger


Sorry for the two pull requests in a row, pulled from main instead of rebasing and had the entire commit history. Created a new branch to clean a bit. 

Comments:
> Thanks for adding those! Without the `TFOPTForCausalLM`, I don't see the point of adding the TF version of OPT since it can't really be used, so would either not add TF yet or make sure this model is added before merging the PR.

Yes I am not done yet! Sorry if I pinged you a bit early

@ArthurZucker let me know if the PR is ready for a review or you need help with the tests :-) 

> @ArthurZucker let me know if the PR is ready for a review or you need help with the tests :-)

I just have 1 last test that behave strangely (its more about padding tokens and positional embedings) but the jax code will be ready for review tomorrow 12am. Then will work quickly on the tf code and the PR should be ready by the end of the week! 

FLAX code is pretty much done. The only test that I can't solve is the difference in output for the jited model generation! 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17227). All of your documentation changes will be reflected on that endpoint.

@ArthurZucker could we add a test similar to this one: https://github.com/huggingface/transformers/pull/17359 to both Flax and TF? 

@Rocketknight1 @gante could you check the TF version here as well? 

@ArthurZucker,

Do you think we could fix the PR (I think the PR history is a bit messed up). Also totally fine to close this PR and just open a new PR (move all the relevant files to a new PR) if the git correction is too difficult

> @ArthurZucker,
> 
> 
> 
> Do you think we could fix the PR (I think the PR history is a bit messed up). Also totally fine to close this PR and just open a new PR (move all the relevant files to a new PR) if the git correction is too difficult

Hey, I think we can close it. 
Will create a new clean branch
Correct & Improve Doctests for LayoutLMv2
Author: garyhlai
Pull request: yes

# What does this PR do?

Ready for Review now. 

Issue #16292

Add correct doctests for LayoutLMv2

Note: LayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`; passing doctests requires their installation. I documented this in the docstring. 


## Who can review?

@patrickvonplaten  @ydshieh  @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @ydshieh Sorry for the late response and thank you for reviewing! Just made the changes addressing all the comments here.

Thanks for this PR - although the update in the docs doesn't seem correct to me.

@NielsRogge just started a follow-up PR #17409. Let me know if there's anything else I can do.
Fix expected value for OPT test `test_inference_no_head`
Author: ydshieh
Pull request: yes

# What does this PR do?

- Update the expected value in the test `OPTModelIntegrationTests.test_inference_no_head` to have more precisions
- Lower atol to `5e-5`

On GPU VM, the test has to be run with TF32 disabled (or without TF32 support).

See: https://pytorch.org/docs/stable/notes/cuda.html
Related discussion: #16588

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Keep only Roberta's position_embeddings initialisation in modeling_roberta.py
Author: AlexandreNap
Pull request: yes

# What does this PR do?

There was two position_embeddings initialisations, first one was left from Bert's code and second one is Roberta's tweaked version. Remove first one.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@LysandreJik

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make check_init script more robust and clean inits
Author: sgugger
Pull request: yes

# What does this PR do?

This PR was triggered by the inits deactivating the formatter (see `wav2_vec2_with_lm` below) which was a bit sad. In one line, `check_inits.py` was unable to parse `_import_structure` initialized in one line, this PR addresses that and cleans many model inits.

HuBERT also contained some reference to Wav2Vec2FeatureExtractor, which should not be the case. This PR cleans that up as well.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix cvt docstrings
Author: AnugunjNaman
Pull request: yes

# What does this PR do?

This PR does the following:
1. Remove the error in `README.md` where `CvT` description was copy of `CTRL`
2. Fix `size` of image for `feature extractor` which was set to `224`.
3. fix the input docstring for forward classes of `CvtModel` and `CvtForImageClassification` (head mask etc not needed).

@NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Will merge as soon as it's green!

@LysandreJik @NielsRogge You can merge it now. Thanks for review! 😊
Inconsistency multiple mask in fill-mask
Author: mo6zes
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.14.1
- Platform: Darwin-21.5.0-x86_64-i386-64bit
- Python version: 3.6.13
- PyTorch version (GPU?): 1.10.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
pipe = pipeline("fill-mask", top_k=1)

pipe("It is <mask>.")
=> [{..., 'sequence': 'It is true.'}]

pipe("It is <mask> <mask>.")
=> [[{..., 'sequence': '
It is very<mask>.
'}], [{..., 'sequence': '
It is<mask>orable.
'}]]
```

### Expected behavior


I would expect that the "sequence"  does not include `
` and `
` tokens. It also seems to remove whitespace before the `<mask>` tokens left in the result, but I believe that does not make a difference for the tokenizer.

What I would expect:
```shell
pipe = pipeline("fill-mask", top_k=1)

pipe("It is <mask>.")
=> [{..., 'sequence': 'It is true.'}]

pipe("It is <mask><mask>.")
=> [[{..., 'sequence': 'It is very <mask>.'}], [{..., 'sequence': 'It is <mask>orable.'}]]
```


Comments:
nvm it seems deliberate  https://github.com/huggingface/transformers/blob/374a2f693f75305eded1a2bb7a7e452f0ab33fad/src/transformers/pipelines/fill_mask.py#L137-L140
Add test for new model parallelism features
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds common tests for the new model parallelism/CPU offload features. Those are activated for models having a `_no_split_modules` attribute, so for now GPT-2, GPT-J, OPT and T5. The tests are only run on GPU and multi-GPU (so the CI won't catch any failure on the PR) but they all pass locally for me.

In passing I hit two blockers which this PR fixes:
- the ability to pass `max_memory` directly into `from_pretrained` to limit the memory used in `device_map="auto"`
- the CPU offload wasn't working with T5 because of some `device` taken from the model parameters instead of the input.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
run_translation.py: resize decoder based on tokenizer
Author: AmitMY
Pull request: yes

# What does this PR do?

Changes the `run_translation.py` script to resize the decoder based on the tokenizer.


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16805). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Adding Bigscience 176B parameters model
Author: thomwolf
Pull request: yes

# What does this PR do?

This PR add BigScience 176B parameters model to transformers.

The model architecture is detailed here: https://bigscience.notion.site/BigScience-176B-Model-Training-ad073ca07cdf479398d5f95d88e218c4

The training of the model can be followed here: https://twitter.com/BigScienceLLM

The model is trained using 3D parallelism with a combination of data (DP), pipeline (PP) and tensor (TP) parallelisms (see an illustration [here](https://twitter.com/BigScienceLLM/status/1506588988278198273?s=20&t=mxyFZQvVTM3YFDaQeHvlHQ)). The training code is a fork of NVIDIA/Microsoft Megatron-DeepSpeed and can be found here: https://github.com/bigscience-workshop/Megatron-DeepSpeed)

For integration in transformers, the following approach is being currently taken:
- focus on using the model in inference only (will impact PP questions - no backward pass, no bubble - may also impact the need for TP - we can also focus on batch-size=1 for now => no DP)
- try to be able to use the model on a rather small workstation (with all caveats but goal is to get the integration to work on less than the 48 GPUs currently used for training)
- be very pragmatic - we can always develop optimized version of the model/code later

# Current status:
This PR is still a not working draft.

During the integration a potential bug in the original training code was discovered where the layer norms on the parallel branches of the TP direction were not identical unlike expected. The integration was then stop to investigate the situation and should start again when the issue is solved on the training side.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
The tokenizer config for OPT-30B is missing a pad token
Author: aninrusimha
Pull request: no
Labels: name: bug

### System Info

```shell
Version 4.20.0.dev0, built from source
Issue is in https://huggingface.co/facebook/opt-30b/blob/main/tokenizer_config.json
```


### Who can help?

@patrickvonplaten has his name in the code :)
Discovered when testing the OPT models on various datasets.
https://huggingface.co/facebook/opt-30b/blob/main/tokenizer_config.json is missing a padding token
https://huggingface.co/facebook/opt-13b/blob/main/tokenizer_config.json looks to have the real config for opt-13b?

https://huggingface.co/facebook/opt-13b/blob/main/tokenizer_config.json 
{"errors": "replace", "unk_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "add_bos_token": true, "special_tokens_map_file": null, "name_or_path": "patrickvonplaten/opt-30b", "tokenizer_class": "GPT2Tokenizer"}

https://huggingface.co/facebook/opt-30b/blob/main/tokenizer_config.json
{"errors": "replace", "bos_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "special_tokens_map_file": null, "name_or_path": "patrickvonplaten/opt_gpt2_tokenizer", "tokenizer_class": "GPT2Tokenizer"}

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following code snippet

```
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="validation")
dataset = [s['text'] for s in dataset if s['text'] != '']
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-30b", use_fast=False)
encoded = tokenizer(dataset, 
            return_tensors="pt", 
            padding=True)
```
You will see an error because the tokenizer lacks a padding token.

### Expected behavior

```shell
The OPT-30B tokenizer has a padding token and pads the input.
```


Comments:
That's an important bug and completely on me! Thanks a mille for spotting it @aninrusimha !

Fixed it - https://huggingface.co/facebook/opt-30b/discussions/1