noamaanMulla-03/transformers-issues huggingface dataset:
Generate: validate `model_kwargs` on FLAX (and catch typos in generate arguments)
Author: gante
Pull request: yes

# What does this PR do?

FLAX version of https://github.com/huggingface/transformers/pull/18261

Adds model_kwargs validation to FLAX generate, which also catches typos in the arguments. See the PR above for more details and an example of the error message users will see.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Rename method to avoid clash with property
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes a bug when the `self.rescale` method was being called, but `self.rescale` was set as a property. This arose after a refactoring of the feature extractors in my PR: https://github.com/huggingface/transformers/pull/18499, causing an issue in any model that had `rescale` as a property in its config. Renaming considered safe as the new `rescale` method was introduced yesterday. 

Issue raised by user here: https://github.com/huggingface/transformers/commit/49e44b216b2559e34e945d5dcdbbe2238859e29b#comments
and caught in doctests. 

Renames the method to be less ambiguous. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The layoulmv2 and layoutlmv3 tests appear to be failing due to an issue when importing from the [detectron2](https://github.com/facebookresearch/detectron2) package. 

When `from detectron2.modeling import META_ARCH_REGISTRY` is called, it fails with the error `ImportError: cannot import name 'is_fx_tracing' from 'torch.fx._symbolic_trace'`. This has been reported by other users of the library - see [commit comments](https://github.com/facebookresearch/detectron2/commit/36a65a0907d90ed591479b2ebaa8b61cfa0b4ef0#comments).

@ydshieh I believe it comes from the `preprocessor_config.json` file: https://huggingface.co/google/owlvit-large-patch14/blob/main/preprocessor_config.json. 



> @ydshieh I believe it comes from the `preprocessor_config.json` file: https://huggingface.co/google/owlvit-large-patch14/blob/main/preprocessor_config.json.

I will update the `preprocessor_config.json` files, the rescale argument is not used anymore.

@ydshieh Yep - will do!
Fix repo consistency
Author: lewtun
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes a repo consistency error introduced by the merge of https://github.com/huggingface/transformers/pull/17976


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix breaking change in `onnxruntime` for ONNX quantization
Author: severinsimmler
Pull request: yes

Using `onnxruntime.quantization.quantize` has been deprecated since a while:

```
WARNING:root:onnxruntime.quantization.quantize is deprecated.
         Please use quantize_static for static quantization, quantize_dynamic for dynamic quantization.
```

and has been removed from `onnxruntime` in v1.12.0 (released 6 days ago). This breaks quantization in `transformers.convert_graph_to_onnx`.

I have migrated `onnxruntime.quantization.quantize` to `transformers.convert_graph_to_onnx.quantize` to preserve compatibility.


## Who can review?
@mfuntowicz 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@JingyaHuang @lewtun @michaelbenayoun Happy to implement changes if you have any suggestions 😊 

Thanks @lewtun :pray: All tests are passing on my machine:

![image](https://user-images.githubusercontent.com/16133277/185413954-4a41a3ee-c6b3-41bd-a887-107ef1a66be0.png)
LayoutLMv2ForRelationExtraction is missing in transformers
Author: binkjakub
Pull request: no

### Feature request

Microsoft's [unilim repository](https://github.com/microsoft/unilm/tree/db1095a693aa0d6d15bb9312cccb7f8af42b0aeb/layoutlmft/layoutlmft), which originally implements all `LayoutLM` models, contains implementation of the model for relation extraction with Biaffine Attention Classifier, namely [`LayoutLMv2ForRelationExtraction`](https://github.com/microsoft/unilm/blob/db1095a693aa0d6d15bb9312cccb7f8af42b0aeb/layoutlmft/layoutlmft/models/layoutlmv2/modeling_layoutlmv2.py#L895). However, this class wasn't included in `transfromers`, due to unknown reasons. Therefore, the implementation for relation extraction `LayoutLMv2ForRelationExtraction` should be included to extend current LayoutLMv2 and LayoutXLM.

### Motivation

This repository should implement all tasks included in papers ([LayoutLMv2](https://arxiv.org/pdf/2012.14740.pdf), [LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf)) and [unilim repository]((https://github.com/microsoft/unilm/tree/db1095a693aa0d6d15bb9312cccb7f8af42b0aeb/layoutlmft/layoutlmft)), thus this missing part should be added to `transformers`. It would enable users to easily reproduce entire paper as well as convenient use of relation extraction in their downstream applications.

### Your contribution

If there are no obstacles unknown to me, I could try to move the implementation from unilim to transformers.

Comments:
cc @NielsRogge 

+1 to this

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
support no gpt_j_residual for gpt-neox
Author: TopIdiot
Pull request: yes

# What does this PR do?

Support "gpt_j_residual == False"  for gpt-neox, which implement "else" branch in https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/transformer.py#L627


## Who can review?
Anyone in the community is free to review the PR.
@sgugger @patrickvonplaten 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18089). All of your documentation changes will be reflected on that endpoint.

@TopIdiot, would existing checkpoints (i.e., https://huggingface.co/EleutherAI/gpt-neox-20b) be usable with this configuration option, or would they output gibberish?

> @TopIdiot, would existing checkpoints (i.e., https://huggingface.co/EleutherAI/gpt-neox-20b) be usable with this configuration option, or would they output gibberish?

@LysandreJik  Yes, it can be used. The default value for gpt_j_residual is "True",  so the exsiting checkpoints would enter the "if self.gpt_j_residual" branch. btw, I tested gpt-neox-20b, the result looks good for me.

The question is whether the model would give sensible results if you set it to `False`. Transformers is not a modular toolbox, we don't add options to models that make no sense with the pretrained checkpoints of that model, we add new models instead.

> The question is whether the model would give sensible results if you set it to `False`. Transformers is not a modular toolbox, we don't add options to models that make no sense with the pretrained checkpoints of that model, we add new models instead.

@sgugger gpt_j_residual  is an option which already existed in gpt-neox (i.e.  https://github.com/EleutherAI/gpt-neox/blob/main/configs/20B.yml#L29)

Recently, we trained EleutherAI/gpt-neox with option gpt_j_residual == False and successfully convert the checkpoint to torch. However, we found that the current version of huggingface's gpt-neox doesn't implement this part. We added it by ourself and got expected result. 

You can share the custom code of your model using the [code in the Hub](https://huggingface.co/docs/transformers/custom_models) API. This is typically the kind of changes we don't accept in existing model files (and the reason there is one for GPT-2, GPT-J, GPT-Neo and GPT-Neo-X which are all very similar).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add evaluate to examples requirements
Author: muellerzr
Pull request: yes
Labels: name: Examples

# What does this PR do?

This PR adds `evaluate` as part of the requirements to all of the examples scripts that need them

Fixes # (issue)

Closes https://github.com/huggingface/transformers/issues/18663

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[bnb] Move documentation
Author: younesbelkada
Pull request: yes

# What does this PR do?


- move bnb documentation to `perf_infer_gpu_many.mdx` as it was previously set to `perf_train_gpu_one.mdx` which is not relevant in the case of `bitsandbytes` integration since it supports inference only.

cc @stas00

I do have a question though, what about `perf_infer_gpu_one.mdx`? I think that the `bnb` documentation could fit well in this file as well since it supports single GPU inference too.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

yeah, I wasn't sure, probably you're right and then like  perf_train_gpu_many.mdx says on top to first read perf_train_gpu_one.mdx - add the same to perf_infer_gpu_many.mdx?

Yep makes sense! I propose a small refactoring at 2018285dd60c34d2654b0c51e523d7b1e815b989 !
Let me know if this works for you 

The proposed change would be difficult to maintain and 2 copies will get out of sync. Only one copy please - if you prefer the one gpu doc that's where it should be. the other one linking to it.

Proposed a change in 5f8a3aeb035e8b086d0f9045550000b6be0fb630 ! Let me know if this works for you 

Thanks a lot @stas00 for iterating on the changes 💪
`model.tie_weights()` should be applied after `accelerator.prepare()`
Author: Gladiator07
Pull request: yes

# What does this PR do?
Weight tying should be done after the model has been moved to XLA device as mentioned on PyTorch/XLA Troubleshooting guide [here](https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md#xla-tensor-quirks). This PR fixes this in `examples/pytorch/language-modeling/run_mlm_no_trainer.py` example script.



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review ?
@sgugger @muellerzr

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @Gladiator07 great work!
OOM error when training with trainer
Author: zunlongzhou
Pull request: no
Labels: name: bug

### System Info

transformers: 4.19.2

When I use trainer to do mlm training on deberta-v3-large, there is out of memory problem.
And the GPU occupancy continues to grow over time, eventually out of memory.and throws OOM error every time at fixed position

By searching forums and issues, I tried to modify the source code of trainer, including

1. in the huggingface transformers trainer code (function create_optimizer), add force_broadcast_object=True
2. rewrite the Trainer saving function to skip saving the optimizer weights
3. disable the optimizer saving by commenting out consolidate_state_dict as well as the optimizer saving part
4. Remove useless intermediate variables and empty_cache

but it didn't work


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


training_args = TrainingArguments(
    output_dir=args.output_dir,
    evaluation_strategy="no",
    learning_rate=args.lr,
    weight_decay=0.01,
    save_strategy='steps',
    per_device_train_batch_size=args.batch_size,
    num_train_epochs=args.num_train_epochs,
    # report_to="wandb",
    run_name=f'output-mlm-{args.exp_num}',
    # logging_dir='./logs',
    lr_scheduler_type='cosine',
    warmup_ratio=0.2,
    fp16=True,
    logging_steps=500,
    gradient_accumulation_steps=args.gradient_accumulation_steps,
    save_steps=5000, 
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    # eval_dataset=tokenized_datasets['valid'],
    data_collator=data_collator,
    # optimizers=(optimizer, scheduler)
)


### Expected behavior

Hope it works without changing batch_size

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generate: add missing `**model_kwargs` in sample tests
Author: gante
Pull request: yes

# What does this PR do?

One call of the `sample`-related tests was missing `**model_kwargs`, which MAY explain the random failures we were seeing. 

I've run all `test_sample_generate_dict_output` tests 100x with no failures. Before this change, it was failing once every ~10 calls of `py.test tests/ -k test_sample_generate_dict_output`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh when the attention mask is not passed, it is inferred from the input ([here](https://github.com/huggingface/transformers/blob/e54a1b49aa6268c484625c6374f952f318914743/src/transformers/generation_utils.py#L490)).

Depending on the pad and eos tokens, the mask inferred from the random input tokens may have issues. This automatic attention mask is a big source of issues in general, especially for the tests :( 

Passing the mask explicitly is always preferred!
What is the meaning of Attention Mask
Author: jianyucai
Pull request: no

Hi, I noticed that there is something called `Attention Mask` in the model.

In the annotation of class `BertForQuestionAnswering`, 

```python
`attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
            selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
            input sequence length in the current batch. It's the mask that we typically use for attention when
            a batch has varying length sentences.
```

And its usage is in class `BertSelfAttention`, function `forward`,

```python
# Apply the attention mask is (precomputed for all layers in BertModel forward() function)
attention_scores = attention_scores + attention_mask
```

It seems the attention_mask is used to add 1 to the scores for positions that is taken up by real tokens, and add 0 to the positions outside current sequence. 

Then, why not set the scores to `-inf` where the positions are outside the current sequence. Then pass the scores to a softmax layer, those score will become 0 as we want.

Comments:
Yes, this conversion is done inside the model, see this line: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L626
(we don't use infinity but a large value that works also when the model is used in half precision mode)

Thanks for your answer. Well, I still have a little problem understanding what you mean in the last sentence:

> (we don't use infinity but a large value that works also when the model is used in half precision mode)

I make a simple experiment:

```python
>>> import torch
>>> import numpy as np
>>> inf = np.array(-np.inf)
>>> inf
array(-inf)
>>> inf = torch.from_numpy(inf)
>>> inf
tensor(-inf, dtype=torch.float64)
>>> inf.half()
tensor(-inf, dtype=torch.float16)
```

It seems `-inf` works well.
In another experiment, I tried the following: 

```python
>>> scores = torch.FloatTensor([1, 2, 3, 4, 4])
>>> mask = torch.ByteTensor([1, 1, 1, 0, 0])
>>> scores.masked_fill_(mask == 0, -np.inf)
tensor([1., 2., 3., -inf, -inf])
>>> scores.half()
tensor([1., 2., 3., -inf, -inf], dtype=torch.float16)
```
It seems that both 2 experiments works well, so I don't get what is the problem to use `-inf` in half precision mode

Thank you

attention mask is -10000.0 for positins that we do not want attention for

it is set to -10000.0 in 

        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
        # masked positions, this operation will create a tensor which is 0.0 for
        # positions we want to attend and -10000.0 for masked positions.
        # Since we are adding it to the raw scores before the softmax, this is
        # effectively the same as removing these entirely.
        
in [get_extended_attention_mask](https://github.com/huggingface/transformers/blob/e95d433d77727a9babadf008dd621a2326d37303/src/transformers/modeling_utils.py#L700)
how to frozen TFGPT2LMHeadModel Embedding matrix?
Author: Orient12
Pull request: no
Labels: name: bug

### System Info

when i use the TFGPT2LMHeadModel structure to train, i want to frozen the embedding matrix, how can i do it?@patil-suraj

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

config = AutoConfig.from_pretrained(pretrained_model_path,vocab_size=VOCAB_SIZE, n_positions=MAX_SEQ_LEN, n_ctx=MAX_SEQ_LEN, n_layer=1,  n_embd=384, initializer_range=0.002)#, initializer_range=0.002
        model = TFGPT2LMHeadModel.from_config(config)
        embedding = np.load(embed_data_path)
        model.set_input_embeddings(embedding)

### Expected behavior

help me to frozen embedding layer

Comments:
Hi @Orient12 👋 As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗

(You can set the `trainable` attribute of a layer to `False`, see [this guide](https://keras.io/guides/transfer_learning/#freezing-layers-understanding-the-trainable-attribute))

> 
Actually, class TFGPT2LMHeadModel model can not split by layer,it just have one layer, so i can't set embedding matrix trainable=False


@Orient12 
Sure you can, you just need to know the right attribute names (for which you might need to dig through the code). The following snippet freezes the embedding layer.

```python
from transformers import TFGPT2LMHeadModel

model = TFGPT2LMHeadModel.from_pretrained("distilgpt2")
print(model.transformer.wte)
print(model.transformer.wte.trainable)

# setting embeddings to not trainable
model.transformer.wte.trainable = False
print(model.transformer.wte.trainable)
```

Our models do not use the Sequential nor the Functional API -- they use the [subclassing](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) method.

Thanks for your help!I have known how to freeze it！

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.