noamaanMulla-03/transformers-issues huggingface dataset:
Add missing tokenizer tests - RemBert
Author: IMvision12
Pull request: yes

# What does this PR do?

Added tokenizer tests for Rembert. I took reference from `test_tokenization_camembert.py`

@SaulLu @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for adding this! Can you first rebase your branch on main? TensorFlow new release broke a lot of things so tests won't pass unless you do this :-)

Looks like something went wrong and GitHub adds a lot of diff. If force-pushing doesn't solve the issue, you might need to close this PR and open a fresh one.
Update Special Language Tokens for PLBART
Author: jordiclive
Pull request: yes

# What does this PR do?

This PR fixes the special tokens for PLBartTokenizer, raised in Issue #19505. Previously the tokenizer treats java, python etc. as special tokens, and removes them when decoding is performed.

@LysandreJik

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @ArthurZucker 

Hey! Great work 👍 
We should make sure the CI tests are all green, and could you add a new test like `test_special_code_tokenization` where we make sure that the expected behavior of #19505 works

@ArthurZucker ok, tests are green. I added to the `test_full_multi_tokenizer,` `test_full_base_tokenizer` tests to check for this behaviour as the test tokenizer is already loaded in these tests. 

@ArthurZucker bumping this. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19980). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19980). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19980). All of your documentation changes will be reflected on that endpoint.

Thanks, @ArthurZucker! I made some changes to the tests with intermediate inputs and accepted several changes. 

> calling `_convert_lang_code_special_format` in the `src_lang.setter` to avoid calling it everywhere / avoid one liner functions

With this, wouldn't the `src_lang would.setter` have to be called everywhere instead? 
I think its quite difficult to make it backward compat and simpler as you suggest, as there are lots of places the user provides the src_lang, tgt_lang and there is also `self._src_lang` as well as` self.src_lang`. This at least preserves the functionality as before and the mapping as under-the-hood.

@ArthurZucker made that readability change. can we merge?
Add Audio Spectogram Transformer
Author: NielsRogge
Pull request: yes

# What does this PR do?

Fixes #16383 

This PR adds the [Audio Spectogram Transformer (AST)](https://arxiv.org/abs/2104.01778) model from MIT.

Similar to Whisper (actually prior to Whisper), the model treats audio as an image and applies a Vision Transformer on it.

The model gets SOTA results on audio classification benchmarks. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey, how can I use it already? I installed the branch, but unsure how to load the model. I'm new with huggingface :D

Hey @FrankFundel - hoping @NielsRogge adds a nice example as part of this PR documenting just that 🤞 In the mean time, you can try adapting the example from https://huggingface.co/docs/transformers/tasks/audio_classification

You'll need to change the repo names from `facebook/wav2vec2-base` to the appropriate Audio Spectrogram Transformer repo name. You'll also need to change the preprocess function (https://huggingface.co/docs/transformers/tasks/audio_classification#preprocess) to something like:

```python
def preprocess_function(examples):
    audio_arrays = [x["array"] for x in examples["audio"]]
    input_features = feature_extractor(audio_array, sampling_rate=feature_extractor.sampling_rate)
    return input_features
```
This is all currently untested, so might require some playing around to make it work.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19981). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19981). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19981). All of your documentation changes will be reflected on that endpoint.
Add inference section to task guides
Author: stevhliu
Pull request: yes
Labels: name: Documentation

Currently, the task guides only show how to finetune a model, but it doesn't directly connect the dots to how you can use that model for inference. For completeness, this PR adds a section to the task guides to show how to use a model for inference after finetuning. This gives users a better overview of the model lifecycle.

In doing so, when we update `task_summary.mdx`, we can focus less on the practical steps of how to use a model for inference (we can add links to the task guides) and instead discuss the more theoretical aspects of these tasks as intended by the Conceptual Guide section.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the suggestions! I reworked the `sequence_classification` task a bit, and if we like the changes, then I can apply them to the other tasks. Main changes are:

- Encourage users to login with their HF accounts so they can push their finetuned models. Along these lines, I've added `push_to_hub` and the `PushToHubCallback`.
- Added a section to include a `compute_metrics` function so users can evaluate their models during training.

I think adding these two will help the task guides be more complete :)
Add Spanish translation of serialization.mdx
Author: donelianc
Pull request: yes

# What does this PR do?

Add the Spanish translation for `serialization.mdx` as part of the #15947 issue.
Changes include the Spanish version of the original document and the updated `_toctree.yml` file.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests) Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://github.com/huggingface/transformers/issues/15947#issuecomment-1312741421)**.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@omarespejel or @osanseviero, can you help me review this PR? Thanks!

@osanseviero, I addressed your comments in my last commit. Thanks for your review. I'll consider your feedback for future translations.

@sgugger, review done :) Please merge when possible. Thanks!
Missing sections in Spanish documentation
Author: donelianc
Pull request: no

**Expected output**: Nested sections with proper documents in Spanish translation.
**Current output**: Some sections are missing (but docs are present), so documents are not correctly organized

-----

After contributing to #15947, I noticed `_toctree.yml` for the Spanish translation is not following the proper order as the original documentation.

For example, there is no `General Usage` section in the Spanish version, but the `Create a custom architecture` ([create_a_model](https://huggingface.co/docs/transformers/v4.24.0/es/create_a_model)) document Is present. 

Contributions to #15947 missed adding the right sections updating `_toctree.yml`.

Comments:
Hi @sgugger. Maybe you can help me to verify this issue I just opened. I attached a PR to fix it (you might also be the right person to review and approve it). Thanks!
Fix toctree for Section 3 in Spanish Documentation
Author: donelianc
Pull request: yes

# What does this PR do?
Fixes #20359 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests) Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)?
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Raised specific types of exceptions on distilbert model 
Author: JuheonChu
Pull request: yes

Co-author: Batese @batese2001
Local test file: ![testDistilbert](https://user-images.githubusercontent.com/35699839/203268440-1b6a01e1-bc25-4a71-a59b-3af64175fe53.png)



Comments:
_The documentation is not available anymore as the PR was closed or merged._

I made a different PR with the same request here: https://github.com/huggingface/transformers/pull/20375
support `t5` for `text-generation` pipeline
Author: younesbelkada
Pull request: yes

# What does this PR do?

A tentative to support `T5` for `text-generation` pipeline. I don't really expect this PR to get merged as it is very hacky and IMO not a good idea to support `T5` for `text-generation` but I would love to have some insights on what we can potentially do to support `text-generation` pipeline for `T5`

Probably the fix would be also to implement `T5ForCausalLM` but not sure if this makes sense!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: shorter XLA contrastive search tests
Author: gante
Pull request: yes

# What does this PR do?

Makes XLA contrastive search tests shorter (in terms of tokens), to avoid flaky tests.

This is due to our recently failing CI for some models. The XLA code path passes the test with XLA compilation off -- i.e. the XLA code path returns the same as the non-XLA code path. However, with XLA compilation on, there is a chance of obtaining different results. I couldn't pinpoint the issue, but there is a possible explanation.

This may be due to numerical stability issues: contrastive search takes the maximum of a cosine distance between two hidden states [built from randomly initialized weights] as a penalty to the logits, which combined with the logits' low values [because the test model was untrained] could explain the mismatch.

👉 In any case, I was already planning on reinforcing contrastive search XLA testing with real examples on key models, like T5 and OPT.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

And as always, if a one-line short comment in the code could keep your explanation more visible for other (HF) developers, don't hesitate.
Revert `keys_to_ignore` for M2M100
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR reverts a change that has been made on `M2M100Model` , to reproduce you can run: 
```
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

src_lang = "eng_Latn"
tgt_lang = "spa_Latn"

model_id = "facebook/nllb-200-3.3B"

tokenizer = AutoTokenizer.from_pretrained(model_id, src_lang=src_lang)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
```
on `main`, users get: 
```
│ /home/younes_huggingface_co/debug_issues/code/transformers/src/transformers/modeling_utils.py:24 │
│ 59 in _load_pretrained_model                                                                     │
│                                                                                                  │
│   2456 │   │   │   for key in missing_keys:                                                      │
│   2457 │   │   │   │   if key.startswith(prefix):                                                │
│   2458 │   │   │   │   │   key = ".".join(key.split(".")[1:])                                    │
│ ❱ 2459 │   │   │   │   param = model_state_dict[key]                                             │
│   2460 │   │   │   │   if param.device == torch.device("meta"):                                  │
│   2461 │   │   │   │   │   if not load_in_8bit:                                                  │
│   2462 │   │   │   │   │   │   set_module_tensor_to_device(model, key, "cpu", torch.empty(*para  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'encoder.embed_positions.weights'
```

cc @Narsil 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20381). All of your documentation changes will be reflected on that endpoint.
Add `accelerate` support for `ESM`
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `accelerate` support for `ESM` models, so that the largest `ESM` models (15B params) can be loaded in 8-bit, therefore easing accessibility for large protein models. This also introduces the first protein model that can be loaded in 8bit.
```
# pip install accelerate bitsandbytes
from transformers import AutoModel

model = AutoModel.from_pretrained("acebook/esm2_t48_15B_UR50D", device_map="auto", load_in_8bit=True)
```

cc @sgugger @Rocketknight1 

slow tests pass

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Doctest] Add configuartion_longformer.py
Author: AShreyam
Pull request: yes

# What does this PR do?
Fixes # 19487

Add configuration_longformer.py to utils/documentation_tests.txt for doctest.

Based on issue https://github.com/huggingface/transformers/issues/19487

@sgugger could you take a look at it?
Thanks :)

Comments:
Hi @AShreyam

This PR is not ready to merge. It contains other changes that is not for this config doctest sprint.


Hi @AShreyam

This PR is not ready to merge. It contains other changes that is not for this config doctest sprint.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix nightly runs
Author: sgugger
Pull request: yes

# What does this PR do?

The pipeline that runs the nightly tests exits with `Unexpected argument(s): nightly` instead of running all tests. I think we just need to add it in the custom config as an argument (unused) so circleCI doesn't complain anymore.

Manually triggered the tests on this branch with `nightly` at `true` and it ran all tests (see workflow nightly in the tests).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Indicate better minimal version of PyTorch in big model inference
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out in https://github.com/huggingface/accelerate/issues/880, the minimum version when using a `device_map` is not PyTorch 1.9 but PyTorch 1.11. This PR fixes that.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[ESM] fix `accelerate` tests for esmfold
Author: younesbelkada
Pull request: yes

# What does this PR do?

FIxes slow tests that were not passing for `ESM`. In fact I was running `RUN_SLOW=1 pytest tests/models/esm/test_modeling_esm.py` and forgot to run `RUN_SLOW=1 pytest tests/models/esm/test_modeling_esmfold.py`

cc @sgugger @Rocketknight1 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
AutoImageProcessor
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds the `AutoImageProcessor` class and makes model image processors available to import.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20111). All of your documentation changes will be reflected on that endpoint.

Still reviewing, but in `processing_auto.py/from_pretrained`, the beginning part

https://github.com/huggingface/transformers/blob/1ebc7bb995c5e43961a7c8079ca3bf29f06f2411/src/transformers/models/auto/processing_auto.py#L197

around this, there is no `ImageProcessingMixin`. I feel this is a miss and should appear here?



> Still reviewing, but in `processing_auto.py/from_pretrained`, the beginning part
> 
> https://github.com/huggingface/transformers/blob/1ebc7bb995c5e43961a7c8079ca3bf29f06f2411/src/transformers/models/auto/processing_auto.py#L197
> 
> around this, there is no `ImageProcessingMixin`. I feel this is a miss and should appear here?

@ydshieh Yes, you're right. I've added a check now [here](https://github.com/amyeroberts/transformers/blob/2e08d16f7758889fe0cb203091d292c968f067b0/src/transformers/models/auto/processing_auto.py#L194). Can you confirm if this matches with what you think should have been added? 

> Can you confirm if this matches with what you think should have been added?

Yes!



One comment (no need to be done in this PR): I think it would be great if we can remove the `feature_extractor_type` key after loading the image processor.

```python
from transformers import CLIPModel, AutoProcessor, CLIPProcessor, CLIPImageProcessor, CLIPFeatureExtractor, AutoImageProcessor

p = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
print(p.feature_extractor_type)
p.save_pretrained("temp-clip")
```
gives
```bash
CLIPFeatureExtractor
```
on the terminal,  and in the output file `preprocessor_config.json`, we have

```python
  "feature_extractor_type": "CLIPFeatureExtractor",
  "image_processor_type": "CLIPImageProcessor",
```
MAE Noise Was NOT Removed In Prediction For TFViTMAEModel / ViTMAEModel
Author: zhoutong-fu
Pull request: no

### System Info

transformers version: v4.24.0 (latest)


### Who can help?

@NielsRogge 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import numpy as np
from transformers import AutoFeatureExtractor, TFViTMAEModel
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-mae-base")
model = TFViTMAEModel.from_pretrained("facebook/vit-mae-base")
inputs = feature_extractor(images=image, return_tensors="tf")
outputs1 = model(**inputs)
outputs2 = model(**inputs)
res1 = outputs1.last_hidden_state.numpy()
res2 = outputs2.last_hidden_state.numpy()
np.allclose(res1, res2, rtol=1e-04, atol=1e-05)  # This returns False as noises are introduced.
```

### Expected behavior

It should return True.

It returns False as noises are added in `random_masking()` which hasn't been removed for prediction. There are no options to do that.

Comments:
Hi,

ViTMAE generates a random mask internally to mask out patches. Hence forwarding the same inputs twice through the model will result in different hidden states. If you want to ensure reproducability, you can pass the `noise` argument. See [here](https://github.com/huggingface/transformers/blob/8503cc755050c6ed5bc771e3244c29b71be1841e/tests/models/vit_mae/test_modeling_vit_mae.py#L317) for an example.

Thanks @NielsRogge for the reply. I'm using this model to extract image features and I'm using some hacks to remove noise and masks for inference.

After reading the [ViTMAE code](https://github.com/facebookresearch/mae/blob/main/FINETUNE.md), I found that the trained ViTMAE model can be loaded as a regular ViT model directly.  Just tested with `TFViTMAEModel.from_pretrained("facebook/vit-mae-base")` and it works. Could you help confirm is this the correct approach to use pre-trained MAE model? 

Yes, that's also shown in the docs. Note that `TFViTMAEModel.from_pretrained("facebook/vit-mae-base")` will just load the base Encoder, which does not include the decoder used during pre-training. For that you would need to load `TFViTMAEForPreTraining`.

However, since you want to use the model for image features, it's indeed adviced to just load the Transformer encoder.
Fix doctest file path issue
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix doctest file path issue.

(Currently, the whole suite fails from the beginning)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20400). All of your documentation changes will be reflected on that endpoint.