noamaanMulla-03/transformers-issues huggingface dataset:
Replace assertion with ValueError exceptions in run_image_captioning_flax.py
Author: katiele47
Pull request: yes

# What does this PR do?

Replaces 4 `assert` with `ValueError` exeception in run_image_captioning_flax.py.
Co-author: @AdiaWu

Related to [#12789](https://github.com/huggingface/transformers/issues/12789).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks, but the issue was intended for library code, not examples. Let's see if @sanchit-gandhi doesn't mind!

Thank you for your suggestions @sanchit-gandhi! @sgugger ahh I should have known that. I'll update the library code next time.

[Updated] @sanchit-gandhi  After incorporating the suggestions I ran into the failed CI tests below, which I'm not sure how to fix. I assume it has to do with missing some code reformatting. I tried running `make style` on the target folder and received the following output:

<img width="470" alt="Screen Shot 2022-11-22 at 4 05 56 PM" src="https://user-images.githubusercontent.com/54815905/203421123-e0a20022-f62e-4e78-a6dc-903a826cd6cf.png">

I'm also new to open source, but I'll look into this issue in the meantime. Would appreciate any of your inputs! Thanks.

Hey @katiele47, sorry for the late reply! 

Could you check that you've installed Transformers form source? https://huggingface.co/docs/transformers/pr_checks#checks-on-a-pull-request

You can make sure you've installed the dev version of transformers by uninstalling transformers then pip installing from within the transformers repo:
```
pip uninstall transformers
pip install -e .[dev]
```

You should then be able to run `make style` for code quality fixes!

Hello @katiele47, in my case, 

I used 
- `pip install black`
- `black [your filepath]`


and this enabled CircleCi to pass the code quality tests.

**Output**

![reformat output](https://user-images.githubusercontent.com/35699839/204171957-d8659cb6-963b-4f10-ae94-aec44a715817.png)


Thank you so much for reviewing my change @sanchit-gandhi!
[Maskformer] Add MaskFormerSwin backbone
Author: NielsRogge
Pull request: yes

# What does this PR do?

This is part 2 of 3 of the big #20204 PR.

This PR adds MaskFormerSwin to the AutoBackbone API. This ensures that the model can be used as backbone with the MaskFormer framework.

As it makes more sense to move MaskFormerSwin to its own modeling files, this PR implements it in a separate `modeling_maskformer_swin.py` file, along with a configuration implemented in` configuration_maskformer_swin.py`.

To do:

- [x] wait for #20407 to be merged to make the backbone get tested by the common tests, add support for hidden states and attentions

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[FLAX] Add dtype to embedding for bert/bart/opt/t5
Author: merrymercy
Pull request: yes

## What does this PR do?

This PR is the follow-up of #18462. It adds dtype to `nn.Embed` for more common Flax models, including bert, bart, opt, t5, and their copies.
This dtype is necessary for mixed precision training.

## Who can review?
@patrickvonplaten, @LysandreJik, @sanchit-gandhi 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @sanchit-gandhi @younesbelkada @ArthurZucker here

Can we merge this @ArthurZucker ? How to check whether the slow tests pass?

Thanks @merrymercy !
You can run slow tests for `t5` for example by running: `RUN_SLOW=1 pytest tests/models/t5/test_modeling_flax_t5.py` - regarding what has been suggested by @sanchit-gandhi , you can just add a new test that initializes a model let's say in `bf16` and tests if the generated sequence is the same than the one that is expected:
```
    @slow
    def test_small_generation_bf16(self):
        model = FlaxT5ForConditionalGeneration.from_pretrained("t5-small", dtype=jnp.bfloat16)
        EXPECTED_OUTPUT = "XXXX"
        self.assertTrue(model.params["shared"]["embedding"].dtype == jnp.bfloat16)
        model.config.max_length = 8
        model.config.num_beams = 1
        model.config.do_sample = False
        tokenizer = T5Tokenizer.from_pretrained("t5-small")

        input_ids = tokenizer("summarize: Hello there", return_tensors="np").input_ids

        sequences = model.generate(input_ids).sequences

        output_str = tokenizer.batch_decode(sequences, skip_special_tokens=True)[0]
        self.assertTrue(output_str == EXPECTED_OUTPUT)

```

A bf16 test case is added as suggested by @younesbelkada . I checked that slow tests are also passed because most slow tests are in fp32 and this PR does not change the behavior of any fp32 tests.

However, I do notice some slow tests fail on my machine (V100, jax=0.3.25, flax=0.6.2). I think this is not related to my PR, because they fail even with the transformers main branch and transformers v4.24.0. Fixing them is outside the scope of this PR. I can confirm that all tests passed on the main branch can still be passed after my PR.

Since we get two approvals and the test case is added, can we merge it now?

Thanks again for your contribution!
A model or a config like 'transformer_iwslt_de_en' for machine translation
Author: logoutAgain
Pull request: no
Labels: name: New model

# 🌟 New model addition

Does huggingface have some models like `transformer_iwslt_de_en` or `transformer_wmt_en_de`  in fairseq for machine translation? 

I plan to write a model for machine translation on huggingface. It would be great to be able to compare directly with the baseline model on huggingface.

@patil-suraj 

Comments:
I am also eager for a transformer base model to train from scratch with HuggingFace 

In huggingface transformers it's called [FSMT](https://huggingface.co/docs/transformers/model_doc/fsmt), short for FairSeq Machine Translation.

I have the same need, looking for a transformer base model.
Will try FSMT.
Thanks!
Add LongT5 to AutoConfig
Author: robbohua
Pull request: no

### System Info

config = AutoConfig.from_pretrained("google/long-t5-local-base")

gives a KeyError: 'longt5'

I am trying to run this as it's part of SentenceTransformer package when initialising a search model - could longT5 be added to AutoConfig?



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

config = AutoConfig.from_pretrained("google/long-t5-local-base")

### Expected behavior

The config is loaded

Comments:
Hey @robbohua -- I can run the script you shared without problems on my end. What version of `transformers` are you using? :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix Swin ONNX export warnings
Author: fxmarty
Pull request: yes

# What does this PR do?

This PR fixes uninformative warnings in the ONNX export for Swin as reported in https://github.com/huggingface/transformers/issues/19780 , namely

```
UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently
rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use
torch.div(a, b, rounding_mode='floor').
```

Since  `torch.jit.trace` assumes expressions like `tensor.size(0)`, `tensor.size()[1]`, `tensor.shape[2]` are tensors in tracing mode ([reference](https://ppwwyyxx.com/blog/2022/TorchScript-Tracing-vs-Scripting/)), the warning is raised.

## Before submitting
- [x] This PR fixes a typo

## Who can review?

@lewtun 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Note that this will only work in recent versions of PyTorch. As was done for the [int div](https://github.com/huggingface/transformers/blob/321ef388fe041f630e65abc26a3be8580d7e858b/src/transformers/pytorch_utils.py#LL35C3-L35C3), you should create a util function in the PyTorch utils that works across PyTorch versions

Good to know, thanks a lot! I think there are other models having the issue of using `torch.div` then, I'll fix as well. I'm a bit skeptic about readability (including this PR), trying to support tracing overall makes the code less readable I feel.

Just realized that the warning above is not shown anymore in PyTorch 1.13. Seem to be https://github.com/pytorch/pytorch/pull/78411

Refer to the release notes: `Updated torch.floor_divide to perform floor division` https://github.com/pytorch/pytorch/releases

This is fine since it influences only negative numbers. I'll close this PR.
Whisper: incorrect list of non speech tokens
Author: guillaumekln
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The lists `NON_SPEECH_TOKENS` and `NON_SPEECH_TOKENS_MULTI` contain the tokens 6 and 12 that are not suppressed by default in the [reference implementation](https://github.com/openai/whisper/).

Consider the following example using the reference `whisper` module:

```python
import transformers
from whisper.tokenizer import get_tokenizer

tokenizer = get_tokenizer(multilingual=True, task="transcribe", language="fr")

suppress_tokens = list(
    sorted(
        tokenizer.non_speech_tokens
        + (tokenizer.sot, tokenizer.sot_prev, tokenizer.sot_lm, tokenizer.no_speech)
    )
)

config = transformers.WhisperConfig.from_pretrained("openai/whisper-tiny")
print(suppress_tokens == config.suppress_tokens)  # prints False

config.suppress_tokens.remove(6)
config.suppress_tokens.remove(12)
print(suppress_tokens == config.suppress_tokens)  # prints True
```

### Expected behavior

The list of suppressed tokens should match the reference implementation.

Comments:
Hey! thanks for pointing that out. 
Will open a PR on the online models and on the repo, just gotta make sure this is not backward incompatible 🤗 

Thanks for the fix and the configurations update! However, there are still 2 pull requests to merge:

* https://huggingface.co/openai/whisper-small/discussions/4
* https://huggingface.co/openai/whisper-medium/discussions/5



Thanks for the notice! 🤗
Fix torch meshgrid warnings
Author: fxmarty
Pull request: yes

# What does this PR do?

This PR fixes unwanted warnings due in `torch.meshgrid`:

```
/home/fxmarty/anaconda3/envs/hf-inf/lib/python3.9/site-packages/torch/functional.py:478:
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing
argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2894.)
  >>> ys = torch.linspace(-5, 5, steps=100)
```

Passing `indexing="ij"` is necessary to keep the current behavior, see https://pytorch.org/docs/stable/generated/torch.meshgrid.html and https://github.com/pytorch/pytorch/issues/50276

```python
import torch

x = torch.tensor([1, 2, 3])
y = torch.tensor([4, 5, 6])

grid_x, grid_y = torch.meshgrid(x, y, indexing="ij")  # vs indexing="xy"

print(grid_x)
print(grid_y)
```

## Before submitting
- [x] This PR fixes a typo

## Who can review?

any core maintainer to approve


Comments:
From a quick search in the PyTorch doc, this argument is only accepted starting in PyTorch 1.10, so this PR will break the corresponding models for older versions. You should add a `meshgrid` function the pytorch utils that passes the argument or not depending on the PyTorch version, then use this util in the modeling code.

_The documentation is not available anymore as the PR was closed or merged._

@sgugger Thank you, hopefully fixed!

edit: will fix the CI tomorrow

@sgugger the CI is good at last!

Arf my comment from yesterday never went through :sweat_smile:
How to reproduce the machine translation experiments in Attention is all you need.
Author: shizhediao
Pull request: no

### Feature request

I want to reproduce the experiments described in _Attention is all you need_, which is a transformer base model from scratch. The model architecture is the same as _Attention is all you need_.
In other words, I am looking for a transformer base model to train from scratch with HuggingFace.



### Motivation

Reproduce the Transformer experiments in machine translation. I found that there are many pre-trained models (e.g., T5, BART, MariaMT), but I would like to train a transformer base model from scratch to compare different optimizers during pre-training.

### Your contribution

I found a previous issue here but not sure whether it is the right way to go. https://github.com/huggingface/transformers/issues/12386

Comments:
Hi,
I am trying to reproduce the performance of transformer-base (from attention is all you need) on WMT14.
I am using `FSMT` because I cannot find an implementation of the transformer.
I was wondering which dataset and tokenizer are the best choices. 
1. `stas/wmt14-en-de-pre-processed` with `facebook/wmt19-en-de`
2. `wmt14` with `facebook/wmt19-en-de`
Especially, I do not know which tokenizer should be used.

Thanks in advance if you could provide some suggestions!
@patil-suraj  @patrickvonplaten 

Please use the [forums](https://discuss.huggingface.co/) to discuss such questions as we keep issues for bugs and feature requests only.

> Please use the [forums](https://discuss.huggingface.co/) to discuss such questions as we keep issues for bugs and feature requests only.

OK, sorry about this.
extract warnings in GH workflows
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR uses the change in #20474 to collect the warnings in our GH scheduled daily CI, and provide a button in the slack reports to access this information.

<img width="493" alt="image" src="https://user-images.githubusercontent.com/2521628/204501762-1ed46a7c-4a91-40ba-9b16-3288168f0dfc.png">


Comments:
_The documentation is not available anymore as the PR was closed or merged._
fixed small typo
Author: sandeepgadhwal
Pull request: yes

# What does this PR do?
This PR fixes a small typo in VAN model.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@NielsRogge 


Models:

- van


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix documentation code to import facebook/detr-resnet-50 model
Author: JuanFKurucz
Pull request: yes

# What does this PR do?

Changes example import line 

`>>> model = DetrForObjectDetection.from_pretrained("facebook/resnet-50")` 

to 

`>>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")` 

As trying to import `"facebook/resnet-50"` raises:

```
OSError: facebook/resnet-50 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make `add_special_tokens` more clear
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix tokenization. The short term goal is to fix #20418 (well, make things more clear)

The long term goal is to be able to have a generic fix for #20401, but the tokenizer thing is somehow complicated, and I can only fix step by step.




Comments:
_The documentation is not available anymore as the PR was closed or merged._
[CI, WHISPER] fix the latest failing test 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
In a recent update, we followed the original code which changed some of the suppress tokens for better performances. This lead to a small change in the output of on particular case. Tested with the original code and we have the correct output now!

See [here](https://huggingface.co/openai/whisper-large/commit/ed97120f929257fb801f99587ada69be0daf5b0a) for the particular commit

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix TF nightly tests
Author: Rocketknight1
Pull request: yes

This PR fixes two issues with the TF tests:

1) `test_saved_model_creation` failed sometimes because the dict being passed to the saved model didn't match the inputs it was traced/compiled with. This should be fixed now.

2) Some of the tests for the new `TFGPT2Tokenizer` (cc @piEsposito) were using `is_tensorflow_text_available` or `requires_tensorflow_text`, but `TFGPT2Tokenizer` actually depends on `keras-nlp`. I made sure the requirements were changed and that `is_keras_nlp_available` is importable from the root.

Comments:
Don't worry, it was a really small fix! Just making sure you saw this so you didn't get confused about why your code was being changed.

_The documentation is not available anymore as the PR was closed or merged._
Update doc examples feature extractor -> image processor
Author: amyeroberts
Pull request: yes

# What does this PR do?

Replaces vision feature extractor references with image processors throughout transformers documentation. 

Places where changes didn't happen: 
* Vision models which use a `Processor` class. Some processor classes still have `feature_extractor_class` property, to be removed in future. 
* `examples/...` - required changes to code outside to scope of this PR and dependant on some changes to `Processor` class

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[CORE] Use model prefix instead of cls
Author: ArthurZucker
Pull request: yes
Labels: name: Core: Modeling

# What does this PR do?
This adresses a very particular bug found when `base_model_prefix` is specific to the instance. 
The `JukeboxPrior` are defined by their level of generation. Thus to each level, the corresponding base model prefix : `priors.0`. 
When load the checkpoints from a pretrained `JukeboxModel`, the `_load_pretrained_model` uses the `cls.base_model_prefix` while the `model.base_model_prefix` is also always available. This means that the weights will [not be properly loaded](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L472), but it will be silent. 
This should adresse any current and futur model loading issues where the `base_model_prefix` is modified per instance. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @ArthurZucker 

For this issue and the fix in the PR, it's would be very nice if you could provide a code snippet to demonstrate what is currently wrong to justify the fix. It will also help future developers (either inside HF or external contributors) to understand it much better and faster (if they ever track back to this PR for some reason).

Thank you, looking forward for it!



Discussed internally with @ArthurZucker. I am very much against using a "dynamic" attribute instead of the class attribute.

Yep, closing this! 
Snipper is impossible to provide as it is very specific to jukebox and an invisible bug! 

Since you close the PR, I would not ask the code snippet. It's still somehow strange that it is impossible to give a code snippet. From the description, it looks like if we have a checkpoint, when loading it, we will get some weights not being loaded correctly. 

One possible way is to create a model, save it, and reload it. Then point out which weights are not loaded correctly.
I might miss many details here, and things may not be so easy. But if you are able to find the invisible bug, it's no longer invisible @ArthurZucker .

And if it is completely different thing than I imagine, just ignore me :-)

No, the weights are not loaded correctly but the error is silent. Here is a snippet (but nothing will be outputed) 

```python 
>>> from transformers import JukeboxPrior, JukeboxModel
>>> model = JukeboxModel.from_pretrained("openai/jukebox-5b-lyrics").priors[0]
>>> prior = JukeboxPrior.from_pretrained("openai/jukebox-5b-lyrics")
>>> assert model.encoder.start_token == prior.encoder.start_token
```

There will be no `missing` or `unexpected keys` but the weights will not be loaded

OK, it's silent, but you speak for it so now it's clear 💯 !
Repurpose torchdynamo training args towards torch._dynamo
Author: sgugger
Pull request: yes

# What does this PR do?

This PR re-uses the current `torchdynamo` training argument to be compatible with the internal module of PyTorch (in the nightlies). This is slightly breaking but at the same time the torchdynamo package has migrated to PyTorch proper, and the integration was marked as experimental.

The "fx2trt-fp16" backend is not advertised by PyTorch, so I removed it.

Comments:
_The documentation is not available anymore as the PR was closed or merged._