noamaanMulla-03/transformers-issues huggingface dataset:
[Swin2SR] Add doc tests
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds Swin2SR to the doc tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yeah I remember a discussion with @patrickvonplaten and @sanchit-gandhi, I used `Swin2SRImageProcessor` here instead of the Auto class to make it more explicit. 

But happy to change. For me, the Auto API is handy when a model doesn't have its own preprocessing class, and for usage in the pipelines.

Yes I'm with @LysandreJik here! The conclusion was to move towards `AutoProcessor`/`AutoTokenizer` in the docs (_c.f._ the compelling argument from @LysandreJik https://huggingface.slack.com/archives/C01N44FJDHT/p1667824904971239?thread_ts=1667816128.702279&cid=C01N44FJDHT).
[Examples] Update big table
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR updates the "big table of tasks" by

- adding a hyperlink to each of the example datasets
- add "image pretraining", and the Colab link for semantic segmentation

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Slightly related, wondering if we shouldn't link from this Big Table to the corresponding task in hf.co/tasks? (cc @merveenoyan)
Use `config.num_channels` in CLIP-like modeling files
Author: ydshieh
Pull request: yes

# What does this PR do?

`config.num_channels` is not used in some CLIP-like modeling files. Unlike previous PRs like #20596 or #20844, we use this attribute in the modeling files in this PR. 

The only breaking case is when a user previously set `config.num_channels=X` with `X !=3`, which is super unlikely IMO. (Even they did so, the actual Conv2D layer still uses `3` as it is hard-coded in the current `main` branch) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
A (possible) bug of sentence permuation in BART pretraining script
Author: StevenTang1998
Pull request: no

### System Info

None

### Who can help?

@sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

As described in the [original BART paper](https://arxiv.org/pdf/1910.13461.pdf), the sentence permutation should be:  A document is divided into sentences based on **full stops**, and these sentences are shuffled in a random order.

However, in the [examples/flax/language-modeling](https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_bart_dlm_flax.py#L318), it uses pad token as the `end_sentence_mask`. 


Comments:
Hey @StevenTang1998!

Great question! In our Flax pre-training script, use the PAD token as the end of sentence indicator:
https://github.com/huggingface/transformers/blob/76924384af6081e58460231c3c637f9c83cabf97/examples/flax/language-modeling/run_bart_dlm_flax.py#L626-L627

This way, the PAD token is appended after the end of a sentence (full stop, exclamation marks, question marks etc) and used to indicate sentence boundaries. So in splitting a document by the PAD token, we are in effect splitting sentences on end of sentence punctuation (equivalent to the original paper!).

Hope this addresses your concerns! Let me know if you have any questions and I'd be happy to answer 🤗

Thanks for kind reply! I understand it now!

But I have a concern that the PAD token may affect the results (through position embedding?) although it will not be attend to.

Hey @StevenTang1998! Glad to hear that clarified things! It should be fine since we build the attention mask based on the position of the PAD token ids :)

OK, thank you for your reply! Wish you a good day!
min_new_tokens option in generate() implementation
Author: gonced8
Pull request: no

### Feature request

Similarly to the `max_new_tokens`, a `min_new_tokens` option would count only the newly generated tokens, ignoring the tokens of the input sequence (prompt) in decoder only models.

### Motivation

The option `min_length` of the `generate()` method might be ambiguous for decoder only models. It is not clear if decoder only models consider the length of the input (prompt) for the `min_length` condition or only the newly generated tokens.
In Encoder Decoder (seq2seq) it is clear though.

### Your contribution

Not that I remember. But I could test it.

Comments:
cc @gante 

Hi @gonced8 👋  Thank you for raising this issue!

This is the same as [this issue](https://github.com/huggingface/transformers/issues/20614) (which is slightly older). I'm closing this issue to avoid duplication of comments/efforts, and [this particular comment](https://github.com/huggingface/transformers/issues/20614#issuecomment-1361225567) might be of your interest :)
Fix past CI by Skipping `LevitModelTest.test_problem_types`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix past CI by Skipping `LevitModelTest.test_problem_types` for `PyTorch 1.9`.

This test failed with torch 1.9 with some CUDA error, but it passes with `torch 1.8` and `torch >= 1.10`.

The error is
```bash
input = <[RuntimeError('CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously repor...incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.') raised in repr()] Tensor object at 0x7fcb37dbd900>
weight = <[RuntimeError('CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously repor...orrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.') raised in repr()] Parameter object at 0x7fcb36c935c0>, bias = None

FAILED tests/models/levit/test_modeling_levit.py::LevitModelTest::test_problem_types - RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix whisper export
Author: mht-sharma
Pull request: yes

# What does this PR do?

Fix the export for the whisper model

The current export for whisper fails with error `Invalid Feed Input Name:past_key_values.3.encoder.value`, because the cross attention key values are not exported as input in the ONNX model after the new condition introduced in the [transformers@97a51](https://github.com/huggingface/transformers/commit/97a51b0c7d483cdf13ea878a987f9aa1c9eecc91).

The error occurs due to incorrect dummy input generation for cross attention key values for export. The PR fixes the same.

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@lewtun 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @ArthurZucker 

Hey, would you mind adding a bit more context? What is the issue related to current export? 

> Hey, would you mind adding a bit more context? What is the issue related to current export?

Hi @ArthurZucker I have updated the PR description.

@ArthurZucker could you please merge this. I do not have the permissions. Thanks!
Add-warning-tokenizer
Author: ArthurZucker
Pull request: yes
Labels: name: Fast Tokenizers

# What does this PR do?
Adds a warning when the user wants to use a fast tokenizer but it doesn't exist. Should help with #20817

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Should I just change the argument? 
I'm in favor of raising an issue rather than a warning here, WDYT? 

No I was talking about the docstring, but this is actually addressed by another PR. We can't suddenly raise an error for this behavior as it would be breaking.
Update `HubertModelIntegrationTest.test_inference_keyword_spotting`
Author: ydshieh
Pull request: yes

# What does this PR do?

Our CI is updated to use torch 1.13 (which also use cuda 11.6) instead of torch 1.12 (cuda 11.3), this test fails. The tolerance `2e-2` is not enough anymore, but it passes with `3e-2`.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[mBART] fix erroneous italics in docstring
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Corrects tensor dims from italics to code-blocks in the mBART doctoring, as discussed in https://github.com/huggingface/transformers/pull/20787#discussion_r1050620673.

The changes as applied to mBART are contained in https://github.com/huggingface/transformers/pull/20835/commits/284e13af61871056f64cbfb7883ea36a4bc70a39. The changes as applied to all other models that inherit using `# Copied from MBart...` are in https://github.com/huggingface/transformers/pull/20835/commits/0a5cd7c66f3be8d8f21dbdb8065cc1d87bd6f405


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts and @NielsRogge
- speech models: @sanchit-gandhi

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger and @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Stop calling expand_1d on newer TF versions
Author: Rocketknight1
Pull request: yes

Tensorflow changed their default `train_step` in version 2.11 to no longer user `data_adapter.expand_1d`, and also deleted that method. Since we copied that code for our train step, this made our `train_step` stop working in 2.11 when the user was using a non-dummy loss!

This PR resolves the issue by not calling `expand_1d` for TF versions >= 2.11.

Fixes #20750 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: post-generate config doctest fix
Author: gante
Pull request: yes

# What does this PR do?

Fixes doctests that were broken as a result of the `generation_config` PR merge.

Note: the failing pipeline test was fixing by adding a missing field to `gpt2`'s `generate_config.json` (which was created before these recent `generation_config` changes). See [this hub commit](https://huggingface.co/gpt2/commit/e7da7f221d5bf496a48136c0cd264e630fe9fcc8).

Comments:
Just a question, so if users do things in the old way like

```
model.config.pad_token_id = model.config.eos_token_id
```
they might get different results before/after the generation config PR ..?

_The documentation is not available anymore as the PR was closed or merged._

> Just a question, so if users do things in the old way like
> 
> ```
> model.config.pad_token_id = model.config.eos_token_id
> ```
> 
> they might get different results before/after the generation config PR ..?

`generate()` supports control from ad hoc model config changes (it has an extra check and handles differences [here](https://github.com/huggingface/transformers/blob/26dd041c6e45379141302e2d293ab4cd9cf805d4/src/transformers/generation/utils.py#L1131)), for retrocompatibility and ease of use. It is deprecated and will be removed soon.

Individual generation methods do not have this check, so they are not supporting control from ad hoc model config changes. The doctests in `main` are failing for this reason. It means that fixing the doctests can be done in two ways:
1. Add the same check to all individual generation methods
2. [current implementation in the PR] Change the doctest itself so they don't rely on ad hoc model config changes

I decided to follow 2. since we are deprecating it soon anyways AND calling the methods directly is an advanced feature (users should not be relying on side effects from the model config to support advanced functionality, it's a recipe for disaster 👀 ). 

WDYT? (cc @sgugger)

Agreed with option 2!
Add Onnx Config for PoolFormer
Author: BakingBrains
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (https://github.com/huggingface/transformers/issues/16308)


Add changes to make PoolFormer models available for Onnx conversion.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ChainYo




Comments:
I have ran 
```RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -k "poolformer"```

![tt](https://user-images.githubusercontent.com/51019420/198833650-39a4aa51-fb61-41fa-828d-d948e5942ad7.jpg)


Conversion output

![tt2](https://user-images.githubusercontent.com/51019420/198833670-9728dfdc-4bdd-461e-958e-2350bd6923a5.jpg)


@ChainYo @lewtun Any suggestions here?

Thanks and Regards.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19962). All of your documentation changes will be reflected on that endpoint.

Any updates on this PR?

@ChainYo Any updates here?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @lewtun and @michaelbenayoun, we should merge this before the Optimum change. 

Could you help on this?

Hi @ChainYo,
The change is already here, I think the PR can be merged once the conflicts are resolved.
Also @BakingBrains could you add it to Optimum as well? It should not require much effort. If not, I can make it myself.

@michaelbenayoun Sure, I will add it to Optimum. 

Thank you

I tried to resolve the conflicts, but I think I messed up

I just reopened a new pull request for the same with resolved conflicts, can you please check @michaelbenayoun 
https://github.com/huggingface/transformers/pull/20868

Thank you
change strings to f-strings in image_processing_utils.py
Author: dhansmair
Pull request: yes

# What does this PR do?
not top priority, it's just that there's a python string which should be a f-string :)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts


Comments:
_The documentation is not available anymore as the PR was closed or merged._
fsdp fix
Author: pacman100
Pull request: yes

# What does this PR do?
1. FSDP fix: Fixes https://github.com/huggingface/transformers/issues/18767. If this argument is not specified and ``module`` is on CPU, FSDP issues a warning mentioning that this argument can be specified for faster initialization.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for the fix! Can you expand a bit the description so that users going back to your PR understand what's happening?

Done, Thanks!
FSTM compatibility issues with other `ForConditionalGeneration` models
Author: gsarti
Pull request: no

### Description

The implementation of FSMT models like [`facebook/wmt19-en-de`](https://huggingface.co/facebook/wmt19-en-de) is atypical with respect to different aspects that are normally supported in other `ForConditionalGeneration` models. In particular:

- Both `FSMTModel` and `FSMTForConditionalGeneration` lack utility methods like `get_encoder` and `get_decoder`
- `FSMTForConditionalGeneration` and all its subclasses do not accept `inputs_embeds` and `decoder_inputs_embeds` as possible alternatives to `input_ids` and `decoder_input_ids` for the forward pass.

In particular, this renders the model unusable when using feature attribution methods through the `inseq` library (see related issue inseq-team/inseq#153).

### Expected behavior

We would expect all models belonging to the same family to expose a consistent API for external usage.

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

### Who can help?

@ArthurZucker @younesbelkada 

Comments:
distributed training
Author: liumaishen
Pull request: no

why the training args n_gpu is set to 1 when use Trainer's distributed training？
which means only one device is allowed in a node？
and the training parameters printed is calculated with the n_gpus=1.
I want to know what should i do when every node has multi gpus. 

Comments:
Please use the [forums](https://discuss.huggingface.co/) to ask such questions as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`MobileNet-v2`] Fix ONNX typo 
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/20856

In order to get which model is exportable with ONNX, we need first to pre-process the model type before checking with `FeaturesManager` by replacing the `_` with `-`. 

This has been forgotten for `mobilenet` family models which leads to errors such as the one described in #20856 

It seems that the proper way to call the method `get_supported_features_for_model_type`, is to first apply some pre-preprocessing as it is done [here](https://github.com/huggingface/transformers/blob/d87e381f9303c7d6a8aa7333dc09ed767de7395f/src/transformers/onnx/features.py#L723). This patch has been added to `tests/onnx/test_onnx_v2.py` too

This PR fixes these issues.

Also I would like to hear from @lewtun as it's my first ONNX-related PR.

cc @sgugger @ArthurZucker 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

I can confirm the script below works:
```
from transformers import  AutoModelForImageClassification
from transformers.onnx import FeaturesManager

model = AutoModelForImageClassification.from_pretrained("google/mobilenet_v2_1.0_224")
model_kind, model_onnx_config = FeaturesManager.check_supported_model_or_raise(model)
```
which yields to successfully retrieving the MobileNet ONNX config, however running `optimum-cli export onnx --model google/mobilenet_v2_1.0_224 onnx/` gives an error 
```
KeyError: "mobilenet-v2 is not supported yet. Only {'xlm', 'deberta', 'distilbert', 'electra', 'mobilebert', 'm2m-100', 'segformer', 'convbert', 'longt5', 'data2vec-text', 'flaubert', 'gptj', 'detr', 'layoutlmv3', 'mobilevit', 'groupvit', 'levit', 'mbart', 'big-bird', 'albert', 'bloom', 't5', 'swin', 'roberta', 'blenderbot', 'bert', 'yolos', 'marian', 'deit', 'layoutlm', 'perceiver', 'xlm-roberta', 'vit', 'gpt-neo', 'mt5', 'bigbird-pegasus', 'codegen', 'clip', 'whisper', 'data2vec-vision', 'squeezebert', 'convnext', 'deberta-v2', 'ibert', 'roformer', 'blenderbot-small', 'bart', 'beit', 'resnet', 'camembert', 'gpt2'} are supported. If you want to support mobilenet-v2 please propose a PR or open up an issue."
```
Mobilenet is not listed in `optimum.exporters.tasks`, probably by mistake or for some other reason. Opened a PR to support `Mobilenet` in optimum, https://github.com/huggingface/optimum/pull/633 I can confirm the model can be safely exported after checking out this PR! 

Now the PR https://github.com/huggingface/optimum/pull/633 being merged the export works as expected, merging