noamaanMulla-03/transformers-issues huggingface dataset:
Any example for Wav2vec2ForXVector training? 
Author: LEECHOONGHO
Pull request: no

Hello, I'm trying to train Wav2vec2ForXVector model on setting like below. But the training loss is not falling from 2.3~2.7. Is there any example for Wav2vec2ForXVector training? Or had anyone 
Experienced like this? 

pretrained_model : korean wav2vec2
num of audio : 2300k
num of speaker : 11223
num of used encoder layer : 1
output_xvector_dim : 512
learning rate : 2e-5
batch size : 512

![Screenshot_20221013-083616_Chrome](https://user-images.githubusercontent.com/44384060/195466991-5e8ab6d5-7943-4a4a-8580-c7ee9e3c4fb1.jpg)


Comments:
There isn't currently an example for XVector training in Transformers! Would you like to contribute this? You can begin simply by opening a PR with the python script that you're using. We can then iterate on it to verify correctness and hopefully get a successfully trained XVector system!

Probably also worth asking the same question on the forum to boost visibility: https://discuss.huggingface.co

Also cc @anton-l who has a speaker verification (SV) checkpoint on the Hub (https://huggingface.co/anton-l/wav2vec2-base-superb-sv), wondering if you had a local script for XVector fine-tuning?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, sorry for missing this! To answer @sanchit-gandhi's question: my SV checkpoint is a ported version of W2V2+XVector from S3PRL: https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/sv_voxceleb1
So no finetuning scripts yet, just inference 

Hey @LEECHOONGHO! If you want to work together to get a working XVector training script, feel free to open a PR with the script that you've got and tag me. We can iterate on it, ensuring correctness and building up to a full Transformers examples script! I think this would be of benefit to others in the community 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Yolo model trigger win10/11 reboot on Intel 13900k
Author: Michael-YongWang
Pull request: no

### System Info

All Yolos models/pipeline case my pc reboot immediately on windows 10 and 11 system ( WSL in win11 and Ubuntu(not virtual) work well on the same machine)

1. I reinstalled the os system and tested (both win10 and win11) two times and make sure all the drivers are up to date. The results are the same. (I test the system with Aida64 system stability test, all ok)

2. I tried Ubuntu(not virtual) and WSL in windows 11, both works ok.

3. I tried other models, such as Bert, all works fine.

``` python
from transformers import YolosFeatureExtractor, YolosForObjectDetection
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = YolosFeatureExtractor.from_pretrained('hustvl/yolos-tiny')
model = YolosForObjectDetection.from_pretrained('hustvl/yolos-tiny') 

inputs = feature_extractor(images=image, return_tensors="pt")

for _ in range(100):
    model(**inputs) # this line causes the PC reboot. (usually after several loops)
```

Hardware info:
CPU: Intel 13900k
Motherboard: ASUS Z690-G, bios 2103
GPU: Intel 13900k

Software info:
Conda env latest
Python 3.10.4
transformers 4.24.0
windows 10 and windows 11

My guess it is caused by the incompatible of the hardware.
Could you please help to take a look?

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Use python 3.10 on windows 10/11 with Intel 13900k

run the code below

``` python
from transformers import YolosFeatureExtractor, YolosForObjectDetection
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = YolosFeatureExtractor.from_pretrained('hustvl/yolos-tiny')
model = YolosForObjectDetection.from_pretrained('hustvl/yolos-tiny') # this line cases the PC reboot.

inputs = feature_extractor(images=image, return_tensors="pt")

for _ in range(100):
    model(**inputs) # this line causes the PC reboot. (usually after several loops)
```

### Expected behavior

The pc reboot immediately after several loop

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

How fast does it work with transformers?
13900k compared to gpu?
Sanity check the type of id2label and label2id arguments of from_pretrained for TokenClassification models
Author: raghavanone
Pull request: yes

Fixes #20773 

Sanity check the type of id2label and label2id arguments of from_pretrained for TokenClassification models

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger One of TF test is failing due to timeout. Not sure if it is related to changes . 


No, this isn't linked to the PR. Thanks for your contribution!
A new test to check config attributes being used
Author: ydshieh
Pull request: yes

# What does this PR do?

Add a new test to check config attributes being used.

For edge cases, I only add rules to 2 files. If the concept is approved,  **I will add more to pass CI**, and continue the work of cleaning up in follow-up PR(s).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger Ready for review - The failing tests will be addressed (by adding specific rules in subclasses) once the PR is approved 🙏 .

The report looks like (if unused attributes are detected)
```bash
ValueError: The following configuration classes contain unused attributes in the corresponding modeling files:
CLIPSegConfig: ['decoder_attention_dropout', 'decoder_hidden_act']
...
DinatConfig: ['patch_norm']
...
```

> Thanks for iterating! Just left a couple more comments.
> 
> As you noticed the job is failing right now, are you planning to add everything in the special map or to fix all the attributes not used?

Yes, that's mentioned in the description/comment (in previous version, it's better doing so, but with this new version you suggested, I can indeed add them earlier though).

FYI: the special map I will update will contain
- some confirmed allowed cases (i.e. we know the reasons and nothing we can do but just allow)
- some skipped cases **for now** to allow temporarily with #TODO comment

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21453). All of your documentation changes will be reflected on that endpoint.
Added onnx config whisper
Author: mht-sharma
Pull request: yes

# What does this PR do?

Fixes # (issue)

This PR adds onnx config and helper functions for export to onnx via optimum and transformers.onnx


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lewtun @echarlaix  The bug is fixed now. The incorrect results were because the export was happening with seqlength 1 due to typo in onnx config generate_dummy_input function.

Thanks for the review @sgugger and catching those last issues - I've checked the latest changes and think this can now be merged @mht-sharma 

Hello,

I tried to generate onnx model using [docs](https://huggingface.co/docs/transformers/serialization#configuration-based-approach),
To inference, I passed audio features and decoder_input_id , and the output was two array with the shape of (1, 2, 768), (1, 1500, 768). Could you please help me how should I use these outputs for generating transcription? 

Thank you.

Hi @zara0m please follow the example in this PR for export and  inference using ONNX model. https://github.com/huggingface/optimum/pull/420

> Hi @zara0m please follow the example in this PR for export and inference using ONNX model. [huggingface/optimum#420](https://github.com/huggingface/optimum/pull/420)

Thank you very much for your help and quick response!

I tested it with base and small model for some non-english audios, but the outputs were not similar to whisper model, or maybe it translates instead of transcribing, how can I fix this?
Also, is there any way that I can have begin/end time of each sentence? (like the transcribe function of whisper model)

Thank you.
Cleanup quality
Author: sgugger
Pull request: yes

# What does this PR do?

This PR cleans up a couple of things after the big #21480 
- adds the ruff cache to gitignore just in case (that folders comes with its own auto-generated gitignore so shouldn't be necessary but better be safe)
- remove all mentions of flake8 and isort from the doc
- use the per file rules to enable back quality checks on all inits.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[tokenizer] sanitize saved config
Author: stas00
Pull request: yes

this PR fixes tokenizer's `save_pretrained` to remove the `name_or_path` entry from `tokenizer_config.json` because:
1. it usually contains the local path that was used to save the model, which is not only invalid once published on the hub, it could potentially reveal some personal information.
2. it is not used anywhere, since one needs to know `name_or_path` before they can load this file.

it also adjusts one tokenizer test not to test for the `name_or_path` entry 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
layoutlmv2 visual backbone pretrained on publaynet 
Author: gauravlochab
Pull request: no

@NielsRogge Amazing work thanks for sharing,
Does layoutlmv2's visual backbone utilize ResNeXt-FPN trained on Publaynet as described in the paper? If so, could you please point out where the weights are used to initialize the model in the repo?

Comments:
Hi,

The LayoutLMv2 authors indeed leveraged a ResNeXT-FPN for the visual backbone. It was taken from this repo: https://github.com/hpanwar08/detectron2.

However, we've just added the [Document Image Transformer (DiT)](https://huggingface.co/docs/transformers/master/en/model_doc/dit), which actually outperforms the ResNeXT model on PubLayNet with the same object detection framework (Mask R-CNN), reaching an AP of 0.945/0.949 for DiT-base/large respectively (vs. 0.906 for ResNeXT).

Take a look at this Space for a demo: https://huggingface.co/spaces/nielsr/dit-document-layout-analysis

@NielsRogge Thanks for responding; yes, I got my hands on DIT, which is a fantastic model, but I'm trying to figure out where the ResNeXT-FPN weights are initialized in the layoutlmv2 model for the current hugging face implementation. 
https://github.com/microsoft/unilm/blob/master/layoutlmft/layoutlmft/models/layoutlmv2/detectron2_config.py i don't see any weights path given here 

I don't think Microsoft open-sourced this weights initialization. This was probably part of their pre-training code (which is not open-sourced).

Weights can be found here: https://github.com/hpanwar08/detectron2#benchmarking

Then how does the visual backbone extracts features from the image? I can see the model config for detectron2 but I can't figure out what weights the model is getting initialized with? 
for example in detectron2, we pass config fille and YAML for weights 
```
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
```

Yes I know, they probably set the MODEL.WEIGHTS attribute in their pre-training code, which they didn't open-source.

But as linked above, you can easily get the backbone weights.

I want to override the weights for the model but when I pass the weight's path in the visual backbone class the model is not picking up the weights but the MODEL.WEIGHTS key in config is showing the model path string,
 Any reasons why? @NielsRogge 

got the weights from the backbone and loaded the custom weights 

Hi @gauravlochab,
Could you please share how you managed load the custom weights?
Fix import in Accelerate for find_exec_bs
Author: sgugger
Pull request: yes

# What does this PR do?

This seems to have slipped through the cracks of the reorg in Accelerate. Should fix the import error seen in the tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Abusing my admin right to merge this right now since all tests are green and this fixes a test failing on many other PRs :-)
Add whisper converter for hf -> openai
Author: faroit
Pull request: no

### Feature request

The inference pipeline in openai whisper has a couple of heuristics that aren't all covered in https://github.com/huggingface/transformers. Therefore, some users would like to fine-tune in huggingface and convert the model back to its original configuration.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/convert_openai_to_hf.py provides a script to convert from openai to hf, so we should also have a script to go the other way.

### Motivation

It is difficult to get the same transcription quality in the current transformers library compared to the openai transcribe function https://github.com/openai/whisper/blob/main/whisper/transcribe.py.

### Your contribution

there is an existing approach in https://github.com/luigisaetta/whisper-app/blob/main/match_layers.py by @luigisaetta 

Comments:
👋 @ArthurZucker, who created the conversion script
 

Hey! I am not entirely sure if we should have this in our repo or just link it in the readme, but I think a contributor already wrote a script so pinging @bayartsogt-ya here.

For visibility, whisper.cpp provides conversion from HF models into their format.
It's much high performing that OpenAI implementation. See: https://github.com/ggerganov/whisper.cpp/tree/master/models

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I've seen a lot of converter codes flying around. Also given that tranformers is catching up with respect to timestamp outputs there is not much need to use whisper itself for inference anymore

@faroit could you add a link to a conversion script? Just so if people end up here, they find what they are looking for.
Wrap RemBert integration test forward passes with torch.no_grad()
Author: katiele47
Pull request: yes

# What does this PR do?

Fixes #14642. Wrapped the forward pass inside RemBert's integration test with `with torch.no_grad()` to ensure no gradients are computed during inference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@LysandreJik Please let me know if this fix works. Thank you:)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Exclude the madeup words from M2M100Tokenizer.vocab_size
Author: guillaumekln
Pull request: yes

# What does this PR do?

The `<unk>` token has an incorrect ID in `M2M100Tokenizer.get_vocab`:

```python
>>> tokenizer = transformers.M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")
>>> tokenizer.convert_tokens_to_ids("<unk>")
3
>>> tokenizer.get_vocab()["<unk>"]
128111
```

The reason is the vocabulary is defined like this:

```
vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
```

but `_convert_id_to_token` converts the "madeup words" to `<unk>`.

We can fix this issue by excluding the "madeup words" from the vocabulary size, which is consistent with how other tokenizers work such as `NllbTokenizer`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @ArthurZucker, do you have some time to review this PR? Thanks!
Add XLM-V to Model Doc
Author: stefan-it
Pull request: yes

Hi,

as discussed in #21330 it would be good to have an extra entry for the new XLM-V model in the Model Doc.

This PR adds it with some additional information about the model and conducted experiments with it.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

CI is failing, I'm going to read the Flan-T5 PR (https://github.com/huggingface/transformers/pull/19892) to see how it should be done!

You just need to add the model type (same as what you picked for the page in the doc) and name in the configuration_auto file. The PR you mention also does it :-) 

Failures are unrelated to this PR so merging!
Generate: TF `compute_transition_scores`
Author: gante
Pull request: yes

# What does this PR do?

This PR adds the TF `compute_transition_scores`, akin to PT's #21191.

What seemingly started off as a simple task, ended up being a complex task -- the TF side had many missing and/or incorrect secondary `generate` outputs 😬 This means that we need to beef up the TF side of `generate` tests, which is much shorter than its PT counterpart.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten reverted the change in the order of operations in beam sample, and added a comment to based on our offline conversation.

LMK if you're happy with the PR 🙏
Make `_LazyAutoMapping` a bit less lazy 😪 🔥 
Author: ydshieh
Pull request: yes

# What does this PR do?

Current `_LazyAutoMapping` is a bit too lazy, and potentially to give silent failure/error. For example, tests being skipped by mistake. One explicit example is
https://github.com/huggingface/transformers/blob/c26b1503f22feb20d37588d4243bd444d297bc13/tests/test_pipeline_common.py#L86
(so far in a not-yet merged PR), which was previously
```python
if model_mapping is None or len(model_mapping) == 0: 
```
and the tests were all skipped, and I am like 😕 .

As a former mathematician, it's hard for me to accept length being 0 in this case 🔢 . I decide to make it less lazy!

## Results

### Before this PR
```python
from transformers import TF_MODEL_MAPPING

print(len(TF_MODEL_MAPPING)) # 0
```

### After this PR
```after
from transformers import TF_MODEL_MAPPING

print(len(TF_MODEL_MAPPING))  # 60
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failing tests are irrelevant.
Generate: make TF `.generate()` signature == PT `.generate()` signature 
Author: gante
Pull request: yes

# What does this PR do?

TF generation test addition PR 3 (out of ???).

In an effort to move generation integration tests to be framework-agnostic, I'll be adding low-hanging fruit to TF. This PR adds framework-agnostic tests for the signature of `.generate()`, which means that the TF signature had to be updated 🙃 

Please note that this change is backward compatible -- regardless of whether the user was passing `.generate(input_ids)`, `.generate(input_ids=input_ids)`, or (e.g.) `.generate(pixel_values=pixel_values)`, no change on their end is needed.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix missing unfinished_sequences
Author: tokestermw
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/pull/21461#issuecomment-1423684619

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you @tokestermw. I confirmed the 2 failed doc-tests pass now with the changes in this PR ❤️ . 
I will leave @gante to double check the logic.