noamaanMulla-03/transformers-issues huggingface dataset:
Enabling vilt and flava auto feature extractor.
Author: Ngheissari
Pull request: no
Labels: name: bug

### System Info

```shell
File ~/anaconda3/envs/vqa/lib/python3.8/site-packages/transformers/models/auto/feature_extraction_auto.py:163, in AutoFeatureExtractor.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    161 if not is_feature_extraction_file and (has_local_config or not is_directory):
    162     if not isinstance(config, PretrainedConfig):
--> 163         config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    165 kwargs["_from_auto"] = True
    166 config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)

File ~/anaconda3/envs/vqa/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py:602, in AutoConfig.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    600     return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)
    601 elif "model_type" in config_dict:
--> 602     config_class = CONFIG_MAPPING[config_dict["model_type"]]
    603     return config_class.from_dict(config_dict, **kwargs)
    604 else:
    605     # Fallback: use pattern matching on the string.

File ~/anaconda3/envs/vqa/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py:319, in _LazyConfigMapping.__getitem__(self, key)
    317     return self._extra_content[key]
    318 if key not in self._mapping:
--> 319     raise KeyError(key)
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

trying to use vilt and flava feature extractors via AutoFeatureExtractor will result in the mentioned errors. 

### Expected behavior

```shell
I get an error when using ato classes with vilt , flava
```


Comments:
ViLT is going to be added in #17286 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This seems fixed for FLAVA now as well :)
AutoModel(s) do not respect the `revision` flag while loading custom models
Author: ankrgyl
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.1
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.10.5
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python

from transformers import AutoModelForImageClassification
m = AutoModelForImageClassification.from_pretrained(
        "sgugger/custom-resnet50d",
        trust_remote_code=True,
        revision="ed94a7c6247d8aedce4647f00f20de6875b5b292"
)
# It will print:
# Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
```

I stepped through the code and observed that `AutoConfig.from_pretrained` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/auto_factory.py#L423) swallows the `revision` from `kwargs`, meaning that later on line [433](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/auto_factory.py#L433) it's no longer there. I believe the same issue applies to `use_auth_token`.

### Expected behavior

I think the revision should propagate to both the configuration and model.

Comments:
cc @sgugger 

Thanks for flagging! The PR linked above should solve this.

Appreciate the quick turnaround :)
TF: XLA-trainable DeBERTa v2
Author: gante
Pull request: yes

# What does this PR do?

As discussed in https://github.com/huggingface/transformers/issues/18476 and https://github.com/huggingface/transformers/issues/18239, there are two problems while training DeBERTa v2 with TensorFlow:
1. `TFDebertaV2StableDropout` doesn't work at training time (actually, its logic is only triggered at training time, so it doesn't work at all :D)
2. TF complains about unknown shapes in `take_along_axis` (forward and backward passes, when the batch dim is `None`)

This PR fixes both problems above :) 

Problem 1. is got a straightforward fix. The gradient propagation code didn't have the right gradient shapes -- this PR simplifies and fixes it by moving all functions inside the special dropout class (compare to the original PT implementation [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L173) -- also notice how much more elegant TF's code is ;)).

Problem 2. is tricker. The exception gets fixed with the addition of a `shape_list`, but the code is super slow on TPU. This PR adds an if/else pair of branches, one that is efficient on TPU, the other on GPU :)

_____________________________________________________

⚠️ These exceptions were not caught because deberta v2 and v3 rely on special config options -- e.g. https://huggingface.co/microsoft/deberta-v3-base/blob/main/config.json#L14

How can we ensure we properly test these configurations?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@gante I think it's better to replace 

```python
flat_x = tf.reshape(x, (-1, x.shape[-1]))
flat_indices = tf.reshape(indices, (-1, indices.shape[-1]))
gathered = tf.gather(flat_x, flat_indices, batch_dims=1)
gathered = tf.reshape(gathered, shape_list(indices))
```
with 
```python
gathered = tf.gather(x,indices,batch_dims=2)
```
which gives the same numerical results and the same performance according to my tests

https://github.com/huggingface/transformers/issues/18239#issuecomment-1193126061

@WissamAntoun thank you for pointing it out, I completely missed it in the original thread! 🙏 Will make the change

EDIT: this change also makes it ~10% faster 👍
Preserve hub-related kwargs in AutoModel.from_pretrained
Author: sgugger
Pull request: yes

# What does this PR do?

As was reported in #18537, when using `AutoConfig` inside the `AutoModel.from_pretrained` method, some kwargs are deleted and not passed to the `from_pretrained` method of the model. This PR makes sure they are preserved for those calls.

Fixes #18537

Comments:
_The documentation is not available anymore as the PR was closed or merged._
can not covert LayoutLMv2ForRelationExtraction model to onnx
Author: githublsk
Pull request: no
Labels: name: bug


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
BART with multiple encoders. 
Author: prakamya-mishra
Pull request: no

Hi @patrickvonplaten , I am trying to evaluate my a model (Custom BART model with 2 inputs) that has 2 encoder and 1 decoder. For evaluation in the prediction_step method, it uses model.generate  (from generation_utils.py) for generating outputs. But the problem is that it only accepts inputs: Optional[torch.Tensor] = None,  but in my case I need to pass 2 set of inputs (one for both the encoders). So is there a way to pass 2 set of inputs to the generate method?

Comments:
Hi @prakamya-mishra -- if your model is designed to take two inputs, then you may be able to pass the second through `model_kwargs`. E.g. if the model signature is `model(input_ids, foo)`, then you may be able to run `model.generate(input_ids, model_kwargs{"foo": foo})`. Although you have to make sure your `prepare_inputs_for_generation` [method](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/modeling_bart.py#L1393) is expecting this argument.

Please note that as per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we do not provide support to custom code. We also reserve these issues for bugs and feature requests -- for other questions, we'd like to invite you to use the [forums](https://discuss.huggingface.co/) :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
issue installing SentencePiece
Author: SamitM1
Pull request: no
Labels: name: bug

### System Info

the error i get when i do "pip3 install SentencePiece":

  ./build_bundled.sh: line 15: cmake: command not found
      ./build_bundled.sh: line 16: nproc: command not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target `install'.  Stop.
      env: pkg-config: No such file or directory
      Failed to find sentencepiece pkg-config
      [end of output]


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

pip3 install SentencePiece

### Expected behavior

unable to install 

Comments:
Hi @SamitM1 👋 The issue seems to be related to the package you mentioned, not `transformers` -- I'd suggest opening an issue there (https://github.com/google/sentencepiece)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TF Examples Rewrite
Author: Rocketknight1
Pull request: yes

This PR is a rewrite of the TF examples, including several modern methods. I'm focusing on updating everything to use modern methods like `prepare_tf_dataset` and the `evaluate` library as well as adding features and functionality I missed when I first ported them, since `transformers` TF support was much shakier when these were first written (and I didn't know the library as well).

Just a draft for now, will ping reviewers when it's ready!

TO DO:

- [x] Draft rewrite for all scripts
- [x] Test run all scripts
- [x] Make sure we're handling batch sizes correctly in multi-GPU/TPU scopes
- [x] Make sure we're correctly using AdamW + LR decay everywhere
- [x] Make sure we're using `evaluate` instead of `load_metric`
- [x] Add metadata for `push_to_hub`
- [x] Add explanatory comments for things like `KerasMetricCallback`, `jit_compile` and `PushToHubCallback` where appropriate
- [x] Replace all the old ad-hoc data loading code with `prepare_tf_dataset`
- [x] Add explanatory links to the docs whenever we use `prepare_tf_dataset`
- [x] Add example tests
- [x] Make sure there's no case where we pass `optimizer=None` to `compile()`
- [ ] Final manual testing
- [x] ~Add HF metrics like MaskedAccuracy?~

Fixes #18334 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is now ready for review @sgugger @gante! I'm tracking down a couple of remaining bugs in the tests and doing some final manual checks, but almost everything should be finished by now.

I realize it's a very large PR, but you can see from the checklist above what the main changes are.

@sgugger Tests are now enabled in `config.yml` and everything still looks green!

That might because your new job did not run ;-) 
You need to add at the end [here](https://github.com/huggingface/transformers/blob/d7e2d7b40b1070cddfe878e13705725f49a2cf1f/.circleci/config.yml#L1000) for the one at each commit and [there](https://github.com/huggingface/transformers/blob/d7e2d7b40b1070cddfe878e13705725f49a2cf1f/.circleci/config.yml#L1024) for the nigthly one ;-)

![image](https://user-images.githubusercontent.com/12866554/183910275-b295a0ef-f39e-47fe-b3a0-5001e1ecddb3.png)


@sgugger tests are now actually passing! I had to skip one - it fails because of a known issue with shape inference on small datasets in `to_tf_dataset`. There is a PR to fix that at https://github.com/huggingface/datasets/pull/4763 , we just need to wait for that to be merged before we can re-enable the test!
Use commit hash to look in cache instead of calling head
Author: sgugger
Pull request: yes

# What does this PR do?

This PR tries to limit the calls to requests.head made for cached models every time we try to load them. Currently on the main branch, a call to the following objects results to the following number of underlying calls to the API:
- AutoConfig: 1 (fiou)
- AutoModel: 2 (model + config)
- AutoTokenizer: 9 (multiple tokenizer files and multiple calls to config)
- pipeline: 13 (all of the above + one extra call to config)
- a sharded model: number of shards + 2

This is a bit excessive, so this PR reduces this to the maximum it can by using the commit hash of the first file downloaded: if it's the same as something we have in the cache, then all files in that subfolder with the same commit hash are up to date.

As you can see in the tests it does not completely succeed, because we can't detect with this reasoning if a file does not exist in the repo: if it's not in the cache, it could be because it's still not downloaded yet. But still it reduces the number of calls seen above to:
- AutoConfig: 1
- AutoModel: 1
- AutoTokenizer: between 2 and 4 depending on the tokenizer
- pipeline: between 2 and 4 depending on the tokenizer
- a sharded model: 2

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Segformer TF: fix output size in documentation
Author: joihn
Pull request: yes

Fixes # (issue)
https://github.com/huggingface/transformers/issues/18557

## Before submitting
- [ X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@sayakpaul 


 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi,

thanks for your PR. Could you also fix this in `modeling_segformer.py` for the PyTorch implementation?

Thanks!
Fix broken pipeline string
Author: mrwyattii
Pull request: yes

# What does this PR do?
#18494 introduced a bug in a cuda device string:
`RuntimeError: Invalid device string: 'cuda:{device}'`

This PR adds the missing `f` before the string
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@julien-c @LysandreJik 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for fixing!
[FX] _generate_dummy_input supports audio-classification models for labels
Author: michaelbenayoun
Pull request: yes

For FX:

- Adds support for audio-classification models for label generation in `_generate_dummy_input`
- Adds a flag, `FX_DEBUG_MODE` to control what's being printed during tracing, this will save the user from seeing a lot of benign warnings, while still providing the possibility to have those while developing.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Return the permuted hidden states if return_dict=True
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes an issue where the shape of the returned hidden states is different if `return_dict` is True or False for ConvNext. 

The outputs of ConvNext are permuted in the final layer `TFConvNextMainLayer` to put them in `(batch_size, num_channels, height, width)` format, to match the pytorch model. However, if `return_dict=True` then the non-permuted hidden states are returned. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix LayoutLMv3 documentation
Author: pocca2048
Pull request: yes

# What does this PR do?

- Fixes documentation of LayoutLMv3Model and some other typos.

Fixes # (issue)
#17833

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@NielsRogge



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Applied your suggestions!

Thanks, could you run `make style` and `make quality` from the root of the repo? This ensures the code quality check will pass. 

Applied and all checks are passing!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sgugger I change into two versions. Could you re-open this pull request?
Load sharded pt to flax
Author: ArthurZucker
Pull request: yes
Labels: name: Core: Modeling

name: Flax

# What does this PR do?
Add conversion to `flax` from sharded `pytroch` checkpoints. Follows #18026  which was closed to rename the branch (no really necessary, sorry for the inconvenience). Should fix #17537


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Okay 👍🏻  Thanks for the review gonna fix that asap !
Add `TFAutoModelForSemanticSegmentation` to the main `__init__.py`
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, `from transformers import TFAutoModelForSemanticSegmentation` fails.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot for fixing!

test failure is irrelevant. Merge now.
model.generate doesn't validate kwargs
Author: stas00
Pull request: no

### Feature request

I made a mistake in a script:
```
model.generate(**tokens, in_length=num_tokens)
```
missing `m` in `min_length` and I was puzzling over why I was getting unexpected results (as it was using the default value which was quite different from mine)

Would it be possible to have `generate` validate its input and assert on unexpected args?

Thank you!

@patrickvonplaten 


Comments:
cc @gante as well

Hi @stas00 👋  -- we do have a plan for it. The rough sketch is [here](https://github.com/huggingface/transformers/pull/17196#issuecomment-1155002093), and I will pick it up after the last wrinkles related to TF generate have been ironed out (which should be very soon!)

excellent. Thank you, @gante!

I guess let's keep this Issue open for tracking unless there is another one already? 

Yeah, let's keep this one open!
Supporting seq2seq models for `bitsandbytes` integration
Author: younesbelkada
Pull request: yes

# What does this PR do?

The previous checks to check which key to not convert in int8 was not sufficient. It appears that T5 models were not able to be converted correctly. This PR addresses this issue by adding an extra check consisting of checking whether the model has tied_weights inside the `get_key_to_not_convert` function. 

cc @philschmid @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._