noamaanMulla-03/transformers-issues huggingface dataset:
[Generate] Fix gradient_checkpointing and use_cache bug for BLOOM
Author: sriagastya
Pull request: yes

Fixes #21737 for Bloom.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).


## Who can review?

cc @younesbelkada @gante 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add missing parameter definition in layoutlm config
Author: Atomnp
Pull request: yes

Four parameters in `LayoutLM` config were missing definitions, Added their definition (copied from BertConfig).

# What does this PR do?
Fix docs, add parameter definition copying them from BertConfig


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @amyeroberts
`from_pretrained` broken 4.26.1
Author: NicholasKao1029
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.0a0+d0d6b1f (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@amyeroberts
@ArthurZucker and @younesbelkada


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code block
```
from transformers import CLIPTokenizer, CLIPTextModel
import torch

tokenizer=CLIPTokenizer.from_pretrained("runwayml/stable-diffusion-v1-5"),
text_encoder=CLIPTextModel.from_pretrained("runwayml/stable-diffusion-v1-5").to("cuda")
```
output
```
Traceback (most recent call last):
  File "diffusers-oneflow/examples/poc.py", line 17, in <module>
    tokenizer=CLIPTokenizer.from_pretrained("runwayml/stable-diffusion-v1-5"),
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'runwayml/stable-diffusion-v1-5'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'runwayml/stable-diffusion-v1-5' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.
```





### Expected behavior

Expect to be able to consume and not throw

Comments:
Hey! If you go to `hf.co/runwayml/stable-diffusion-v1-5` you will se that there are no `tokenizer.json` or any files related to tokenization. However there is a `tokenizer` folder. The new version of transformers supports `subfolder`. 
The model on the hub was modified, thus the command that you are looking for is probably: 
```python 
>>> from transformers import AutoTokenizer
>>> AutoTokenizer.from_pretrained("runwayml/stable-diffusion-v1-5",subfolder= "tokenizer")
```
which worked for me 😉 

oh yes this is great thank you
Use larger atol in `torch.allclose` for some tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Running CI against torch 2.0 (and therefore CUDA 11.7), some tests for `BridgeTowerModel` failed:
- test_disk_offload
- test_cpu_offload
- test_model_parallelism

While with torch `1.13.1` (with CUDA 11.6), the difference between `base_output` and `new_output` in these 3 tests are 0.0, we get `1e-7 ~ 3e-6` as difference for torch `2.0`.

**Deep debugging reveals that the first non-zero difference occurs when `nn.MultiheadAttention` is called.**

This PR increase the atol to `1e-5` (the default one is `1e-8`) in `ModelTesterMixin`.

If we don't feel super good with this for all model tests, we can override these 3 tests in `BridgeTowerModelTest`.
(But when more models start using `nn.MultiheadAttention`, it's best to keep this larger value in common testing file)



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TF contrastive image text finetuning example
Author: Rocketknight1
Pull request: yes

This PR adds a TF port of the PyTorch example for finetuning the `TFVisionTextDualEncoderModel` class. Functionality is largely the same, but I used a `tf.data` pipeline to efficiently stream images instead of `torchvision`. I also added the ability to specify separate image/text models with arguments, whereas in the PyTorch example you have to create the dual encoder with a separate script.

I also caught a small bug in the original model code while writing this - loss is a scalar rather than having shape `(1,)`. That's fixed in here too!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Update: Woah! The argument names to `PushToHubCallback` across our examples became outdated and tests weren't picking that up. I'm fixing it in this PR too.
Update expected values for `test_xglm_sample`
Author: ydshieh
Pull request: yes

# What does this PR do?

Despite the `probs` value has only a difference of  `7.59e-07`, the `torch.multinomial(probs, num_samples=1)` still gives different `next_tokens`:

  - 3967 (`sun`) in `torch 1.13.1`
  - 4565 (`water`) in `torch 2.0`

Currently I keep both values, but we can remove the one for `torch 1.13` soon.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix gradient checkpointing bug in BigBird Pegasus
Author: KMFODA
Pull request: yes

This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes Issue https://github.com/huggingface/transformers/issues/21737

cc @younesbelkada or @gante

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix gradient checkpointing bug in Blenderbot Small
Author: KMFODA
Pull request: yes

This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes Issue https://github.com/huggingface/transformers/issues/21737

cc @younesbelkada or @gante

Comments:
_The documentation is not available anymore as the PR was closed or merged._
docs: improve clarity for language modeling
Author: pdhall99
Pull request: yes

# What does this PR do?

- Improve clarity of tasks/language_modeling and tasks/masked_language_modeling docs.
- In example preprocessing, remove `truncation=True` parameter from `tokenizer` so texts aren't truncated before being concatenated and chunked.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger, @stevhliu

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Regarding Ragend2endRetriever
Author: Rajdoshi99
Pull request: no

Unable to Change the Generator to T5ForConditionalGeneration. 
Shows error.raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'T5ForConditionalGeneration' object has no attribute 'rag'

--model_name_path t5-base
--model_type t5
@shamanez 

Comments:
add MaxViT [TF]
Author: awsaf49
Pull request: no
Labels: name: New model

### Model description

MaxViT: Multi-Axis Vision Transformer is one of the nice papers of late 2022 which is also published in [ECCV 2022](https://arxiv.org/pdf/2204.01697v4.pdf) by Google AI.  
* This paper introduces a new attention module called "multi-axis attention" which consists of blocked local and sparse global attention for efficient and scalable spatial interactions on arbitrary input resolutions.
* It demonstrates superior performance on various vision tasks including image classification, object detection, and so on.

I think it would be nice to have it on Hugging Face. I would be happy to contribute it on Hugging face.

cc: @alara @NielsRogge 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Code and Weights: 
* TF1: https://github.com/google-research/maxvit
* TF2: https://github.com/leondgarse/keras_cv_attention_models

Comments:
are you working on this? @awsaf49 

Yes
Add check before int casting for PIL conversion
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds safeguards when images are converted to PIL images:
* Additional condition if inferring `do_rescale` 
* Raise an error if values cannot be cast to `uint8` 

The PIL library is used for resizing images in image processors. If not explicitly set, whether or not to rescale pixel values is inferred based on the input type: if float then values are multiplied by 255. If the input image has integer values between 0-255, but are of floating type, then these pixel are rescaled to values between [0, 65025]. This results in overflow errors when cast to `uint` [here](https://github.com/huggingface/transformers/blob/bc33fbf956eef62d0ba8d3cd67ee955ad5defcdb/src/transformers/image_transforms.py#L162) before converting to a `PIL.Image.Image`.

Fixes #21915  


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix MinNewTokensLengthLogitsProcessor when used with a list of eos tokens
Author: eladsegal
Pull request: yes

# What does this PR do?

MinNewTokensLengthLogitsProcessor is missing support for a list of eos token ids.
This PR adds the missing support, in the same way it was added in MinLengthLogitsProcessor.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante



Comments:
_The documentation is not available anymore as the PR was closed or merged._
[DETR, YOLOS] Fix device bug
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes a device bug for DETR and YOLOS's `post_process_object_detection` methods, which currently give a device mismatch between CPU and CUDA when running the model on CUDA.

The PR also makes sure the postprocess methods are tested in the integration tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update `notification_service.py`
Author: ydshieh
Pull request: yes

# What does this PR do?

While working on CI report with PyTorch `2.0.0`, the first run failed to send the report due to the following problem:

- The original code checked each line in `summary_short.txt` by `if re.search("FAILED", line):`
- We expect such occurrences have 1-1 correspondence in the file `failures_line.txt`
- However, some lines in `summary_short.txt` might have ` FAILED` but not really what we are looking for. For example, some tests in `tests/extended/test_trainer_ext.py` using `execute_subprocess_async` and we get some lines like
  ```
  /transformers/examples/pytorch/translation/run_translation.py FAILED
  ```
- In such cases, we get error `stacktraces.pop(0)` at some point, as there is no more element to pop (`stacktraces`, obtained from `failures_line.txt`)
- **This PR avoids this situation by checking with `if line.startswith("FAILED "):` which should give the desired 1-1 correspondence.**


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip `test_multi_gpu_data_parallel_forward` for some model tests
Author: ydshieh
Pull request: yes

# What does this PR do?

This test fails for some models in CI with torch 2.0, causing CUDA in a bad state, and many other tests fail in this situation.
The only way (I could find online) that it won't fail is to use other GPUs, like `P100` or `V100`.

Let's skip it for now for a few model tests. It's likely it will work again in a future PyTorch release.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TensorFlow Whisper model for audio classification
Author: sanchit-gandhi
Pull request: no
Labels: name: TensorFlow

name: Good Second Issue

name: Feature request

### Feature request

The PR https://github.com/huggingface/transformers/pull/21754 adds the PyTorch version of `WhisperForAudioClassification`. It would be great to add the TensorFlow equivalent.

### Motivation

Whisper is an encoder-decoder model for speech recognition. However, we can repurpose the model for other speech tasks, such as _audio classification_.

Audio classification is the task of mapping from an input speech sequence to a single class prediction. For more details, refer to the task page on the Hub: https://huggingface.co/tasks/audio-classification

For audio classification, we only require a _single_ model output. Thus, we do not need the auto-regressive generation capacities of the Whisper decoder (which is used to generate a _sequence_ of text tokens during speech recognition). Instead, we can just use the Whisper encoder to get hidden states, and add a classification head on top to make class label predictions.

This is analogous to using a Wav2Vec2 model for audio classification: the Wav2Vec2 encoder is used to get hidden states, and a classification head added on top to make class label predictions.

The PR https://github.com/huggingface/transformers/pull/21754 adds the PyTorch version of `WhisperForAudioClassification`. It required adding a projection layer and classification layer on top of the `WhisperEncoder`. For more details, refer directly to the pull request.

It would be great to add the TensorFlow equivalent of this model for cross-framework support.

The most difficult part of this PR will be getting the model tester to work. You can see from the PyTorch PR that we require a standalone tester for the audio classification model. This is because the original Whisper model is an encoder-decoder model, but the audio classification model is an encoder-only model. Thus, we require different testing logic.

### Your contribution

Opening this one up to the community! If you're interested in tackling this, free to drop a comment in this thread and open a PR when you're ready. More than happy to answer any questions / queries about this integration!

Comments:
Hey @sanchit-gandhi, if we're just using the encoder do you think a CTC head could also work, i.e. `WhisperForCTC`?

Hey @OllieBroadhurst! I don't think a an encoder-only Whisper model for speech recognition would be super practical since we'd then need an _external_ language model to correct the phonetic errors made by the CTC model. IMO we're better off using the _internal_ language model provided by the decoder in the original encoder-decoder architecture. The encoder-decoder model is trained end-to-end and on all of the Whisper pre-training data, so likely going to be better than any combination of CTC + LM we train ourselves

Hello @OllieBroadhurst are you currently working on this? I would love to help out if I can/you need it. Otherwise, I would like to take a look at this issue.

Hi @adit299 ! I'm not so you can take it away!

Great, will do!
Support for `Flax` Trainer
Author: Shubhamai
Pull request: no

### Feature request

A `Trainer` class similar to PyTorch/Tensorflow (deprecating in v5) . 

### Motivation

The process of training HuggingFace models in `PyTorch` has become more accessible due to the availability of the `Trainer` class and extensive documentation and tutorials. Similarly, in `Tensorflow`, training models require only a single line of code (`model.fit`). and it makes sense to deprecate the current trainer to avoid redundancy. 


However, the process of training `Flax` models currently requires boilerplate code similar to `PyTorch` which a `Trainer` class would be helpful to eliminate. Making flax `Trainer` available will allow for really fast training in GPU/TPUs and with the support of weights conversion, one can instantly convert the model to `Tensorflow`/`PyTorch` for inference and deployment.

### Your contribution

I can submit the PR. 

cc @sanchit-gandhi since it's related to flax. 

Comments:
This is also something we do not want to add or maintain for Flax, as researchers usually dislike Trainer classes very much.

Understood, thanks for such a quick reply :hugs:
[TF] Can't open a pr when pushing 
Author: ArthurZucker
Pull request: no

Creating a PR when uploading a model should work. 

```python 
>>> from transformers import FlaxT5ForConditionalGeneration, TFT5ForConditionalGeneration
>>> import jax.numpy as jnp
>>> model = FlaxT5ForConditionalGeneration.from_pretrained("./art/flan-ul2", dtype = jnp.bfloat16, from_pt = True)
>>> model.push_to_hub("google/flan-ul2", use_auth_token = "XXXXX",create_pr = True)
>>> del model

>>> model = TFT5ForConditionalGeneration.from_pretrained("./art/flan-ul2", from_pt = True)
>>> model.push_to_hub("google/flan-ul2", use_auth_token = "XXXXX",create_pr = True)
  File "/home/arthur_huggingface_co/transformers/src/transformers/modeling_tf_utils.py", line 2986, in push_to_hub
    self.create_model_card(**base_model_card_args)
TypeError: create_model_card() got an unexpected keyword argument 'create_pr'
```


Comments:
cc @gante
[TF] Fix creating a PR while pushing in TF framework
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #21967, where models in TF can't push and open a PR. A test should probably be added

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yep, adding this

I had a question, the `create_pr` function parameter is not available in the corresponding `PyTorch` or `Flax` implementation, I was wondering if this is intended. 

Not sure I follow, torch already has this parameter.
[DETR and friends] Remove is_timm_available
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR:
- removes the `is_timm_available` dependency check for DETR and friends, uses `is_torch_available` instead and uses `requires_backends["timm"]` in case `config.use_timm_backbone=True`.
- adapts DETR's conversion script to make DETR work with our `AutoBackbone` class, rather than the timm backbone. This way one can use DETR by only installing `Transformers`. 

To do:

- [x] upload checkpoint to the hub
- [x] add integration test

Comments:
_The documentation is not available anymore as the PR was closed or merged._

There is a pretty big conflict in the test modeling DETR file, can you fix it Niels?