noamaanMulla-03/transformers-issues huggingface dataset:
add customizable ending learning rate arguments to warmup schedulers
Author: NoTody
Pull request: yes

# What does this PR do?

I noticed that in the original implementation, the learning rate for cosine and linear scheduler with warmup is always scheduled to 0. However, in many new research such as Masked Autoencoder, BEiT, they scheduled the learning rate to some no 0 end learning rate (1e-6 in their case). Hence, I added this features for these schedulers, so people who don't want to schedule learning rate directly to 0 can also use huggingface schedulers.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@Narsil 
@sgugger

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21434). All of your documentation changes will be reflected on that endpoint.

Not sure I can properly review this code, sorry I never touch that file.

Thank you for your PR, but the Transformers library is primarily a library of models. Those schedulers are just implemented for ease of use in our Trainer (which wouldn't be able to set that `end_lr` new argument), anything more involved should come from another library/custom user code :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Help with implementing strides into features for multi-label classifier
Author: alvin-leong
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

As you might know, BERT has a maximum wordpiece token sequence length of 512. 
The SQuAD example actually uses strides to account for this: https://github.com/google-research/bert/issues/27
I want to implement something like what Jacob Devlin described in that post for Kaushal's multi-label BERT classifier: https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d

What I did was basically just copy and edit a bit the doc_stride functions from the SQuAD example in huggingface into Kaushal's code: https://colab.research.google.com/drive/1aqcIdm2Pn2rvmWHvOSEMUTzw_CNmoseA

However, this is excruciatingly slow, I don't know if it's an issue of stride length?

Also, I'm not sure how to implement the thing about reshaping the combined minibatches and getting predictions from there. 

Anyone who has tried to adapt BERT for longer texts, could you please help?

Thanks


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@alvin-leong did this make any progress? I am interested in doing something similar for the sequence classifier. 

@alvin-leong @maxzzze , just wondering if there is any progress on this? 
Thanks.

No 😞 I ended up implementing several methods to grab different spans of length X (hyperparamter) from all over my documents which have lengths > 512.

Thanks @maxzzze . I am doing the same now, and just concerned about the answers from different spans. You may have right answer with low scores and wrong answer with high scores. Finger cross not. Not sure if scores from different spans is comparable?!

I modified https://github.com/kyzhouhzau/BERT-NER to implement strides https://github.com/anupamsingh610/bert_ner_stride

Hope it helps :)

@alvin-leong Hi, I have you made any progress on this issue? I checked your notebook. What is the problem with that? Is just the train very slow?

Hi, we have implemented BERT binary classifier for longer texts [here](https://github.com/mim-solutions/bert_for_longer_texts).
Adding Type Hints to TF_Pegasus model
Author: pmollerus23
Pull request: yes

# What does this PR do?

Added type hints for remaining `call()` functions for the Pegasus model (changes made only in the models/pegasus/modeling_tf_pegasus.py file). 


Fixes # [(16059)](https://github.com/huggingface/transformers/issues/16059)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
[approval](https://github.com/huggingface/transformers/issues/16059#issuecomment-1441895896)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @mollerup23, mostly looks good! One thing to watch out for is that in some cases the default value of the argument has been changed. It's easy to see if you look in the GitHub "Files changed" tab (see the image - `return_dict` had its default argument changed)
![image](https://user-images.githubusercontent.com/12866554/223161978-7abf2912-865d-4975-80bb-f38899440350.png)

If you fix the instances where that happened and double-check that it's all okay in the Files Changed tab, we should be good to go!

Hi @Rocketknight1, I updated and committed again. Hopefully these fixes help, let me know if there is anything else I should do!
Transformers `pipeline` documentation is missing text2text_generation pipeline
Author: mathcass
Pull request: no

**Is your feature request related to a problem? Please describe.**

I had the question, "How can I use the new Seq2Seq model I've trained in a Transformers pipeline?" There's pipeline tasks for summarization, generation, etc, but nothing listed on this page for how to use the model I've trained. (I'm training a T5 model.) https://huggingface.co/docs/transformers/v4.26.1/en/quicktour#pipeline

I discovered by digging into the source code that the pipeline I want is [text2text_generation](https://github.com/huggingface/transformers/blob/ea55bd86b9a452c87c5383afc707ab7d710a3043/src/transformers/pipelines/__init__.py#L272-L278).

**Describe the solution you'd like**
It'd be great if this pipeline were listed on https://huggingface.co/docs/transformers/v4.26.1/en/quicktour#pipeline

Is it intentionally omitted or is that just an oversight?

**Describe alternatives you've considered**

Digging into the source code solved my problem.

**Additional context**
NA


Comments:
cc @stevhliu and @MKhalusova 

Thanks for the feedback! I left the `text2textgeneration` pipeline out intentionally because I didn't want to make the table in the Quicktour super long with all the supported pipelines (I think we have 25 pipelines now!). I intended for it to give you a quick idea of some of the example tasks you can do with the pipeline, and if you're interested in seeing all the supported tasks, you can check out the [Pipeline API Reference](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines). 

I think there are two things we can do to improve:

1. List all the pipelines in the table for completeness.
2. Make it super clear this isn't a comprehensive list of all the pipelines; for that, you should head over to the Pipeline API docs.

Thanks @stevhliu ! Useful to know that this was an intentional decision and that all the available pipelines are under the API reference. 

If you went with (1), that might help with completeness, but could be hard to keep up-to-date, since documentarians would need to remember to update. 

For the reason above, I like (2) more than (1). However, I would caveat it with the feedback that I think there's already too many items on the table currently. If you move forward with (2) by making it clear that more exist, I might recommend removing a few of them to trim it down.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closed in #21607.
WhisperForAudioClassification can't be prepared for int8 training 
Author: Nikhil-Paleti
Pull request: no

### System Info

System: Google Colab Pro

latest version of transformers installed through:
!pip install -q git+https://github.com/huggingface/transformers.git@main

### Who can help?

The code that I used is: 

```
from transformers import WhisperFeatureExtractor
from transformers import WhisperTokenizer
from transformers import WhisperProcessor
from transformers import WhisperForAudioClassification

# Select CUDA device index
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
model_name_or_path = "openai/whisper-small"

feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path)
processor = WhisperProcessor.from_pretrained(model_name_or_path)

model = WhisperForAudioClassification.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto" , num_labels=2, label2id=label2id, id2label=id2label)

from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
```

and the error I got is :
![image](https://user-images.githubusercontent.com/68870951/224712923-41d626a7-e326-4db4-aca4-9fda26ee0992.png)


@sanchit-gandhi


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Open colab
2. Install latest version of transformers and peft 
```
!pip install -q git+https://github.com/huggingface/transformers.git@main
!pip install git+https://github.com/huggingface/peft.git@main
```
3. use the code
```
from transformers import WhisperFeatureExtractor
from transformers import WhisperTokenizer
from transformers import WhisperProcessor
from transformers import WhisperForAudioClassification

# Select CUDA device index
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
model_name_or_path = "openai/whisper-small"

feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path)
processor = WhisperProcessor.from_pretrained(model_name_or_path)

model = WhisperForAudioClassification.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto" , num_labels=2, label2id=label2id, id2label=id2label)

from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
``` 

### Expected behavior

Expected to get a model which is prepared for int 8 training 

Comments:
cc @younesbelkada  @pacman100  

https://github.com/huggingface/transformers/pull/22133 should fix the issue
[`Whiper`] add `get_input_embeddings` to `WhisperForAudioClassification`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/peft/issues/173

Fixes: https://github.com/huggingface/transformers/issues/22131

This PR adds `get_input_embeddings` method to `WhisperForAudioClassification` and `WhisperForConditionalGeneration` to avoid some issues such as [here](https://github.com/huggingface/peft/issues/173)

In my understanding `get_input_embeddings` method should return the first module that converts the input to the first hidden states and not necessarly an `nn.Embedding` layer

cc @ArthurZucker @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes I can confirm the script provided by the user:
```python
from transformers import WhisperFeatureExtractor
from transformers import WhisperTokenizer
from transformers import WhisperProcessor
from transformers import WhisperForAudioClassification

# Select CUDA device index
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
model_name_or_path = "openai/whisper-small"

feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path)
processor = WhisperProcessor.from_pretrained(model_name_or_path)

model = WhisperForAudioClassification.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto")

from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
```
work perfect now with this fix!

I am a bit surprised by the fix, as it is not an embedding layer, this kind of break the usages we have for `get_input_embeddings`, which are for example `_resize_token_embeddings` and `tie_weights` which are both incompatible with the Whisper encoder as it does not have an embedding layer. 
So not really sure this is the way to go.
Also, the `test_model_common_attributes` has to be updated
```python 
    # WhisperEncoder has no inputs_embeds and thus the `get_input_embeddings` fn is not implemented
    def test_model_common_attributes(self):
        pass
```
as well as :
```python 
    # input embeds is meaningless for an encoder-only acoustic model
    def test_inputs_embeds(self):
        pass
```
in whisper tests


Even if it's not text, the layer converts an input to a hidden size, so it's kind of like an embedding. Not sure if there is a better name for this function but it doesn't shock me to use this for Whisper, esp since peft needs a common API across models.

As for the potential problems in `_resize_token_embeddings` and `tie_weights` I would wait for a user to actually raise an issue on this before giving them more thought. We can always override and add specific error messages.



@ArthurZucker Added the common tests and there was an issue when running whisper without `decoder_input_ids` and `decoder_embeds` instead that I fixed. Can you please double check ? Thanks
Trainer: let generate pick its inputs
Author: gante
Pull request: yes

# What does this PR do?

Trainer's `predict_with_generate` seems to have been designed for an older version of `.generate()`, where manual selection of the inputs was needed. The current version of `.generate()` can do it on its own. This is particularly relevant for multimodal models, which can take more than one modality as input. As such, this PR removes the `.generate()` input selection logic from Trainer.

This PR is a requirement for Amazon's [MM-CoT](https://github.com/amazon-science/mm-cot).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[trainer] fix bug in grad accum with multiple epochs
Author: stas00
Pull request: yes

Please see https://github.com/huggingface/transformers/issues/22082 for the analysis printout of the problem.

But basically we have a bug in grad accum machinery when `steps_in_epoch % gradient_accumulation_steps != 0`

we always check for `step+1 % gradient_accumulation_steps != 0` and when we hit the epoch boundary we end up running more than `gradient_accumulation_steps` in that iteration.

I proposed a fix using a total step counter - please feel free to suggest a different fix.

I left the debug prints if you'd like to validate the situation yourself. will remove when happy.

Fixes: https://github.com/huggingface/transformers/issues/22082


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Remove backend check for torch.compile
Author: sgugger
Pull request: yes

# What does this PR do?

Since many of the choices in the list of backends for `torch.compile` do not work, this PR removes any check on the backend selected and let PyTorch itself errors if not happy. This also cleans up a bit the integration and mark everything as experimental.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Safetensors] Add explicit  flag to from pretrained
Author: patrickvonplaten
Pull request: yes

## Give user more control when loading safetensors

It would be very helpful if we could give the user more control over loading safetensors checkpoints by adding a `use_safetensors` flag to `from_pretrained`. 

At the moment, `safetensors` weights are **always** loaded when `safetensors` is installed and are **silently** not loaded if `safetensors` is installed, **but** the model has no `safetensors` weights. 

By giving the user the option to do `use_safetensors=True/False` we enable two new use cases:

1.) If a user only wants to load models from safetensors checkpoints, they can now pass `use_safetensors=True` which will lead to an error if no safetensors checkpoints are available => this can then e.g. give the user the guarantee that no pickle is used when loading checkpoints

2.) If a user doesn't want to load `safetensors` checkpoints, but not uninstall the library, they can now pass `use_safetensors=False` which will never load safetensors checkpoitns. This is super helpful for testing as well.

Also this feature would unblock this PR: in `diffusers`. https://github.com/huggingface/diffusers/pull/2123

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Prepare daily CI for torch 2.0.0
Author: ydshieh
Pull request: yes

# What does this PR do?

~~⚠️⚠️ Don't merge before I run it (again) ⚠️⚠️~~

This PR changes docker files / workflow files to use the upcoming torch `2.0.0`.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Whisper does not respect `config.forced_decoder_ids`
Author: sanchit-gandhi
Pull request: no

### System Info

- `transformers` version: 4.27.0.dev0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.13
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.6.3 (cpu)
- Jax version: 0.4.1
- JaxLib version: 0.4.1

### Who can help?

@sanchit-gandhi @ArthurZucker

### Reproduction
Previously, the recommended method for setting the language and transcription for Whisper inference was by setting the `config.forced_decoder_ids`. However, this method no longer works on the main branch, for example if we force the model to generate in French:

```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

# load model and processor
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]
input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features 

# set the forced ids
model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(language="french", task="transcribe")

# generate token ids
predicted_ids = model.generate(input_features)
# decode token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
print(transcription)
```
**Print Output**:
```
['



 Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.
']
```

### Expected behavior

Language token should correspond to the forced decoder token id (e.g. `"
"` in this example). This was the case in previous transformer versions, so there has been a breaking change that we need to remedy.

Comments:
This is based on the behavior that we enforce through the generate function. If there is a `generation_config`, it will be used and has priority over the `config`. 
There should be a warning deprecating the control of generation through the `config`. See [here](https://github.com/ArthurZucker/transformers/blob/main/src/transformers/generation/utils.py#LL538C17-L538C17).
This is a minor breaking change but should also be adresses with deprecation cycle
Add ConvNeXT V2
Author: alaradirik
Pull request: yes

# What does this PR do?
Adds [ConvNeXT V2](https://arxiv.org/pdf/2301.00808.pdf) to transformers, including a backbone. ConvNeXT V2 features minimal changes to the `ConvNextLayer` and achieves an average 1% accuracy gain over ConvNext V1. Original repo is over [here](https://github.com/facebookresearch/ConvNeXt-V2).

- [X ] Upload ImageNet 1K fine-tuned models
- [x] Upload ImageNet 22K fine-tuned models
- [x] Update model cards
- [ ] Fix TF model bugs

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ X] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._