noamaanMulla-03/transformers-issues huggingface dataset:
Faster TrOCR (in particular) and faster batch text generation (in general)
Author: IlyasMoutawwakil
Pull request: no

### Feature request

Text generation in auto-regressive decoders can be optimized by not computing next token logits for sentences that already reached `eos` or `pad`. This is a [notebook](https://github.com/IlyasMoutawwakil/Faster-TrOCR/blob/main/Faster_TrOCR_with_ONNX%2BAutoRegressive_Hack.ipynb) where I did my experiments on performances. The last part contains a modified `forward` method of the `VisionEncoderDecoder` class. One important thing to note is that it should only be used in inference (eval) and not training phase.

### Motivation

I have been using TrOCR for a while and was trying to make it faster. I tried ONNXizing it but unfortunately that only made it slower (ONNX communication bottleneck ?). I digged deeper in the source code and noticed that when generating text, it computes the next token logits of the whole batch (I had batches of different length text lines because I was using it on extracted text lines from documents) which is not necessary and only makes sense in the case of a batch of a fixed text length.

### Your contribution

I made a class where I overrode the `forward` function of the `VisionEncoderDecoder` class and it worked (half the compute time).

This is the performance of the native implementation on a 17 text line document:
```python
%timeit preprocessor.tokenizer.batch_decode( \
                                            cuda_hf_model.generate( \
                                                                   pixel_values.to('cuda'), \
                                                                   max_length=96 \
                                                                   ), \
                                            skip_special_tokens=True \
                                            )
# 1.68 s ± 2.98 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
This is the performance of the modified implementation:
```python
%timeit preprocessor.tokenizer.batch_decode( \
                                            modified_cuda_hf_trocr.generate( \
                                                                          pixel_values.to('cuda'), \
                                                                          max_length=96, \
                                                                          ), \
                                            skip_special_tokens=True)
# 894 ms ± 1.21 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
I tried it on both CUDA and CPU and the gain is even greater on CPU.

Comments:
cc @NielsRogge 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

hey @IlyasMoutawwakil is there any way we can get the confidence scores of the inferences?
Confusing example in Object Detection Task
Author: govindrai
Pull request: no

### System Info

python 10, latest datasets/transformers

### Who can help?

@sgugger, @stevhliu and @MKhalusova

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Small nit: In https://github.com/huggingface/transformers/blob/main/docs/source/en/tasks/object_detection.md#load-the-cppe-5-dataset, there is a snippet that translates bboxes onto an image:

```
for i in range(len(annotations["id"])):
    box = annotations["bbox"][i - 1]
    class_idx = annotations["category"][i - 1]
    x, y, w, h = tuple(box)
    draw.rectangle((x, y, x + w, y + h), outline="red", width=1)
    draw.text((x, y), id2label[class_idx], fill="white")
```

### Expected behavior

Wherever the example queries for the index `i - 1`, I believe it should query at index `i`.

The answer does come out to be the same since python arrays work with negative indices, but it makes the example confusing/complex because it's not clear why the approach to get the last element, then the first, second and third approach is used. 

Comments:
 @MKhalusova

Could you take a look 🙏? I see you added (or modified) this block.
[TTA Pipeline] Fix MusicGen test
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

The PR #26136 added the `sampling_rate` as a **property** to the MusicGen config, such that it is compatible with the TTA pipeline. However, using a property means it is not registered when we call `config.to_dict()`, since it is not an attribute of the config.

This PR updates the TTA pipeline to avoid converting the config to a dict, thus keeping compatibility with MusicGen.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
add option to use peft for training
Author: prathikr
Pull request: yes

This PR adds an option to use peft for finetuning text-classification task. It is required to run meta-llama/Llama-2-7b-hf which we would like to add to our internal benchmarking pipelines. Also adds an explicit padding token which resolves padding issues.

Comments:
Hey! I'm afraid we can't do this! As the doc for our exemple folder mentions:

> While we strive to present as many use cases as possible, the scripts in this folder are just examples. It is expected that they won’t work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs. To help you with that, most of the examples fully expose the preprocessing of the data. This way, you can easily tweak them. 



@ArthurZucker given we are adding an optional feature, can you explain what is the concern here? Thanks!

Hey @prathikr, @askhade, thank you for your contribution, we appreciate you opening a PR!

Arthur references this specific README for the examples: https://github.com/huggingface/transformers/tree/main/examples#examples

And more specifically:

> Please discuss on the [forum](https://discuss.huggingface.co/) or in an [issue](https://github.com/huggingface/transformers/issues) a feature you would like to implement in an example before submitting a PR; we welcome bug fixes, **but since we want to keep the examples as simple as possible it's unlikely that we will merge a pull request adding more functionality at the cost of readability.**


We are aligned with you that adding examples leveraging PEFT and parameter efficient fine-tuning would be cool, but this kind of feature can very easily blow up the size of our examples which are arguably already way too long. 

While the feature you add is optional and therefore doesn't add "bloat" to the service itself, the examples' goal is really readability and adaptability; adding too many optional features unfortunately reduces this readability.

We appreciate your understanding!
feat: adding num_proc to load_dataset
Author: pphuc25
Pull request: yes

# What does this PR do?
As the demand for LLM models continues to grow, the size of the data being downloaded can slow down the process. To address this issue, I've submitted a PR to incorporate the "num_proc" feature into the "load_dataset" function, making data loading faster and more efficient.

I would like cc @sanchit-gandhi to review my PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26326). All of your documentation changes will be reflected on that endpoint.
feat: adding gradient_accumulate for run_clm_flax
Author: pphuc25
Pull request: yes

# What does this PR do?
I've added gradient accumulation for Flax file training, recognizing that resource constraints can be a limitation. Gradient accumulation assists people in training models with large batch sizes, mitigating these limitations.
I would like to cc dual @sanchit-gandhi and @ArthurZucker for review my PR


Comments:
thank you for your information

Thanks for opening this PR @pphuc25! To expand a bit on what @ArthurZucker mentioned, the Flax examples scripts are designed to be extremely minimalistic, i.e. contain the minimum level of complexity to perform a training run with the library models. This helps make them easy to build on and expand for custom use cases, since there is very little code and complexity to comprehend. To this effect, gradient accumulation is a bit too advanced for the Flax examples, since it won't be leveraged by all users. If you're interested, you ca use Optax MultiSteps for storing mini-batch updates until the final update step: https://optax.readthedocs.io/en/latest/gradient_accumulation.html#splitting-updates-for-one-batch-over-multiple-steps. I believe this is the recommended way for implementing gradient accumulation in Flax, and is one I've tried before with good success.

This explanation is well-done and provides valuable information. I appreciate it, @sanchit-gandhi. It was very informative for me. Thank you
Add image to image pipeline
Author: LeviVasconcelos
Pull request: yes

# What does this PR do?
Adds ImageToImagePipeline.

Fixes #[25349](https://github.com/huggingface/transformers/issues/25349)

## Before submitting
- [ x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ x ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #[25349](https://github.com/huggingface/transformers/issues/25349)
- [ x ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ x ] Did you write any new necessary tests?


## Who can review?
@NielsRogge
@Narsil

Comments:
@NielsRogge, @Narsil 

Which other models could we add / support for this pipeline?

also, regarding the postprocess function,  I believe the best way would be to invoke the post_process method of the image processor when available.  WDYT?

The pipeline code looks good.

Comments addressed. Thanks!

Comments addressed. Thanks!

Tests passing

Gently pinging @Narsil here

@Narsil friendly ping here

Gently pinging @NielsRogge and @Narsil here.

rebased and tests passing, it still lacks approval from a maintainer, who can help us here @Narsil @NielsRogge  @merveenoyan ?

Licenses added.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25393). All of your documentation changes will be reflected on that endpoint.

Rebasing on main should help clean this error 😉 

Hi @ArthurZucker it should be already rebased, am I missing anything?

Let me try to re-run te workflow and see if that helps

@ArthurZucker all green :D

@LeviVasconcelos congrats on this awesome contribution, feel free to tweet/linkedin about it and we'll amplify

@NielsRogge, @Narsil, @ArthurZucker, @merveenoyan Thanks for the support and help, y'all ;). Let it be the first of many... already looking for a second contribution!
Linkedin post: 
https://www.linkedin.com/posts/leviovasconcelos_ai-machinelearning-huggingface-activity-7111044322942234624-ek0J?utm_source=share&utm_medium=member_desktop
Fixed unclosed p tags
Author: HanSeokhyeon
Pull request: yes

# What does this PR do?

Fixed unclosed p tags

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26240). All of your documentation changes will be reflected on that endpoint.
Updated convert_llama_weights_to_hf.py to support llama2 chat models and default llama director name 
Author: varunfb
Pull request: yes

# What does this PR do?

When running the ./downloads.sh the options available to download are ("7B,13B,70B,7B-chat,13B-chat,70B-chat"). The download.sh script creates the folder in the following format based on the model chosen ('llama-2' + model_size.lower())

Hugging face script does not support the above directory naming convention. Resulting in having to rename the folders required by hugging face options ("7B", "7Bf", "13B", "13Bf", "30B", "34B", "65B", "70B", "70Bf", "tokenizer_only"). 

Also, the existing options does not support 7B-chat, 13B-chat and 70B-chat. 

This scripts adds support to the following

1. llama-chat models
2. New directory naming format

## Before submitting
- [] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@chauhang @HamidShojanazeri

Comments:
Closing the request due to errors. Will create a new request post fixing the issue.
AugViT TensorFlow implementation 
Author: ushareng
Pull request: no
Labels: name: New model

### Model description

I would like to contribute AugViT Tensorflow implementation to the Transformers library. You can find the implementation in TensorFlow here https://github.com/ushareng/AugViT

I have created Model card here https://huggingface.co/tensorgirl/TFaugvit/tree/main

Kindly let me know if the above is correct.

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

TensorFlow implementation of AugViT https://github.com/ushareng/AugViT

Comments:
Fix how we get a quantization method from config
Author: Rexhaif
Pull request: yes

# What does this PR do?

Fixes #25550 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @younesbelkada and @SunMarc 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25551). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
General Question - Is it possible to do MLM on models like MPT ? If yes, Could you share any resources or relevant codebase related to it?
Author: Tarun3679
Pull request: no

### System Info

the current version should be fine.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Is there a way to use AutoModelForCausalLM using MLM.

### Expected behavior

Being able to do MLM pertaining and fine-tuning on the LLMs like MPT.

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add_tokens misses to add some certain words in tokenizer vocab
Author: Kushdesh
Pull request: no

### System Info

transformers : 4.33.2
torch: 2.1.0a0+29c30b1
Python: 3.10.12
OS: Ubuntu 22.04.3 LTS

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I wanted to add a few words in tokenizer vocabulary so that encoding it result in a single integer. This I want to use in my custom head. But for some reason I couldn't create it for few words. For example

```python
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-small", legacy=False)
tokenizer.add_tokens(['abs'])
tokenizer('abs', add_special_tokens=False)
```
prints
`{'input_ids': [703, 7], 'attention_mask': [1, 1]}`

'abs' word is not in tokenizer's vocab and it is not being added.


### Expected behavior

'abs' should have been added to tokenizer vocab and input_ids of tokenizing 'abs' should produce single integer.

Comments:
I am really sorry I found that  'abs' is already in T5Tokenizer.from_pretrained("t5-small", legacy=False). I checked by running following
```python
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-small", legacy=False)
tokenizer.get_vocab()['abs']
```
which prints 14623

Probably when we call tokenizer('abs', add_special_tokens=False) it is being tokenized using other smaller tokens which have token_ids  703 and 7 rather than tokenized by 'abs' token146323
How can i make a PR for autotokenzier to adapt RWKV world
Author: xiaol
Pull request: no

### Feature request

Ususally we use own tokenzier with the transformer pipeline, 
like this https://github.com/xiaol/Huggingface-RWKV-World/blob/fca236afd5f2815b0dbe6c7ce3c92e51526e2e14/generate_hf_cfg.py#L79C1-L79C1

So far we have a lot of models using new tokenzier, using pipeline with autotokenizer is critically needed.

How can i add new tokenizer to autotokenzier to make this pipeline smooth and peace. 

Thank you.


### Motivation

1. make everyone use RWKV world smoothly,  and RWKV v5 world is coming.
2. can support huggingface communtiy with this awesome models , make opensource more open.
3. i really don't like llama models always on the top of open llm leardboards.
4. more...

### Your contribution

I made a lots of models based on RWKV 4 world ,https://huggingface.co/xiaol , especially 128k context models.

Comments:
Hey! This kind of question should be ask on [the forum](https://discuss.huggingface.co/).
However here are two different path: 
- Use the `GPTNeoXTokenizerFast`, see [here](https://huggingface.co/RWKV/rwkv-4-169m-pile/blob/main/tokenizer_config.json)
- Host your tokenizer code on the hub using https://huggingface.co/docs/transformers/custom_models

I don't really know if the tokenizer needed specific steps for conversion cc @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.