noamaanMulla-03/transformers-issues huggingface dataset:
DeepSpeed ZeRO-3 handling when resizing embedding layers
Author: pacman100
Pull request: yes

# What does this PR do?
1. PRs https://github.com/huggingface/transformers/pull/25394 and https://github.com/huggingface/transformers/pull/25732 resulted in 17 DeepSpeed slow tests failing. The reason is that those PRs might have missed the fact that the shape will be 0 when working with layers that have been initialized by DeepSpeed init for ZeRO-3. With this PR, all the failing tests pass.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[FIX] resize_token_embeddings
Author: passaglia
Pull request: yes

# What does this PR do?

When `resize_token_embeddings(new_num_tokens, pad_to_multiple)` is called with `new_num_tokens` a multiple of `pad_to_multiple`, the model should be resized to `new_num_tokens`. Due to a math error, it is instead resized to `new_num_tokens+pad_to_multiple`. This PR fixes that bug.

@ArthurZucker https://github.com/huggingface/transformers/pull/25088


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26102). All of your documentation changes will be reflected on that endpoint.

I'm not so familiar with the transformers repo -- where should a test for this code go?

SHould be part of [this tests](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L1364)

@ArthurZucker please confirm this test works, I couldn't run it myself since `pytest tests/test_modeling_common.py` returned 0 collected tests.

Sure, I'll test this, can you run ` make style` as well? To have green cis

@ArthurZucker Ready for merge
[docs] LLM Prompting Guide
Author: MKhalusova
Pull request: yes

# What does this PR do?

This PR adds an LLM Prompting Guide to the docs that covers the following topics: 

- basics of prompting, 
- encoder-decoder models vs decoder-only models, 
- base vs instruct models, 
- basic prompts to solve common NLP tasks, 
- best practices for prompting, 
- advanced techniques like few-shot learning and chain-of-thought
- prompting vs fine-tuning

Let me know, if there's anything missing that has to be included.

Comments:
Add BROS
Author: jinhopark8345
Pull request: no
Labels: name: New model

### Model description

[BROS(BERT Relying On Spatiality)](https://arxiv.org/abs/2108.04539)  is a pre-trained multimodal transformer for Document Understanding using OCR results of document images (text and bounding box pairs). 

and I would like to add this model to Huggingface as my first contribution!

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/clovaai/bros

Comments:
Would be great if you could add it, it should be very straightforward :)

I have a few demo notebooks on fine-tuning BROS, let me share them here:

- https://colab.research.google.com/drive/1PUpiKcSdXjBYM6a300ayC9TaYwzxdMus?usp=sharing
- https://colab.research.google.com/drive/1pTjxx4_46Sk1Zs0W_yzceP_bstmu4vfz?usp=sharing.

The first one is fine-tuning BROS on the FUNSD dataset, the second one is the same but with support for creating more training examples using the `return_overflowing_tokens` feature.

Let me know if you need any help to start contributing, feel free to start opening a draft PR
Remove redundant code
Author: hzhiyuan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Remove redundant code: The variable "has_default_max_length" will always be "False" for the 2 lines of code:

  1. model_kwargs = generation_config.update(**kwargs)
  2. has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None

So I remove the code related to it.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @gante

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Add TUNet
Author: OllieBroadhurst
Pull request: no
Labels: name: New model

# 🌟 New model addition

The TUNet model for audio superresolution

## Model description

[TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretraining](https://arxiv.org/abs/2110.13492v3) is a paper by Viet-Anh Nguyen, Anh H. T. Nguyen, and Andy W. H. Khong, introducing a model for audio superresolution based on transformers.

Audio superresolution allows for the upsampling of audio with minimal loss of quality. This is very useful for ASR tasks that require audio to be resampled during preprocessing, which has a big impact on transcriptions depending on the native sample rate.

## Open source status

* [x] the model implementation is available: [The official repo](https://github.com/NXTProduct/TUNet)
* [x] the model weights are available: [ONNX weights](https://github.com/NXTProduct/TUNet/tree/master/lightning_logs)
* [x] who are the authors: Viet-Anh Nguyen, Anh H. T. Nguyen, and Andy W. H. Khong


Comments:
cc @patrickvonplaten @anton-l 

Hi @patrickvonplaten @anton-l @NielsRogge @NielsRogge @amyeroberts @patil-suraj 
Is this model still available for contribution?
I would like to contribute to this model. Can you please guide me!
Looking forward from hearing from you all!


Hi @02shanks, 

We have recently been trying to push for `model on the hub` and have as much support as we can there. This is the recommended way of adding new models and it will also be easier to integrate it! Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

cc @sanchit-gandhi 

yaa sounds good to me. Will go through tutorial and try to implement it!
fix name error when accelerate is not available
Author: pacman100
Pull request: yes

# What does this PR do?
1. Fixes the non-torch test failures `ERROR tests/fsdp/test_fsdp.py - NameError: name 'require_fsdp_version' is not defined`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merge now to prevent further failure on main

You should also link the PR that was failing, #26180 if I am not mistaken
illegal hardware instruction
Author: Jasperty
Pull request: no
Labels: name: bug

### System Info

```shell
i use macbookpro, and pip install transformers, but i get this error:
>>> from transformers import DistilBertConfig
zsh: illegal hardware instruction  python
could you help me?
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

zsh: illegal hardware instruction  python

### Expected behavior

```shell
zsh: illegal hardware instruction  python
```


Comments:
I think this is due to you installation of tensorflow and not necessarily the library  . 
Did you try installing with 
```
conda install -c apple  tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal
```

Also it is recommended to install using [MiniForge](https://github.com/conda-forge/miniforge#miniforge3)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm having the same issue. It happened with the same code recently after installing Big Sur.

I am still getting this in macOS.
Integrate AMD GPU in CI/CD environment
Author: mfuntowicz
Pull request: yes


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh @LysandreJik I think we are in a good shape for review and merging.

What we did: 
- [x] Added custom runners with tags `docker-gpu`,  `single-gpu`, `amd-gpu`, `mi210`
- [x] Provide a custom PyTorch GPU Dockerfile for AMD dependencies
- [x] Create a new `self-push-amd.yml` workflow file for everything related to AMD testing
- [x] Validated the workflow against a simple BERT modifications 

What we cannot ensure as of today:
- [ ] All the current tests being executed on main will be green 😅 

Hi @mfuntowicz 

Looking at the runs in https://github.com/huggingface/transformers/actions/workflows/self-push-amd.yml, you will see no test job (Model test) is being triggered (as no test is being collected).

Also the slack report won't work as the tag is sitll using `single-amdgpu` instead of `single-gpu`.

@LysandreJik in case you want to take a final look :-)

Merge now so @mfuntowicz can show progress to AMD team today.
fix deepspeed available detection
Author: fxmarty
Pull request: yes

As per title, make the deepspeed available function more robust as https://github.com/huggingface/accelerate/blob/69e4c3c54da3201eda288b500d138761e7a5221c/src/accelerate/utils/imports.py#L72

Having `tests/deepspeed` & [`get_env`](https://github.com/huggingface/transformers/blob/eb8489971ac1415f67b0abdd1584fde8b659ced9/src/transformers/testing_utils.py#L1344) that adds `tests/` in the path then makes `is_deepspeed_availebl()` returns `True` although it should not, and in turn trainer.py [tries to import](https://github.com/huggingface/transformers/blob/eb8489971ac1415f67b0abdd1584fde8b659ced9/src/transformers/trainer.py#L217) `DeepSpeedSchedulerWrapper` that is [not imported in accelerate](https://github.com/huggingface/accelerate/blob/69e4c3c54da3201eda288b500d138761e7a5221c/src/accelerate/utils/__init__.py#L92) as accelerate rightfully detects that DeepSpeed is not available.

This issue makes the test `tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_apex` fail when APEX is installed but DeepSpeed is not.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Could you show us how to see the following?

> Having tests/deepspeed & [get_env](https://github.com/huggingface/transformers/blob/eb8489971ac1415f67b0abdd1584fde8b659ced9/src/transformers/testing_utils.py#L1344) that adds tests/ in the path then makes is_deepspeed_availebl() returns True

@ydshieh Reproduction (maybe add relevant `breakpoint()` and print `sys.path` as well):

```
docker run --rm -it --gpus all nvcr.io/nvidia/pytorch:23.08-py3 /bin/bash
pip list | grep apex  # apex is here!
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .[dev-torch]
pytest tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_apex -s -vvvvv
```

I tried to do it in a simple way

```python
import os
# given by `get_env`
os.environ["PYTHONPATH"] = "/transformers/src:/transformers/tests:"
# shows `/transformers/src:/transformers/tests:`
print(os.environ["PYTHONPATH"])
from transformers.deepspeed import is_deepspeed_available
print(is_deepspeed_available())
```

and it prints `False`. Therefore, I am not very certain about the issue. Even with `tests/`, still `False`

@ydshieh You would need to use: `PYTHONPATH=/path/to/transformers/tests python -c "from transformers.deepspeed import is_deepspeed_available; print(is_deepspeed_available())"` to reproduce the issue.

I believe setting `PYTHONPATH` within a python script does not change `sys.path` (or somthing like this).

For the record, to reproduce (inside our docker env)

```
PYTHONPATH=/transformers/tests python3 -c "from transformers.deepspeed import is_deepspeed_available; print(is_deepspeed_available())"
```
CodeLlama Tokenizer encoding bug
Author: jcao-ai
Pull request: no

### System Info

transformers: 4.33.0


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

There is one more space than original text, and there is a mismatch in `non-fast` version:
```python
>>> txt = '''
 # Python\ndef fibonacci(n): 


'''
>>> t_fast = AutoTokenizer.from_pretrained('/models/CodeLlama-34b-Instruct-hf', use_fast=True)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
>>> t_slow = AutoTokenizer.from_pretrained('/models/CodeLlama-34b-Instruct-hf', use_fast=False)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
>>>
>>> t_fast.decode(t_fast.encode(txt))
'
 
 # Python\ndef fibonacci(n): 

 
'
>>>
>>> t_slow.decode(t_slow.encode(txt))
'
 
 # Python\ndef fibonacci(n): 

/INST]'
```

### Expected behavior

Decoded text should be the same as the original text ?

Comments:
can confirm the inconsistency cc @pacman100 @ArthurZucker, minor differences in tokenization may not significantly impact the final results though.

I'll have a look, LlamaTokenizer does not seem to suffer from this so should be quick to fix!
include changes from llama
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #26239, where we have an edge case of the tokenization. When porting from CodeLllama a condition in `_tokenize` was ommited, which leads to merging the unk token with part of the input, and then stripping it. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Synk_Dive
Author: yasmws
Pull request: no
Labels: name: New model

### Model description

O projeto Synk Dive cria um ambiente de harmonia multisensorial que oferece uma experiência imersiva única, combinando inputs visuais e textuais. Ao fornecer um vídeo como entrada, por exemplo, o sistema gera uma trilha sonora que se adapta às emoções transmitidas em tela. Esta trilha sonora sincroniza-se com as diferentes cenas ao longo do vídeo, resultando em uma experiência que integra elementos visuais e auditivos para intensificar a imersão e a conexão emocional do usuário com o conteúdo. 


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
[`Trainer`] Refactor trainer + bnb logic
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/26207

Some users try to perform pure fine-tuning on quantized models; this is simply not supported.
Before this PR, in that case we only pass a logger.info which is not a strong enough warning for users to understand why training does not work.

cc @SunMarc @ArthurZucker 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
special token ids are different in LlamaTokenizerFast
Author: x54-729
Pull request: no

Recently we are working on a model based on Llama. When I;m trying to convert our `tokenizer.model` to `tokenizers`, a problem occurs.
`tokenizer.model`'s vocab is expanded to fit our model:

```python
from transformers import LlamaTokenizer, LlamaTokenizerFast
from sentencepiece import SentencePieceProcessor

f = LlamaTokenizerFast("tokenizer.model")
s = LlamaTokenizer("tokenizer.model")
origin = SentencePieceProcessor("tokenizer.model")

print(f.bos_token_id, f.eos_token_id, f.unk_token_id) # 1, 2, 0
print(s.bos_token_id, s.eos_token_id, s.unk_token_id) # 0, 1, 2
print(origin.bos_id(), origin.eos_id(), origin.unk_id()) # 0, 1, 2
```

This problem leads to different behaviors between `AutoTokenizer` and `LlamaTokenizer`. What may cause this problem?

transformers==4.31.0
tokenizers==0.13.3

Comments:
Could you try using #23909. I believe the order of token addition is wrong. 

> Could you try using #23909. I believe the order of token addition is wrong.

I'm sorry but could you please tell about more details about the solution? I didn't call `add_special_tokens` and just loaded the tokenizer model in this case.

That's the problem, the special tokens are not added to the model by default for the slow tokenizer, while they are for the fast tokenizer. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as #23909 should have fixed it
add custom RMSNorm to `ALL_LAYERNORM_LAYERS`
Author: shijie-wu
Pull request: yes

# What does this PR do?

It fixed a issue discovered during discussion of PR https://github.com/huggingface/transformers/pull/26152. 

> @ArthurZucker: the `ALL_LAYERNORM_LAYERS` should contain all the custom layer norm classes (from `transformers` modeling files) and should be updated if that is not the case

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 

Comments:
@ArthurZucker done! `MegaRMSNorm` - a part of `MegaSequenceNorm` - is already in `ALL_LAYERNORM_LAYERS` https://github.com/huggingface/transformers/blob/493b24ba109ed680f40ec81ef73e4cc303e810ee/src/transformers/models/mega/modeling_mega.py#L317

do we want to add all these to `ALL_LAYERNORM_LAYERS`?

https://github.com/huggingface/transformers/blob/493b24ba109ed680f40ec81ef73e4cc303e810ee/src/transformers/models/mega/modeling_mega.py#L294-L301

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26227). All of your documentation changes will be reflected on that endpoint.

after discussion with @ArthurZucker, it seems best to limit to scope of this PR to adding custom RMSNorm to `ALL_LAYERNORM_LAYERS`. Adding nn.BatchNorm1d might create unintentional impact.
[New model] ImageBind: One Embedding Space To Bind Them All
Author: xenova
Pull request: no
Labels: name: New model

### Model description

As stated in their [blog post](https://ai.facebook.com/blog/imagebind-six-modalities-binding-ai/), 

> "[ImageBind is] the first AI model capable of binding information from six modalities. The [model](https://github.com/facebookresearch/ImageBind) learns a single embedding, or shared representation space, not just for text, image/video, and audio, but also for sensors that record depth (3D), thermal (infrared radiation), and inertial measurement units (IMU), which calculate motion and position."



### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

GitHub repo: https://github.com/facebookresearch/ImageBind
Paper: https://facebookresearch.github.io/ImageBind/paper
Blog: https://ai.facebook.com/blog/imagebind-six-modalities-binding-ai/
Demo: https://imagebind.metademolab.com/
Video: https://dl.fbaipublicfiles.com/imagebind/imagebind_video.mp4
Weights: https://dl.fbaipublicfiles.com/imagebind/imagebind_huge.pth (currently only 1 that I can see)

Comments:
Hi @xenova , I would like to work on implementing this model.

> Hi @xenova , I would like to work on implementing this model.

Sweet!

Hi, since it looks like the PR for this model (#23284) has been closed, I would be interested in working on a new PR to implement the ImageBind model :)

I have opened a new PR to implement the ImageBind model: #26310.
[Trainer API] Mention that the total number of steps per epoch is separate from total number of steps on training run
Author: suvadityamuk
Pull request: no

### Feature request

Is there a possibility to maybe mention within the Trainer API that the number of steps mentioned during a `trainer.train()` call is actually the `num_steps_per_epoch * epochs` instead of just `num_steps_per_epoch` and also shift the Epoch note to the front?

![image](https://github.com/huggingface/transformers/assets/70141886/3bc43e57-f9e9-4491-846d-71d2535a690f)

### Motivation

This was very confusing for me as a first-time user of the Trainer API, coming from Keras and it looked like while I had 9688 samples, I had 241700 steps which looked like an error at first (spent 2 hours trying to debug my `datasets` loading :sweat_smile: behind this).

Not sure if it's a major thing, but still something I thought I'd report just in case. 

@sgugger 

### Your contribution

Not sure how I can help, but happy to make the PR if guided.

Comments:
The progress bar clearly indicates the count of epochs (here we can see 0/25 in your screenshot), I'm not sure what more we can add to make this clearer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.