noamaanMulla-03/transformers-issues huggingface dataset:
Sinusoidal positional encodings overwritten by post_init()
Author: hovnatan
Pull request: no
Labels: name: Should Fix

name: Good First Issue

I am trying to use DistillBert with sinusoidal positional encodings. At https://github.com/huggingface/transformers/blob/e68ff304194b4fba17cd13dc32fc3d3d61e9c0a7/src/transformers/models/distilbert/modeling_distilbert.py#L713 aren't sinusoidal positional encodings overwritten by `init_weights()` at line 632? 

Comments:
Hi @hovnatan, thanks for raising this! 

Indeed, this doesn't seem desirable at all. My suspicion is that this hasn't been caught as: 
* Most people are loading in pretrained weights, in which case they won't be randomly initialized
* The default in the config is `sinusoidal_pos_embds=False`, so the weights should be randomly initialized

As a sense check, I looked at the weights for the embeddings under different settings, and can confirm that if `sinusoidal_pos_embds=True` is only correct if we disable the post_init step

![sinusoidal_embeddings](https://github.com/huggingface/transformers/assets/22614925/f4552d1f-688b-4c04-8fd2-58d4f42ff759)

In fact, searching through the library, it looks like this might be an issue with a few other models. Specifically: 
* Flaubert
* Informer
* XLM 

@hovnatan Would you like to open a PR to fix this? 

cc @ArthurZucker @younesbelkada as it effects language models. 

- Flaubert
- Informer
- XLM

would still benefit from this! 

Should I try fixing Flaubert, ... ?

Fixed in Flaubert, Informer, XLM. The fixes are somewhat different for them, since the code structure is different.
[Flax] Migration from frozen to regular dicts with v0.7.1+
Author: sanchit-gandhi
Pull request: no
Labels: name: WIP

### Feature request

As of version 0.7.1, Flax defaults to returning **regular dictionaries** with the methods `.init` and `.apply`, not **frozen dictionaries** as was the case before: https://github.com/google/flax/discussions/3191

The `.init` method is called in the Transformers method `model.init_weights`, where we randomly initialised the model's parameters:
https://github.com/huggingface/transformers/blob/4ab5fb8941a38d172b3883c152c34ae2a0b83a68/src/transformers/models/gpt_neo/modeling_flax_gpt_neo.py#L370

Therefore, this Flax update is a breaking change for Transformers: previously, calling `model.init_weights` returned a frozen dict of params, whereas now it returns a regular dict. However, blindly reverting to using frozen dicts might cause issues for Flax users, since they will get regular dicts of params from Flax, but get frozen ones from Transformers.

This leaves us with two options:
1. Update the `model.init_weights` method to always return a frozen dict, even in the `module.init` returns a standard dict. This mitigates the breaking change and reverts to the behaviour we had before
2. Follow the Flax behaviour and return regular dicts of params with v0.7.1+. This would keep Transformers in-line with the latest Flax philosophy, at the expense of a breaking change

 A PR to implement 1 is in #28367: it is a single line change for each of the Flax modelling files. To implement 2, we would need to check if the `random_params` return by the `module.init` method are frozen or not, and match the dictionary type on the returned outputs.

Note that the change in behaviour will only really affect users who are initialising parameters themselves (with `_do_init=False`). These are typically advanced users who are familiar with the Flax library, and want an easy way of dropping-in Transformers Flax modules into other Flax scripts. Therefore, I would be in favour of 2, in order to maintain equivalence between the Flax and Transformers libraries. For users who rely on automatic init (`_do_init=True`), there's unlikely to be any friction, since they tend not to access the model params anyway.

Comments:
cc @patrickvonplaten @ArthurZucker @patil-suraj for Flax discussions

I'm also in favor of 2.) Also given we don't have that much use in Flax, I wonder whether we should do a bigger breaking change by changing the default of `_do_init` to `_do_init=False` - it's just the correct design for Flax

+1 on this! In doing so, we could also refactor the Flax module classes such that they perform all the necessary pre and post processing (e.g. as per https://github.com/huggingface/transformers/pull/22866), rather than how it currently is with a mix of the model and modules. In doing so, the modules can be used as standalone Flax modules

Re-opening as the PR was a "quick fix" to ensure the CI passed for new model additions - leaving this issue open until we perform the bigger refactor (option 2) and align ourselves with the Flax stateless design
[bnb] Fix bug in `_replace_with_bnb_linear`
Author: SunMarc
Pull request: yes

# What does this PR do 

This PR fixes a bug in `_replace_with_bnb_linear` when we use `modules_to_not_convert`. Before this PR, we were skipping the conversion of some modules if their name starts with "model.layers.1" (e.g. "model.layers.10.self_attn.q_proj") in the case where `modules_to_not_convert = ['model.layers.1']`. 

Fixes https://github.com/huggingface/transformers/issues/29691

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29958). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Flash Attention 2 support to Musicgen and Musicgen Melody
Author: ylacombe
Pull request: yes

# What does this PR do?

Supersedes #27924

The attention tests all pass but there are no integration equivalence between the original attention models and the FA ones. I don't hear any difference in quality despite not being the same song, though.

cc @sanchit-gandhi and @amyeroberts, could you review please?




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29939). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I've also added SDPA!

cc @amyeroberts or @ArthurZucker could you review when you have time?
[Docs] Make an ordered list prettier in add_tensorflow_model.md
Author: windsonsea
Pull request: yes

Add indentations to a 1-2-3 list

cc @amyeroberts

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29949). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
include ChannelDimension.NONE + function replicate_channels
Author: rafaelpadilla
Pull request: yes
Labels: name: WIP

# What does this PR do?

Fixes #25694 

The ViT model currently doesn't support grayscale images with a (height, width) format, leading to preprocessing errors.

This PR addresses the issue with a new replicate_channels function. This function converts images in (height, width) format to a 3-channel RGB format (3, height, width), replicating the grayscale channel across all three RGB channels.

While it's possible to integrate format checks and modifications within each processing function (like resize, rescale, normalize, to_channel_dimension_format, etc.), doing so might affect other modules using these functions. To avoid potential complications, I've opted for a direct solution.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25767). All of your documentation changes will be reflected on that endpoint.
Fix 29807 sinusoidal positional encodings in Flaubert, Informer and XLM
Author: hovnatan
Pull request: yes

# What does this PR do?

Fixes huggingface/transformers#29807,  sinusoidal positional encodings overwritten by post_init(), for Flauber, Informer and XLM models

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
      https://github.com/huggingface/transformers/issues/29807
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ArthurZucker, @younesbelkada, @amyeroberts

Comments:
I moved XLM and Flaubert to `init_weight()`
Flash Attention 2 support for Wav2Vec2ForCTC
Author: Adlinga
Pull request: no
Labels: name: Feature request

Hello
Usage of "optimum.bettertransformer" for Wav2Vec2ForCTC gave me the inference speedup for ~2x. 
How do you think, is it reasonable to try to implement flash attention 2 for Wav2Vec2ForCTC?
I saw the same implementation in the Whisper family of models, and it works very well (speedup ~1.8x on batch inference).

Comments:
Tokenization Error with Xglm?
Author: goru001
Pull request: no

### System Info

transformers version -> 4.36.2

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Thanks team for the great work with this library!

I was seeing a weird issue wherein following piece of code for XGLM was leading to tokenization with `eos_token` being prepended at the start and no `bos_token`, while I was expecting it to have `bos_token` at the start and `eos_token` at the end. 

```
tokenizer = AutoTokenizer.from_pretrained("facebook/xglm-564M")

xglm_token_out = tokenizer(' I live in chd ')

print(tokenizer.decode(xglm_token_out['input_ids'])) # 
 I live in chd

print(tokenizer.bos_token, tokenizer.eos_token) # 
 

```

As a sanity check above code is working as expected for BERT. 

It'll be great if you can share pointers in case I might be missing something here. Thanks!

### Expected behavior

As explained above, I was expecting the above tokenization to be `
 I live in chd 
`.

Comments:
Hey! Sorry I think the documentation is just wrong, the results you are getting are expected: https://github.com/huggingface/transformers/blob/416711c3ea88109cf25a9c5f85b4aeee2cb831b5/src/transformers/convert_slow_tokenizer.py#L1229.
Generate: fix logits processors doctests
Author: gante
Pull request: yes

# What does this PR do?

The doctests got stale 👀 (related PR to prevent this from happening again: https://github.com/huggingface/transformers/pull/29716)

There are 2 main categories of fixes:
1. Fixes where there is randomness involved: updates seed and potentially the output of the "bad" example. I can reproduce the existing doctest results if I go back to an older version (like `v4.35`), but I don't think it's worth diving through to find the root cause, as many harmless things can change the output of sampling;
2. Whisper fixes (cc @sanchit-gandhi )

All tests are passing after these changes (`pytest --doctest-modules src/transformers/generation/logits_process.py -vv`)

Comments:
cc @zucchini-nlp, to rebase your PRs after this gets merged :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29718). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `remove_columns` in `text-classification` example
Author: mariosasko
Pull request: yes

`Dataset.remove_columns` returns a new `Dataset` object instead of modifying the input dataset in-place.

Fixes https://github.com/huggingface/datasets/issues/6700

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29351). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@mariosasko Before I ping a core maintainer, could you make sure the doc will be updated on `datasets` side?

@ydshieh I've just merged a fix in `datasets`.
Update `tests/utils/tiny_model_summary.json`
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove Phi model entries in `tests/utils/tiny_model_summary.json` as they have some layer names issue (checkpoint and model layer names not matching).

This PR is necessary for those tiny models to be recreated (automatically by our workflow). Once done, we will add them back to this file.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29941). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add dinov2 model
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


Documentation: @stevhliu and @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29272). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Transformers Generation config for ImageToText models
Author: romyschneider04
Pull request: no

### System Info

![IMG_2871](https://github.com/huggingface/transformers/assets/154731707/e4e89360-eda5-4887-9463-8217da70387b)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I have transformers version 4.37.2. When I try to send prediction to ImageToText models, an attribute error raises- “GenerationConfig object has no attribute ‘_original_object_hash’”. 
I initiate Pipeline object for the model, which is called model_pipeline, and I call  model_pipeline.forward function. I have tried to debug it, and saw that there is a call to a key in the dict which does not exist, that’s why it fails. 
The GeneratorConfig has been changed during the versions, so is the code in transformers/generation/utils. 
In the old version there is no ‘_original_object_hash’ key, and the code does not uses this key. In the new version there is no such key, but the code requests this key in order to make sure the model config was not changed. 
That’s why the program fails in version 4.37.2.
When I run the code with transformers version 4.31.0 everything seems to be working just fine. 
The code runs in GPU environment 

### Expected behavior

I would expect ImageToText models to return prediction.

Comments:

![IMG_2872](https://github.com/huggingface/transformers/assets/154731707/38e83e1e-a92f-47cf-8a0d-e6f3fbcead4f)
![IMG_2873](https://github.com/huggingface/transformers/assets/154731707/c52e947e-2793-4638-999c-1df8ae12d931)
![IMG_2874](https://github.com/huggingface/transformers/assets/154731707/8b2a1920-205e-4377-a48e-fbf880943770)
![IMG_2875](https://github.com/huggingface/transformers/assets/154731707/89ce27fb-1ccc-4e2e-b91c-ca646193c1b2)
Added pictures to demonstrate the generation_config object- its keys and values. 
As you can see there is no such key as ‘_original_object_hash’, nor the code in version 4.31.0 uses it. Unlike version 4.37.2.


Hi, thanks for the issue! For readability, feel free to copy&paste your code and errors encountered in the issue conversation.
Do you have a reproducible script for this error? For instance, the example script with your modifications? `_original_object_hash` is initialized with a hash of the config if `generation_config` is `None` only, but you seem to pass it.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`CI` / `Workflows`] Attempt to slightly refactor the slow tests workflow
Author: younesbelkada
Pull request: yes

# What does this PR do?

Complementary PR to https://github.com/huggingface/transformers/pull/29046

WIP, will keep exploring a bit

cc @ydshieh @SunMarc @LysandreJik 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29197). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This PR will be replaced by another approach, but thank you a lot @younesbelkada for initializing the work!
Swinv2ForImageClassification often outputs NaN at initialization
Author: norabelrose
Pull request: no
Labels: name: Good Second Issue

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.4.0-164-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from transformers import Swinv2Config, Swinv2ForImageClassification
import torch

with torch.inference_mode():
    cfg = Swinv2Config(image_size=56)
    model = Swinv2ForImageClassification(cfg)

    out = model(torch.rand(1, 3, 56, 56))

out.logits
```

### Expected behavior

Should not output NaN; throw an error if the implementation doesn't currently support a certain combination of architectural hyperparameters

Comments:
Hi @amyeroberts and everyone! 
Is this issue open for someone to work on? It's the first time I'm contributing to Transformers and I would be happy to help!

@miguelm-almeida You're very welcome to work on it!

Thanks! :)

Hi @amyeroberts! I believe I have solved this issue. Will open a PR just now.
Fix Qwen2Tokenizer
Author: jklj077
Pull request: yes

# What does this PR do?

This PR fixes the slow tokenizer of Qwen2, where tokens starting with `#` were ineffective. The fast tokenizer is not affected. The issue was originally raised at <https://github.com/QwenLM/Qwen1.5/issues/199>.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments: