noamaanMulla-03/transformers-issues huggingface dataset:
Unexpected behaviour of logit processor during beam search generation in Flax
Author: giganttheo
Pull request: no

### System Info

* `transformers` version: 4.38.2

### Who can help?

@sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When running beam search with a Flax model, *ie* calling `.generate` with `do_sample=False` and `num_beams>1`, the logits processors are supposed to be called with the running sequences in order to perform some processing of the logits based on the candidate sequences. However, the sequences that are passed are the ones initialized at zero, not the current `state.running_sequences`, resulting in an unexpected behaviour of the logits processors.

The line containing the error is https://github.com/huggingface/transformers/blob/b340d90738fa14bd6f81b65e4148173cbec62ff6/src/transformers/generation/flax_utils.py#L914

### Expected behavior

The current candidate sequences should be passed to the logits processor for it to perform its expected behaviour.

Comments:
Also cc @gante
Fix docker image build for `Latest PyTorch + TensorFlow [dev]`
Author: ydshieh
Pull request: yes

# What does this PR do?

It's unknown why the image/layers could not be exported and pushed. The best guess is that it is way too large for which I don't know the limitation.

This PR combines several steps in one single `RUN ...` command in the dockerfile so the size is smaller.

Successful build: https://github.com/huggingface/transformers/actions/runs/8365229494/job/22902558338

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29764). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hm, the last run on `main` works now despite this PR hasn't been merged.
[`LlavaNext`] Fix llava next unsafe imports
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes the imports of LlavaNext and the auto-mapping. this was pushed to the release

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29773). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cast bfloat16 to float32 for Numpy conversions
Author: Rocketknight1
Pull request: yes

There is no direct conversion from Torch <-> TF, so passing tensors between the two requires us to go through Numpy. Unfortunately, Numpy doesn't support `bfloat16` - this patch fixes an issue when TF tries to load a PyTorch checkpoint where weights have been stores in `bfloat16`, by upcasting the weights to `float32` before the Numpy conversion - they can be downcast later when they're assigned as the TF weights.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29755). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts thankfully, the NumPy values are never assigned directly as TF weights! The way our weight loading works, a TF model is first created with random weights, and then we loop over `zip(tf_weights, state_dict_weights)` and assign the NumPy values to the TF weights.

As a result, the exact dtype we use to load the weights from PyTorch doesn't matter, as long as it doesn't lose any precision (which is why I upcast to float32 for safety here). If the TF model is `bfloat16`, then it'll get downcast to that again when it's assigned.

That said, our support for full-bfloat16 TF models is still a little shaky, but fixing that is probably a separate PR!

@Rocketknight1 Great! All we need is a test then we're good to go 🚀 

@amyeroberts test added!
Silence deprecations and use the DataLoaderConfig
Author: muellerzr
Pull request: yes

# What does this PR do?

Now that accelerate v0.28.0 is out, removes the super annoying future warnings and opts to use the `DataLoaderConfig` 

![image](https://github.com/huggingface/transformers/assets/7831895/72985ebb-dc82-4922-9fe3-7086efd703af)

Follow up to https://github.com/huggingface/transformers/pull/28664


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@pacman100 @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29779). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add deterministic config to `set_seed`
Author: muellerzr
Pull request: yes

# What does this PR do?

Allows for deterministic algorithms to be reproducible. Can come at the cost of things being slower. 

Requested in internal slack thread: https://huggingface.slack.com/archives/C038TF8B7UZ/p1711016602277049


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
run_mlm example is missing `block_size` and `torch_dtype` args (present in run_clm)
Author: galtay
Pull request: no
Labels: name: Examples

name: Good First Issue

name: Feature request

### System Info

- `transformers` version: 4.39.0.dev0
- Platform: Linux-6.5.0-25-generic-x86_64-with-glibc2.35
- Python version: 3.11.8
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

inspection of args in https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py
vs args in https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py



### Expected behavior

the ability to use `torch_dtype` and `block_size` args in the `run_mlm.py` script as they are used in the `run_clm.py` script. 

Comments:
Hi @galtay, thanks for opening this feature request! 

Would you like to open a PR to add these?

thanks @amyeroberts agree this is more a feature request than a bug ;) will send a PR if I get the time ... was also thinking to try and update the examples so you can run with pre-tokenized data
Generate: remove legacy generation mixin imports
Author: gante
Pull request: yes

# What does this PR do?

Removes the legacy generation mixin imports, scheduled to be removed on `v4.40`. 

Side effect: a set of warnings removed from `make fixup` 🙌 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29782). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llama: always convert the causal mask in the SDPA code path
Author: gante
Pull request: yes

# What does this PR do?

Removes the `if` condition to apply the used-defined `attention_mask` on `causal_mask`: it is not required for correctness, and it prevents correct left-padding behavior in compile mode (related PR: https://github.com/huggingface/transformers/pull/29374).

I could not observe any performance degradation with eager dynamic cache nor with compiled static cache 🙌 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29663). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Change in-place operations to out-of-place in LogitsProcessors
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #29551 . In fact we do not really fix it. As @gante already explained why the logits and scores for contrastive decoding are same in that setting, because we did not apply any logits processors.

This PR changes all in-place operation on `scores` to out-of-place, so that the `logits` and `scores` are actually different when logits processors are used. Actually, this is mostly copy-paste from PR for `compile` compatibility, we also has to get rid of in-place operations there.  

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29680). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add support for metadata format MLX
Author: alexweberk
Pull request: yes

Add support for loading safetensors files saved with metadata format mlx.

Fixes #29333 



Comments:
This doesn't work in my tests when loading using `from_pretrained`. It raises here: https://github.com/huggingface/transformers/blob/45c065109074d60c587d3e562f16531d02a422f6/src/transformers/modeling_utils.py#L3301

In addition, it requires `mlx` to be installed, which should not be necessary, as described in #29511 (sorry, I did not see this PR when I created mine).
[Gemma] The weights of `embed_tokens` and `lm_head` in GemmaForCausalLM are not seem to be tied
Author: oouya
Pull request: no

Although Gemma is implemented to share weights between `embed_tokens` and `lm_head`, the Hugging Face Transformers implementation of Gemma does not seem to follow this design. 

### google-deepmind's gemma implementation
https://github.com/google-deepmind/gemma/blob/84ed38b4adbc108473ccecf6f85052eddf24f49b/gemma/transformer.py#L186-L200

### [gemma-7b](https://huggingface.co/google/gemma-7b/tree/main) / model.safetensors.index.json
https://huggingface.co/google/gemma-7b/blob/359f554e6b7dcfe8026a8705a20ff166141a5adf/model.safetensors.index.json#L5

### huggingface transformer's gemma implementation
https://github.com/huggingface/transformers/blob/f4364a6ff16e33186cb40f1d3fafd3792556d1b8/src/transformers/models/gemma/modeling_gemma.py#L1030C1-L1058C26


Comments:
check 1031 line in code block that you highlight.
`_tied_weights_keys = ["lm_head.weight"]` means weight tie.

> check 1031 line in code block that you highlight. `_tied_weights_keys = ["lm_head.weight"]` means weight tie.

It was my misunderstanding. I have confirmed that embed_tokens.weight and lm_head.weight have the same values. Thanks.
Huggingface Agents Error 422: {'error': 'Input validation error: `max_new_tokens` must be <= 192
Author: dashapetr
Pull request: no

### System Info

Transformers v4.29.0, v4.36.2

### Who can help?

@ArthurZucker, @younesbelkada

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run in Google Colab:

```
#@title Setup
transformers_version = "v4.36.2"  # you can use "v4.29.0", the issue and output are the same

print(f"Setting up everything with transformers version {transformers_version}")

!pip install huggingface_hub>=0.14.1 git+https://github.com/huggingface/transformers@$transformers_version -q diffusers accelerate datasets torch soundfile sentencepiece opencv-python openai

from huggingface_hub import notebook_login
notebook_login()

from transformers import HfAgent

agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder", token='hf_my_token')  # token is passed directly here to avoid the issue https://github.com/huggingface/transformers/issues/28217
agent.run("Is the following `text` (in Spanish) positive or negative?", text="¡Este es un API muy agradable!")
```


### Expected behavior

It should generate results, but instead, I am getting an error:

`ValueError: Error 422: {'error': 'Input validation error: `max_new_tokens` must be <= 192. Given: 200', 'error_type': 'validation'}`

![image](https://github.com/huggingface/transformers/assets/54349415/a6f7b720-befe-41b9-8973-d40d717a5b15)


To my mind, it seems like it could be related to a strict limitation on max_new_tokens [here](https://github.com/huggingface/transformers/blob/a7cab3c283312b8d4de5df3bbe719971e24f4281/src/transformers/tools/agents.py#L640C40-L640C40)

Comments:
Hey! Thanks, you are probably right. Would you like to open a PR to change this and make it more friendly? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`quality`] update quality check to make sure we check imports 😈 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update the quality check

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix type hint for train_dataset param of Trainer.__init__() to allow IterableDataset.  Issue 29678
Author: stevemadere
Pull request: yes

Prevents introducing confusion for developers using PyRight and other type checking
systems when they try to use Trainer.train() as documented with an IterableDataset
for training.

No new dependencies.

Fixes #29678


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



@muellerzr , @pacman100


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29738). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enable AMD docker build CI
Author: IlyasMoutawwakil
Pull request: yes

# What does this PR do?

This PR re-enables the CI that build AMD docker images

successful build: https://github.com/huggingface/transformers/actions/runs/8388045121/job/22971789820

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29803). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: remove unused attributes in `AssistedCandidateGenerator`
Author: gante
Pull request: yes

# What does this PR do?

I was working on another PR and noticed these attributes were unused 🤗 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29787). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`cleanup`] vestiges of causal mask
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Clean causal mask vestiges.
Slow test are catching this!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging so I can prepare the patch - thanks again!
Support batching sequence labels pairs for zero shot classification pipeline
Author: ABasharEter
Pull request: no
Labels: name: Core: Pipeline

name: Feature request

### Feature request

Currently zero shot classification pipeline dose not support specifying different set of labels for each sample input when batching multiple sequences of input. We want support input of a pair of sequence and its labels so that we can run prediction on them in batch.

Following is an example of what sample input we want to specify:

```
samples = [
("Iphone 15 pro max", ["mobiles", "appliances", "toys"]),
("Iphone 15 pro max", ["Samsung", "Apple"]),
("galaxy s21 ultra", ["mobiles", "appliances", "toys"]),
("galaxy s21 ultra", ["Samsung", "Apple"])
]
```


### Motivation

By grouping labels with sequences pairs we can batch different type of labels for each sample and by batching it runs on gpu will be faster.

### Your contribution

I will work on this feature and create a pr for it then update this issue. Making the issue now just to make sure that no one is working on this while I am doing it.

Comments:
Extremely slow model loading for BNB 4bit saved weights
Author: aliencaocao
Pull request: no

### System Info

- `transformers` version: 4.39.0
- Platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.28.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): 2.16.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@SunMarc @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am converting the original llava1.6 mistral 7b into BNB 4 bit saved model by inserting `model.save_pretrained()` at https://github.com/haotian-liu/LLaVA/blob/7440ec9ee37b0374c6b5548818e89878e38f3353/llava/model/builder.py#L114
The saved model is mere 4.6GB: https://huggingface.co/panoyo9829/llava-v1.6-mistral-7b-bnb-4bit
However, it takes a good 77.8 seconds on a AMD 5800X (8core16 thread) to load it. It takes 9.2seconds on a 80 core Intel Xeon CPU to load. This seems too slow. I am on a NVMe SSD but I see close to 0% disk IO.

### Expected behavior

When loading fp16 models, it take less than 30 seconds and disk IO is very much occupied, while when loading a BNB 4bit model, i see almost 0 disk IO.
Something related to loading BNB model must have been affecting it a lot.
Possible regression from https://github.com/huggingface/transformers/pull/26037

I am fully aware that HF now supports llava-next natively and I have yet to try saving bnb 4bit weights from the hf converted weights, but my point is this is not related to the model itself since fp16 loads fine.

Comments:
My bad, was actually a WSL2 bug...https://github.com/microsoft/WSL/issues/4197

But as suggested by https://www.reddit.com/r/bashonubuntuonwindows/comments/li9c7k/has_anyone_been_able_to_fix_the_delay_with_wsl/, maybe HF was querying git status
Llava 1.5 doesn't work with transformers 4.39.0
Author: xwjiang2010
Pull request: no
Labels: name: Should Fix

name: bug

name: Multimodal

### System Info

- `transformers` version: 4.39.0
- Platform: Linux-6.2.0-1017-aws-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.21.2
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import LlavaForConditionalGeneration
model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf")
```

### Expected behavior

run through without error

Comments:
Hi @xwjiang2010, thanks for flagging! 

We'll merge in a fix + patch release soon

Closing as this is resolved by #29389