noamaanMulla-03/transformers-issues huggingface dataset:
If num_return_sequences>1, sample_beam may return the same result
Author: hukuda222
Pull request: no

### Feature request

I would like to change the output returned by `sample_beam` when `num_return_sequences > 1`. The current implementation executes the same process `num_return_sequences` times. 
https://github.com/huggingface/transformers/blob/080a97119c0dabfd0fb5c3e26a872ad2958e4f77/src/transformers/generation/utils.py#L1692-L1707

Therefore, when the following code is executed, 3 out of 5 will produce exactly the same output.

code:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
inputs = tokenizer(["The full name of Donald is Donald"], return_tensors="pt")

outputs = model.generate(**inputs, num_beams=5, num_return_sequences=5,do_sample=True)
print("\n".join(tokenizer.batch_decode(outputs, skip_special_tokens=True)))
```

output:
```
The full name of Donald is Donald J. Trump Jr., the son-in-law and senior
The full name of Donald is Donald J. Trump Jr., the president's son-in-law
The full name of Donald is Donald J. Trump Jr., the president's son-in-law
The full name of Donald is Donald J. Trump Jr., the son-in-law of the
The full name of Donald is Donald J. Trump Jr., the president's son-in-law
```

This behavior is undesirable and should be like normal `beam_search`, which extracts multiple sentences from the beam candidates.

### Motivation

In the current implementation, `num_return_sequences` has no reason to exist, since the almost same result can be obtained by executing N times with `num_return_sequences=1`. What the two codes below do is much the same.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
inputs = tokenizer(["The full name of Donald is Donald"], return_tensors="pt")

outputs = model.generate(**inputs, num_beams=5, num_return_sequences=5,do_sample=True)
print("\n".join(tokenizer.batch_decode(outputs, skip_special_tokens=True)))
```

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
inputs = tokenizer(["The full name of Donald is Donald"]*5, return_tensors="pt")

outputs = model.generate(**inputs, num_beams=5, num_return_sequences=1,do_sample=True)
print("\n".join(tokenizer.batch_decode(outputs, skip_special_tokens=True)))
```

If we set `num_return_sequences>1`, we want to all outputs that are guaranteed to be different, so it is preferable to behave like a normal `beam_search`.



Please let me know if there is a reason for the current implementation.

### Your contribution

It's only about 3 lines to correct and I can do it if you need.

Comments:
cc @gante 

Hey @hukuda222 👋  Thank you for opening this issue!

The situation you described makes sense, I'd be happy to accept a PR to fix it :)
Update Bark generation configs and tests
Author: ylacombe
Pull request: yes

# What does this PR do?

#25381 revealed inconsistencies in Bark generation configurations, as highlighted in PR #25386.
As discussed internally with @gante, this PR aims at resolving those inconsistencies by notably:

- using a custom `validate` function in `BarkFineGenerationConfig`, because only its temperature is used.
- changing default generation parameters of the other Bark sub-models by setting greedy as default.

I also takes this opportunity to correct two things:

- update default bark hub repositories to `suno/bark` and `suno/bark-small` instead of `ylacombe/bark-large` and `ylacombe/bark-small`
- allows `BarkFineModel.generate` to accept `temperature=1.0`. Precedently it raised an error if set to 1.0.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.


## Who can review?

Hey @gante and @sgugger, what do you think of that PR?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Enable passing number of channels when inferring data format
Author: amyeroberts
Pull request: yes

# What does this PR do?

Enables passing in the number of channels to check for when trying to infer the image's data format, instead of the hard coded (1, 3).

This is part of a series of changes which will make the processing pipelines more robust to different input image types.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Un-pin JAX from <= 0.3.6
Author: sanchit-gandhi
Pull request: no

### Feature request

JAX was pinned to <= 0.3.6 in #16808 when a minor release of JAX was published that broke Optax:
https://github.com/huggingface/transformers/blob/8b2501b4b9c5afad0dca2c964bc31b9fca09df4e/setup.py#L123
There have been 18 subsequent minor releases since (_c.f._ https://jax.readthedocs.io/en/latest/changelog.html), with v0.3.24 the latest. We should un-pin this hard requirement to allow users to install the latest version of JAX with Transformers. 

### Motivation

Should a user have JAX == 0.3.24 installed and then tries to install Transformers using pip:
```
pip install transformers[flax]
```
JAX is downgraded to 0.3.6 due to the pinning requirement. Should a user be using the latest JAX features, this then requires JAX to be _re-upgraded_ to 0.3.24:
```
pip install -U jax
```
The same holds true for JAX dependencies (e.g. Flax or Optax).

### Your contribution

Investigate all versions of JAX/Flax/Optax triplets compatible with Transformers. Black-list those that break the CI, otherwise allow users to install that version of JAX in `setup.py`.

cc @cgarciae

Comments:
Until the Jax/Flax team guarantees they will maintain compatibility in their ecosystem during releases, they should be an upper pin to avoid sudden breaks of the CI.

Very much agree @sgugger! I'll maintain an upper-bound on the versions of JAX and JAX derived libraries to ensure there aren't minor releases that break our CI 🤗 We can then deal with new versions of JAX on a version-by-version basis.

[](url)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closed by https://github.com/huggingface/transformers/pull/24791
[WIP] Adding Grounding DINO
Author: EduardoPach
Pull request: yes

# What does this PR do?

This PR adds Grounding DINO

Related to: #25423 

To-do:

- [ ] Port vision backbone
- [ ] Port Text backbone
- [ ] Port Feature Enhancer
- [ ] Port Cross-Modality Decoder

Comments:
I clearly did something wrong so I'll make a new branch and add a new PR
contrastive_search is slow
Author: insist93
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

I found `contrastive_search `to be much slower than top p sampling, about 2x, more significantly in the case of large language models .I guess the reason is that `torch.stack` and `torch.cat` are called multiple times in the method. 

Is there any way to optimize?

### Motivation

accelerate `contrastive_search` 

### Your contribution

😀

Comments:
cc @gante 

Hi @insist93 👋 

Contrastive search is more elaborate than normal sampling-based strategies. Its execution time is known to grow quickly with `top_k` -- try setting a lower value if execution speed is important on your end. The tips in [this guide](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one) may also help.

As for speeding up the method, we are short on bandwidth, so it is very low on our priorities. We're happy to include PRs, if you find a way to speed it up! :)

Thank~ i will try it😀
[DINOv2] Update pooler output
Author: NielsRogge
Pull request: yes

# What does this PR do?

Fixes #25377.

The pooler was copied from ViT and set a bit arbitrary. This PR fixes it by instead taking the final hidden state of the CLS token after the final layernorm as "pooler output", based on the [original implementation](https://github.com/facebookresearch/dinov2/blob/c3c2683a13cde94d4d99f523cf4170384b00c34c/dinov2/models/vision_transformer.py#L231-L236).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Younes was actually not involved in adding this model, it was me :D 

Feel free to merge whenever.

Oh sorry, mixing things up!
🌐 [i18n-KO] Translated `philosophy.md` to Korean
Author: TaeYupNoh
Pull request: yes

# What does this PR do?

Translated the `philosophy.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@sronger, @TaeYupNoh, @kj021, @HanNayeoniee, @eenzeenee, @sim-so 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
May you please review this PR? @sgugger, @ArthurZucker, @eunseojo

Comments:
_The documentation is not available anymore as the PR was closed or merged._

May you please review this PR? :)  
@sgugger, @ArthurZucker, @eunseojo
eval_loss returning nan
Author: NHendrickson9616
Pull request: no

### System Info

- `transformers` version: 4.28.0
- Platform: Linux-3.10.0-1160.24.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu118 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Still new to this, don't know

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Here is the set up before I run the code:
```
trained_path = '/data/user/home/nchendri/biobertuab/modelInfo'
from transformers import Trainer, TrainingArguments
training_args = TrainingArguments(
    output_dir=trained_path,
    overwrite_output_dir=True,
    num_train_epochs=32,
    per_device_train_batch_size=16,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
    optim="adamw_torch",
)
 
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dsTok['train'],
    eval_dataset=dsTok['valid'],
)

eval = trainer.evaluate()
```

### Expected behavior

It was supposed to output an eval_loss, very new at this, could have missed something in above code setting stuff up.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Having the same problems.

Only getting the following out
`({'eval_runtime': 5.4991, 'eval_samples_per_second': 65.829, 'eval_steps_per_second': 1.091, 'epoch': 2.0},)`

@glaand 

Could you provide a code snippet that could run directly? Thanks in advance!
Doc checks
Author: sgugger
Pull request: yes

# What does this PR do?

This PR keeps on cleaning some of the utils and documenting them better (with proper docstrings, type hints...)
The only real change is in the `check_repo` where this is cleaning some of the constants since we catch model with `Decoder`/`Encoder` in their names earlier on.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the many comments, @ydshieh ! Should have addressed all of them.
Generation: strict generation config validation at save time
Author: gante
Pull request: yes

# What does this PR do?

As discussed in #25389, we should be flexible at load time, but strict at save time. This PR adds validation to the `save_pretrained` in the `GenerationConfig`, and skips saving if there is any issue there.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

ok, fyi, [this script](https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k?usp=sharing#scrollTo=E0Nl5mWL0k2T) is linked on the bnb/nf4 [launch page](https://github.com/huggingface/blog/blob/main/4bit-transformers-bitsandbytes.md)

The syntax used is:
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_id = "EleutherAI/gpt-neox-20b"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map={"":0})
```
This script is copied fairly widely on youtube and google colab scripts. It started to give an error on the last transformers release.

@RonanKMcGovern even if it was crashing before, #25407 (which is already on `main`) should have fixed it. In any case, I'm running your script as we speak, to double-check 👍 

Note that this PR does not introduce new exceptions :) it simply skips saving the generation config if it is incorrect, so we don't perpetuate errors, and throws an informative warning

@RonanKMcGovern I now see what you mean, there an issue loading a generation config file when `device_map` is passed to `from_pretrained`. Found the cause and will open a fix :)

Thank you for reporting it!
WavLM paper points to UniSpeech
Author: gau-nernst
Pull request: no

### System Info

NA

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/models/wavlm/modeling_wavlm.py#L1060-L1063

https://arxiv.org/abs/2101.07597 is UniSpeech.

WavLM should be https://arxiv.org/abs/2110.13900 instead.

### Expected behavior

NA

Comments:
cc @sanchit-gandhi 

Thanks for reporting @gau-nernst! That's a great spot
[WavLM] Fix Arxiv link and authors
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #25367 by correcting the Arxiv link and paper authors for the WavLM modelling code. Note that the remainder of the docs were correct, it was just the modelling code that required updating.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: Load generation config when `device_map` is passed
Author: gante
Pull request: yes

# What does this PR do?

Thank you @RonanKMcGovern for reporting this issue.

In `PreTrainedModel.from_pretrained`, `kwargs` is getting redefined when `device_map` is passed ([here](https://github.com/huggingface/transformers/blob/944ddce8bfd09ebbbdc71fb1d116421db42149b2/src/transformers/modeling_utils.py#L2852)). Later on, when attempting to load the generation config, the original `kwargs` are expected. However, since `kwargs` was rewritten into something else and the resulting exception was being caught, loading the generation config was failing silently.

The fix is simple: don't rewrite `kwargs` :)

___________________________________
### Impact of this PR on Llama 2

⚠️ We may indeed need a patch to run llama 2 models with good default behavior -- users loading the model with `device_map="auto"` (which I suspect is the most common case) are not loading the generation config, resulting in poor default behavior.

Here is an example of a script whose behavior changes drastically after this PR:
```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", device_map="auto", load_in_4bit=True)

# After this PR loads do_sample=True and max_length=4096, before do_sample=False and max_length=20
for i in range(10):
    inputs = tokenizer(["The quick brown"], return_tensors="pt").to("cuda")
    gen_out = model.generate(**inputs)
    print(tokenizer.decode(gen_out[0]))
```

___________________________________
### Retrocompatibility note
In the model `.from_pretrained`, the generation config needs to receive `kwargs` the same way the model config does. For retrocompatibility purposes, we need to accept things like

```py
model = AutoModelForCausalLM.from_pretrained("gpt2", temperature=0.9) 
```

and the extra parameter should be loaded in the generation config.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix issue with ratio evaluation steps and auto find batch size
Author: muellerzr
Pull request: yes

# What does this PR do?

Modifies step ratios in the case of when `auto_find_batch_size` is used, otherwise it will still maintain the old ratio step (so if we went from 10% starting at 100 steps, at 1000 steps it would still try and evaluate at step 10 instead of step 100)

Fixes # (issue)

Solves #24248 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger correct, those are the only places. (There are references in the tensorflow class, however I'm unsure if they need the migration or not). 

What other aspects of the trainer should we look for when determining if it should go into the state?

Borked the rebase 😭 Will open a new PR

New PR opened in https://github.com/huggingface/transformers/pull/25436

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25390). All of your documentation changes will be reflected on that endpoint.
Add `examples`  to tests to run when `setup.py` is modified
Author: ydshieh
Pull request: yes

# What does this PR do?

#25095 modified `setup.py`, but `examples` job is not triggered, see [here](https://app.circleci.com/pipelines/github/huggingface/transformers/70003/workflows/9d952427-a622-429d-8ad1-155b4523048b)

We should include it, right?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Loading GPT-Neo-2.7B has error
Author: YIYANGCAI
Pull request: no

### System Info

transformers==4.28.1, torch==1.13.1, dgx-A100, python=3.8.15

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
model_name = "/models/gpt-neo-2.7B"
model = AutoModelForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True)
```
Then I get following bug:
```
OSError: Unable to load weights from pytorch checkpoint file for '/models/gpt-neo-2.7B/pytorch_model.bin' at '/models/gpt-neo-2.7B/pytorch_model.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
```

However, for gpt-neo-125m and gpt-neo-1.3b, no bug occurs. 

Could you please help me with this issue? Many thanks!

### Expected behavior

Load the model successfully.

Comments:
Hi @YIYANGCAI 

Do you have a checkpoint in your local machine with path `/models/gpt-neo-2.7B`?

yes, I pre-downloaded it at this path.

could you check if you have absolute path `/models/gpt-neo-2.7B` or you intend to use relative path?



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.