noamaanMulla-03/transformers-issues huggingface dataset:
Returning history prompt from BarkModel.generate() 
Author: sourabharsh
Pull request: no
Labels: name: Feature request

### Feature request

Hi, 
I have noticed that the original implementation of Bark (https://github.com/suno-ai/bark) has added a feature where one can get the history_prompt for the audio being currently generated using the parameter  output_full. 
history_prompt, out_arr = generate_audio(text_prompt, output_full=True)
where history_prompt is a dict object with semantic_prompt, coarse_prompt, and fine_prompt as its keys. 

But the generate method of the  huggingface version of Bark (BarkModel) doesn't support this parameter. I tried to modify the code by creating a dict of these under the generate method but the prompts in the output prompt don't meet the criteria of a valid history_prompt to be used next time because of the mismatch in ndarray.

Even the ndarray shape is also different for semantic, coarse, and fine prompts are different in the original implementation and the HuggingFace implementation. 

Can you please help me in fixing it?

### Motivation

 I want to generate a continous long-form audio for an audiobook for a better experience. I believe this will help in helping the Suno/Bark decide the tone based on the last sentence which can not be achieved using it at a sentence level  based on a single fixed history_prompt. 


### Your contribution

I need to go through and understand why there is a difference in the shape of different prompts. If that's achieved, I can contribute with a PR. 

Comments:
FYI @ylacombe 

Hey @sourabharsh, I'd be happy to help you understand how to do that. Have you checked the difference between the input `history_prompt` that you get with the processor and the output `history_prompt` that you get with your code ?
Leverage Accelerate for object detection/segmentation models
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

### Feature request

Currently there are 6 object detection models which don't support multi-GPU training out-of-the-box. The distributed code was explicitly left out of the modeling code as they wouldn't be compatible with the Trainer API. Refer to [these lines of code](https://github.com/huggingface/transformers/blob/502a10a6f89b2919444aba68cd0def51d5ba618c/src/transformers/models/detr/modeling_detr.py#L2207-L2210) as an example.

However, now that the Trainer class uses 🤗 [Accelerate](https://huggingface.co/docs/accelerate/index) behind the scenes, we can include it now by leveraging the following code:
```
from accelerate import PartialState
from accelerate.utils import reduce

# Check that we have initialized the distributed state
world_size = 1
if PartialState._shared_state != {}:
    num_boxes = reduce(num_boxes)
    world_size = PartialState().num_processes
```

See this commit as an example: https://github.com/huggingface/transformers/pull/27990/commits/526a8b0801d075ad5f99e87fbfc5de49ea347a9a.

I'll add a list here with models to be fixed:

- [ ] DETR
- [ ] Conditional DETR
- [ ] Deformable DETR
- [ ] YOLOS
- [ ] Table Transformer
- [ ] DETA

Additionally, there are 3 segmentation models which require a similar update:

- [ ] MaskFormer
- [ ] Mask2Former
- [ ] OneFormer.

For these, the `get_num_masks` function requires an update similar to what is present in the [original repository](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169), using Accelerate.

### Motivation

Would be great to support multi-GPU training of these models leveraging Accelerate

### Your contribution

I can do this but this is a perfect opportunity for a first open-source contribution

Comments:
Hi @NielsRogge, I would like to take this up

Hi @NielsRogge is this open for just 1 person taking it up as @sam99dave already indicated or multiple contribution? If it is the latter, I can take one of this up.

Hi, it looks like the PRs above addressed all models at once, which is fine for me.

Closing as the PR above was merged.
TypeError: unhashable type: 'dict' with BGE-M3 using AutoTokenizer
Author: NirantK
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.0 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The notebook to reproduce it is here: https://colab.research.google.com/drive/1OAsds_zEjGEzkfuYlJfkSPTDw5MoVYsx#scrollTo=b1ecf0b6-db81-4da3-b47f-e31460ccfbf1

1. Install dependencies: optimum, onnx, onnx-runtime
2. Try to load the BGE-M3 using the AutoTokenizer

### Expected behavior

1. AutoTokenizer works with BGE-M3 too

Comments:
Hey! I cannot reproduce this on main:
```python 

In [3]: from transformers import AutoTokenizer

In [4]: model_id = "BAAI/bge-m3"
   ...: hf_tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.31k/1.31k [00:00<00:00, 2.65MB/s]
sentencepiece.bpe.model: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.07M/5.07M [00:00<00:00, 7.40MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.1M/17.1M [00:00<00:00, 22.4MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 964/964 [00:00<00:00, 2.80MB/s]
```
works as expected for me. Make sure to use the latest version of transformers

你的transformers库太老了，更新一下就没这个问题了，我也出现了和你一样的更新到4.37.2问题消失

Updating to the latest transformers worked. Thanks @ArthurZucker @song4875343 

Updating to `transformers==4.37.2` solved the issue for me.
```bash
pip install -U transformers
```
[i18n-fr] Translate accelerate tutorial to French
Author: NoB0
Pull request: yes

# What does this PR do?
Translates the `accelerate.md` file of the documentation to French.

Part of #21456 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

French speaking contributors.
Documentation: @stevhliu and @MKhalusova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28418). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
CausalLMOutputWithPast does not output hidden states
Author: Tiziano41
Pull request: no

### System Info

sentence-transformers      2.2.2
transformers                      4.31.0
numpy                               1.19

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This is the code I'm using : 


model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)

inputs = tokenizer('''
Name the writings of Dante Alighieri.

Answer: 
''', return_tensors="pt", return_attention_mask=False)


embeddings = model(**inputs, output_hidden_states = True)



![image](https://github.com/huggingface/transformers/assets/156085316/040a75ea-18ba-4d4b-89c9-c7336869fb67)




### Expected behavior

As you can see in the image, the hidden_states attribute is None, while I would expect the intermediate layers representations.

I tried to specify the output_hidden_states = True, both in model instantiation and on inference but neither works.

Is there any parameter that I'm missing ? That's all I have seen on the available documentation.

Thank you in advance for you support.

Comments:
Hi @Tiziano41 
Thanks for the issue! It seems you are using the `trust_remote_code=True` model, therefore the bug should be on their end. Would you mind opening a similar issue on the original repository on 🤗  Hub? https://huggingface.co/microsoft/phi-2 

Alternatively you can use the HF version of Phi-2 (i.e. without `trust_remote_code=True`) which should be here https://huggingface.co/susnato/phi-2 if I am not mistaken cc @ArthurZucker @susnato 

Yes please use it from `susnato/phi-2` until the codebase is changed and in proper order at `microsoft/phi-2` 😃 

@younesbelkada @susnato OK, Thank you both for the fast reply! I will open the issue on their repository.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
cannot import name 'TrainingArguments' from 'transformers'
Author: takfarine
Pull request: no
Labels: name: bug

### System Info

Traceback (most recent call last):
  File "dv2xxl.py", line 30, in <module>
    from transformers import TrainingArguments,Trainer
**ImportError: cannot import name 'TrainingArguments' from 'transformers'** (/lustre06/project/6005433/takfa/UW/ue/lib/python3.7/site-packages/transformers/__init__.py)


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Traceback (most recent call last):
  File "dv2xxl.py", line 30, in <module>
    from transformers import TrainingArguments,Trainer
ImportError: cannot import name 'TrainingArguments' from 'transformers' (/lustre06/project/6005433/takfa/UW/ue/lib/python3.7/site-packages/transformers/__init__.py)


### Expected behavior

looking for help 

Comments:
Hey! Could you please provide your transformers version? How did you install transformers?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Problem in "from transformers import TrainingArguments,Trainer" How to resolve it
Add HTDemucs
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Adds HTDemucs, required for the MusicGen melody model. Supersedes #25660.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28358). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix ffmpeg_microphone under WSL2 (use pulseaudio)
Author: jamon
Pull request: yes

# Fix ffmpeg_microphone under WSL2
This attempts to detect if it is running under WSL2 and defaults to using pulseaudio with an input of "RDPSource" in order to work with WSL2.

@Narsil - tagging you since you contributed most of this file.  I'm also happy to update this to just accept the format and input as parameters if you prefer (and either keep or roll back the defaults for WSL2).

Comments:
cc @Narsil @sanchit-gandhi 

Pulse or Also has nothing to do with WSL2. You could use either on regular linux too.

I think this is the point where this demo should stop trying to support everything that might exist. Giving a good error message when this triggers might be better imo.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Error while fetching adapter layer from huggingface library
Author: Muskanb
Pull request: no

### System Info

```
        pa_extractor = LlamaForCausalLM.from_pretrained(LLAMA_MODEL_NAME,
                                                        token=HF_ACCESS_TOKEN,
                                                        max_length=LLAMA2_MAX_LENGTH,
                                                        pad_token_id=cls.tokenizer.eos_token_id,
                                                        device_map="auto",
                                                        quantization_config=bnb_config)
        pa_extractor.load_adapter(PEFT_MODEL_NAME, token=HF_ACCESS_TOKEN, device_map="auto")
```
       
                            
       # getting the below error while executing :
       401 client error, Repository Not Found for url: https://huggingface.co/muskan/llama2/resolve/main/adapter_model.safetensors. Please make sure you specified the correct `repo_id` and `repo_type`. If you are trying to access a private or gated repo, make sure you are authenticated. Invalid username or password. Even though works fine while fetching models but fails at load_adapter step.
       

### Who can help?

 @Narsil @ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The model is present in my private repo, should be replicable if you will try to use load_adapter to fetch any adapter layer from hf directly.

### Expected behavior

Should be able to download the peft adapter layer successfullt 

Comments:
Hi @Muskanb, thanks for raising an issue! 

Is the error occurring when calling `LlamaForCausalLM.from_pretrained` or `pa_extractor.load_adapter`? 

Hey @amyeroberts, Thanks for responding. It occurs while calling ` pa_extractor.load_adapter`

hi @Muskanb what are your transformers & peft versions? 

These are the versions : 
 ```
peft==0.6.2
transformers==4.35.2
```


`token` argument should be supported in the latest transformers version: https://github.com/huggingface/transformers/blob/002566f398cd6dbf7053a89c26646ac45be540f4/src/transformers/integrations/peft.py#L74 , can you try to update peft & transformers? `pip install -U transformers peft`

@younesbelkada It worked. Appreciate the prompt help!! :)

Awesome thanks @Muskanb !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Very slow on conditional check of HF tokenizers
Author: pseudotensor
Pull request: no

### System Info

transformers==4.36.2
Python 3.10
Ubuntu 20


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

It's pretty pythonic to avoid "x is not None" if could have just done "if x".  I know for numpy objects this isn't same thing, but I don't know why tokenizer would do alot of extra work here.  Seems like bug.

```
import time

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)

t0 = time.time()
if tokenizer:
    pass
print(time.time() - t0)

t0 = time.time()
if tokenizer is not None:
    pass
print(time.time() - t0)

tokenizer = AutoTokenizer.from_pretrained("h2oai/h2ogpt-4096-llama2-7b-chat", trust_remote_code=True)

t0 = time.time()
if tokenizer:
    pass
print(time.time() - t0)

t0 = time.time()
if tokenizer is not None:
    pass
print(time.time() - t0)
```


```
0.0909724235534668
9.5367431640625e-07
0.0019714832305908203
2.384185791015625e-07
```


### Expected behavior

Not be 100000x slower if do:
```
if tokenizer:
    pass
```
vs. faster:
```
if tokenizer is not None:
    pass
```


Point is that I might check tokenizer many times, and actually tokenizing things is much faster than that check, which is bad.

Comments:
Hey! Thanks for raising this, I have no idea how to fix this 🤣 python is trying to convert the object to a boolean, but I don't know what's happening other than that! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Loading safetensor version of mistralai/Mistral-7B-Instruct-v0.1(7b) in Triton Server results in  cuda OOM
Author: jimmymanianchira
Pull request: no

### System Info

transformers==4.36
Triton Instance - nvcr.io/nvidia/tritonserver:23.09-pyt-python-py3
torch==2.1.2


### Who can help?

We are trying to load and serve [Mistral 7B Instruct v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) using Triton Inference server.

We initially was loading the weights using .bin and it worked fine. But we later moved to the safe tensor version. We noticed that after calling the model couple of times, its getting OOM. I did it in 4 gpu and same thing is happening. It works for the 1st inference call and after that, we end up with OOM.  Its very strange and .bin weights work normally

@ArthurZucker and @younesbelkada @SunMarc 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Load Mistral 7b in Triton Inference Server using safe tensor weights
2. Send couple of inference calls
3. Will notice OOM

### Expected behavior

Loading weights with safetensors shouldn't cause OOM.

Comments:
Thanks for reporting, pinging @Narsil for visibility, do you have any reproducers / examples of inputs that can lead to this? 

@jimmymanianchira this is quite unusual as safetensors do not interact with CUDA right @Narsil ? 
Are you sure you are using the same transformers version across your tests? How did you moved your tests to use the safetensors version? 

Safetensors doesn' t interact with it in anyway. I supposed something else is at play.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FIX [`bnb` / `tests`]: Fix currently failing bnb tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/29001 changed the logic of handling how to get linear layers from testing models. In fact, the `model_type` should always stay `"gpt2"` and not `"openai-community/gpt2"`

cc @amyeroberts @Titus-von-Koeller 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29092). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
CLIPForImageClassification
Author: Andron00e
Pull request: no
Labels: name: New model

name: Good First Issue

### Model description

CLIPForImageClassification is an add-on CLIPModel from hub. With one linear classifier on top of the CLIP it performs in Image Classification task as good as [ViTForImageClassification](https://github.com/huggingface/transformers/blob/v4.35.2/src/transformers/models/vit/modeling_vit.py#L756) does. Moreover, there are several options how to produce a classification with this model.
The first one is to use "image_embeds" output of CLIP and then produce a classification with [projection_dim, num_classes] Linear Layer. 
The second one is to use "logits_per_image" output of CLIP and then do classification.

Currently, the model implementation is partially available. With weights of it, retrained on CIFAR10 dataset. My next goal is to expand the range of labels and implement the second variant of this model.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

[Repo with implementation](https://github.com/Andron00e/CLIPForImageClassification)
[Pre-trained model on hub](https://huggingface.co/Andron00e/CLIPForImageClassification-v1)

Comments:
Thanks for the suggestion, I've review the PR above

This is now supported by #28952.
Fix the `bert-base-cased` tokenizer configuration test
Author: LysandreJik
Pull request: yes

In the process of updating the tokenizer configurations on the Hub, this test needs to be updated to reflect the new value of the configuration.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29105). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.