noamaanMulla-03/transformers-issues huggingface dataset:
Add CAV-MAE audio-image encoder model
Author: rationalism
Pull request: no
Labels: name: New model

### Model description

Contrastive Audio-Visual Masked Autoencoder (CAV-MAE) combines two major self-supervised learning frameworks: contrastive learning and masked data modeling, to learn a joint and coordinated audio-visual representation. It appears to be the open source SOTA on the AudioSet and VGGSound datasets (the OmniVec and Facebook MAViL models seem to have never had weights released). 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/YuanGongND/cav-mae

@YuanGongND

Comments:
I totally support this proposal. The code and checkpoints are available at https://github.com/YuanGongND/cav-mae.

Created an initial PR here by cloning the visual-audio multimodal model TVLT: https://github.com/huggingface/transformers/pull/28246

(haven't actually added the CAV-MAE code yet! this is just a scaffold)
fix bug: avoid divide by zero in _maybe_log_save_evaluate()
Author: frankenliu
Pull request: yes

set logging_strategy="steps" and logging_steps=10,  
when that one epoch have 100 steps, the should_log will be set to True in last step.
And self._globalstep_last_logged will be assign to self.state.global_step in _maybe_log_save_evaluate() method by line 1917 in trainer.py.
the line 1933 in trainer.py , self.callback_handler.on_epoch_end() will keep the should_log=True, then in line 1934 run _maybe_log_save_evaluate() method  (self.state.global_step - self._globalstep_last_logged) will be zero in line 2247.

@muellerzr @pacman100 

Comments:
> Hello, thank you for the PR, but I think better solution would be to the following check in `_maybe_log_save_evaluate`:
> 
> ```
> if self.control.should_log and self.state.global_step>self._globalstep_last_logged:
> ```

Yes, your solution are better. I just think to avoid in _inner_training_loop() when training.

@frankenliu can you either try and fix the addition of all the commits (after rebasing I don't think you did a force push) or open a new clean PR? It's hard to read just what's changed with now 30+ files modified. Thanks!

> @frankenliu can you either try and fix the addition of all the commits (after rebasing I don't think you did a force push) or open a new clean PR? It's hard to read just what's changed with now 30+ files modified. Thanks!

OK，i will open a new clean PR.
Obtaining text embeddings from CLIP
Author: preethiseshadri518
Pull request: no

I am trying to obtain text embeddings from CLIP as shown below. However, I am confused about the difference between text_embeds vs. pooler_output, since they output different things. According to the documentation, text_embeds is "the text embeddings obtained by applying the projection layer to the pooler_output", but I am not sure what this means? Are both acceptable to use as text embeddings (if I want to compare text similarity), or is one more correct than the other?

```
from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)
text_embeds = outputs['text_embeds']
pooler_output = outputs['text_model_output']['pooler_output']
```

Comments:
Hi,

You can use both as text embeddings.

The former (`text_embeds`) are embeddings which are in the same embedding space as the image embeddings (so it allows you to compare images and text - which is what people mainly use CLIP for). However if you just want text embeddings, and don't care about image embeddings, then you can use the `pooler_output`.

Btw, if you only need text embeddings (and no image embeddings), it's more memory efficient to only load the text encoder of CLIP. You can choose between `CLIPTextModel` (which is the text encoder) and `CLIPTextModelWithProjection` (which is the text encoder + projection layer, which projects the text embeddings into the same embedding space as the image embeddings):

```
from transformers import AutoTokenizer, CLIPTextModelWithProjection

model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32")
tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")

inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")

outputs = model(**inputs)
text_embeds = outputs.text_embeds
```

Also, please ask such questions on our [forum](https://discuss.huggingface.co/) - we'd like to keep Github issues for bugs/feature requests.

Thanks!

Will do, apologies! Wasn't sure which was the appropriate place.
Time Series Transformers for Classification
Author: borhenryk
Pull request: no

### Feature request

Is it planed to introduce Time Series Transformers for Classification at some point?

### Motivation

Time Series Transformers are already introduced 

### Your contribution

I could contribute models 

Comments:
cc @kashif 

thanks @borhenryk I have a PR for that currently underway here: https://github.com/huggingface/transformers/pull/24803 

do you have some sample/open classification time series datasets in mind you can point me to?

sure I am working on Healthcare applications where there are many use-cases for TS classification but I think we could start wit an Human Activity Recognition Problem with this data https://archive.ics.uci.edu/dataset/240/human+activity+recognition+using+smartphones

Also happy to support with the PR wherever you need help @kashif 

awesome @borhenryk I will add this dataset to the huggingface datasets if the license allows it and then coordinate with you  the potential `TimeSeriesForClassification` model's output head


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@borhenryk  we have a classification model in #25927 

@kashif great let me know if I can support with testing or training etc. 

thanks! will ping you once it's merged!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This is now supported with models like PatchTST: https://huggingface.co/docs/transformers/main/en/model_doc/patchtst#transformers.PatchTSTForClassification.
ImportError: tokenizers>=0.11.1,!=0.11.3,<0.14 is required for a normal functioning of this module, but found tokenizers==0.15.0. Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git main Press any key to continue . . .
Author: NiuDaVinci
Pull request: no

### System Info

[DxDiag.txt](https://github.com/huggingface/transformers/files/13400236/DxDiag.txt)


 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I was following this tutorial.
https://youtu.be/O01BrQwOd-Q

I got this error ImportError: tokenizers>=0.11.1,!=0.11.3,<0.14 is required for a normal functioning of this module, but found tokenizers==0.15.0. Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git main Press any key to continue . . .

I'm an artist, I don't even know if following this tutorial , but I was because I read this in [lora reddit](https://www.reddit.com/r/StableDiffusion/comments/10ubpts/does_anyone_know_the_solution_lora/ )" in your cuda version looks like not correct

can you install cuda as shown here? but before do a fresh installation of automatic1111 and let it install latest

[8 GB LoRA Training - Fix CUDA Version For DreamBooth and Textual Inversion Training By Automatic1111
](https://www.youtube.com/watch?v=O01BrQwOd-Q)  " **but I have 12gb of ram** 

### Expected behavior

don't know , can anybody tell me what's the best for me?

Comments:
Hey! Sorry I'm not really sure I can help you with this. Do you have a code snippet of what you were using? (Other wise this should fix it: `pip install --upgrade transformers`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Visualbert VQA model inference lower accuracy in validation
Author: guanhdrmq
Pull request: no


Comments:
Hi @guanhdrmq, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Hi @amyeroberts Thanks for letting me know. I reckon the probelem already posts in hugginface.
Just share another link alreayd talk about it: https://github.com/huggingface/transformers/issues/17360

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to add new merge rules in AutoTokenizer
Author: Sandspeare
Pull request: no
Labels: name: New model

### Model description

I'm training new tokenizer from llama2, however, it seems that BPE tokenizer will clear the origin "vocab" and "merge" dict, and the training result is highly bias in my own datasets (about 6M C function) with some ugly tokens.

I wonder that is it possible to train a tokenizer from llama2 with the origin "vocab" and "merge" dict unchanged, only add some new vocab and merge rules from our datasets to support my requirement?


### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
raise ValueError("You have to specify either input_ids or inputs_embeds") 
Author: Ngheissari
Pull request: no
Labels: name: bug

### System Info


Hi, 

I keep getting this error:

raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds


### Sample code :

```shell
# Initializing a ViT & BERT style configuration
config_encoder = ViTConfig()
config_decoder = BertConfig()
config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)

# Initializing a ViTBert model from a ViT & bert-base-uncased style configurations
model = VisionEncoderDecoderModel(config=config)

# Accessing the model configuration
config_encoder = model.config.encoder
config_decoder = model.config.decoder

# set decoder config to causal lm
config_decoder.is_decoder = True
config_decoder.add_cross_attention = True

# Saving the model, including its configuration
model.save_pretrained("my-model")

# loading model and config from pretrained folder
encoder_decoder_config = VisionEncoderDecoderConfig.from_pretrained("my-model")
model = VisionEncoderDecoderModel.from_pretrained("my-model", config=encoder_decoder_config)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")

# load image from the IAM dataset
url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
pixel_values = feature_extractor(images=image, return_tensors="pt").pixel_values

inputs = feature_extractor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code is above from the samples provided. 

### Expected behavior

```shell
I get this error :

raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds
```


Comments:
Thanks for reporting, I'll take a look at this

Hi, @Ngheissari 

For `VisionEncoderDecoderModel`, we have to provide the following as the inputs
- `pixel_values`
- either `decoder_input_ids` or `labels`

In you code snippet, you only prepare `pixel_values`, that's why the error occurs. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Thanks for looking into this @ydshieh! Closing this issue.
fix FA2 when using quantization
Author: pacman100
Pull request: yes

# What does this PR do?
1. when I use QLoRA+Flash Attention with bf16, I get the following warning of casting to `float16` which is incorrect as it should be casting to bf16:
```bash
The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in torch.float16.
```

This PR resolves this issue.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28203). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi, this behavior of casting to `float16` is still present in these models - `whisper`, `bart`, `phi`, `distilbert`...I will create a PR to fix it.
Whisper language not returned when return_timestamps="word" and return_language=True
Author: Oscaarjs
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (False)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi 
@Narsil 
@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce can be seen here;
[https://colab.research.google.com/drive/1RplLaZdD2DBy5NDDz9pSnk53jfEFM-Cm?usp=sharing](url)

Essentially it works as expected when return_timestamps=True, but not when return_timestamps="word".

### Expected behavior

When setting;
return_timestamps="word"
return_language=True

I expect the language to be returned as well.

As shown in linked colab it does work as expected when;
return_timestamps=True
return_language=True

Comments:
This more a feature request than a bug, and is probably related to the fact that the language is not detected for the small chunk but for the full 30sec chunk of audio. + repeating the language does not seem to make sense for me, but we should also de-limit the start and end of detected language, which would bloat the code. I'll think about a potential solution 😉 
just FYI @Narsil who worked ont his refactor a while ago 


Thank you for the reply @ArthurZucker! I agree it would be weird to repeat the language, especially for each "word chunk". Is there a reason "language" has been placed on a chunk-level if it's not computed on the same? Why is the language not shown on the "top" level, similar to how "text" is shown? Best regards

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Fuyu and Persimmon] Add FA2 and fused kernels
Author: jzhang38
Pull request: yes

# What does this PR do?
- [ ] Add FA2 to Fuyu and Persimmon(just like it has been done to Llama through use_flash_attention_2=True)
- [ ] Add other [fused kernels from the FA2 repo](https://github.com/Dao-AILab/flash-attention/blob/2c3baba4a63c4007c8a132c5380edc9430f88a22/training/README.md?plain=1#L66)
- [ ] Add a working finetuning script.

1 and 2 are what's implemented in [OtterHD](https://huggingface.co/papers/2311.04219). I am not sure if you want to have 2 (fused kernels of rotary embed and layernorm) added into transformers main because that means adding cuda kernel dependecies (see [here](https://github.com/Dao-AILab/flash-attention/blob/2c3baba4a63c4007c8a132c5380edc9430f88a22/training/README.md?plain=1#L66))
3 is related to  https://github.com/huggingface/transformers/pull/26997
The goal is to accelerate the fine-tuning process for Fuyu.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @molbap 




Comments:
@jzhang38 

I'm close to finishing the FA2 implementation for Persimmon -- see #27052.  Also am developing a separate package for loading cuda kernels dynamically that would allow for incorporating fused kernels (including FA's fused ops / layers). 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`Mixtral`] Fix loss + nits
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Properly compute the loss. Pushes for a uniform distribution. 

fixes #28021
Fixes https://github.com/huggingface/transformers/issues/28093

Comments:
> # What does this PR do?
> Properly compute the loss. Pushes for a uniform distribution.
> 
> fixes #28021 Fixes #28093

What were the side effects of the issue? Did it actually degrade training runs
how to remove punctuation marks.
Author: chanyong-owl
Pull request: no

### System Info

i trained t5-large for translation.

the result of train was good

But when i input some sentence, the result is like that "What are you doing now?.??....."

[?.??......] <- how to delete that punctuation marks.

i put some parameter like max_length. But i can not solve that situation

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

c

### Expected behavior

cfdvf

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
In which function it is best way to use the temperature parameter .from_pretrained()  or .generate()
Author: pradeepdev-1995
Pull request: no

### Feature request

in which function it is the best way to use the temperature parameter .from_pretrained  or .generate()
It seems that in both functions we can pass the temperature parameter. Which is the best way to do this?
```
Model = AutoModelForCausalLM.from_pretrained(model_name,
                                                                   device_map="auto", 
                                                                   temperature=0.1,
                                                                   do_sample=True,
                                                                   torch_dtype=torch.bfloat16)
```
or
```
Model.generate(**model_input,max_length=1000, temperature=0.1,do_sample=True,)
```

### Motivation

In which function it is the best way to use the temperature parameter .from_pretrained  or .generate()

### Your contribution

in which function it is the best way to use the temperature parameter .from_pretrained  or .generate() 

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? 
I think the answer depends on the use case 😉 
Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.