noamaanMulla-03/transformers-issues huggingface dataset:
Compatibility of flash attention 2 and type conversion due to accelerator.prepare
Author: bellos1203
Pull request: no

Hi, I'm trying to fine-tune my model, which is BLIP-2, using flash attention 2 on OPT 2.7B, but **using FA2 produces significantly higher loss** than using eager attention mode, which seems similar to issues reported previously (#26498, #28925, #28142).
From the comments from those issues, the best way to use fa2 normally is to load the model in full precision and train the model with autocast context.
However, when using `accelerate` library, `accelerator.prepare` function converts the model into a specified dtype (for me, `bf16`) including layer norm.
I guess this caused the problem for me, but I'm not sure.

Could you check this behavior and give any suggestions? I'm using `transformers==4.40.0.dev0`, `accelerate==0.23.0` and `flash_attn==2.5.5`.
Or if there is any more detail that I have to elaborate on, please let me know.
Thanks in advance :)

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Also think you should be able to prevent accelerate from preparing to a different dtype! 
Thanks!

Ok, I posted my question on the [forum](https://discuss.huggingface.co/t/compatibility-of-flash-attention-2-and-type-conversion-due-to-accelerator-prepare/80406)! Thanks for your reply :)
hugging face microsoft/trocr-base-printed model freezing the video while detecting
Author: nirmal-dls
Pull request: no

### System Info

- `transformers` version: 4.38.1
- Platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NVIDIA GeForce GTX 1650
- Using distributed or parallel set-up in script?: NA

### Who can help?

@ArthurZucker
@amyeroberts
@younesbelkada



### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

There is no issue with the code. I'm working on char recognition of license plate in a video using "microsoft/trocr-base-printed" model, actually i'm detecting plate with yolov7 tiny (using half precision) & the recognising chars with "microsoft/trocr-base-printed", model is 90% correctly recognising chars but the issue is while recognising it is playing video very slow, may be due to size(1.33GB) of the model even after loading model on cuda. Please advise a solution to this problem?

### Expected behavior

The video should not slow down while recognising chars.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Otherwise you could look into:
- quantization (smaller model -> more RAM available -> more frame processed by seconds)
- compilation (torch.compile, ONNX export etc)

Thanks!

Sir, I've tried converting it to onnx but size did not reduced & video stucked for very long time after loading onnx model. Request you to resolve the issue here at this platform. FYI, we like to deploy model on Jetson nano, will it be get deployed without any issues?

@nirmal-dls I'd suggest profiling the forward pass to find out if there's any parts of the code which are taking a particularly long time. If you identify a troublesome area and have a solution we'd be very happy to review a PR to amend. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`TorchFp8Quantizer`] Attempt to integrate `pytorch-labs/float8_experimental` in HF transformers
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, this PR attempts to integrate https://github.com/pytorch-labs/float8_experimental in transformers for efficient FP8 inference. To potentially combine it with https://github.com/huggingface/transformers/pull/27931 for faster text generation on newer hardwares (compute capability >= 9.0)

I need to address these points before 

- [ ] are fp8 layers serializable?
- [ ] make sure the code runs
- [ ] end-to-end training script
- [ ] Docs
- [ ] add tests

Currently the target API is: 

```python
import torch
from transformers import AutoModelForCausalLM, TorchFp8Config, AutoTokenizer

quantization_config = TorchFp8Config(
    linear_type="dynamic",
    modules_to_not_convert=["lm_head"]
)

model_id = "facebook/opt-125m"

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    quantization_config=quantization_config, 
    device_map="auto",
    torch_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Hello my name is"

inputs = tokenizer(text, return_tensors='pt').to(0)
out = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(out[0]))
```

cc @ArthurZucker @drisspg @vkuzo @pacman100 @SunMarc @Titus-von-Koeller 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28809). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Any plans to support KV Cache offloading to CPU (and NVMe)?
Author: goelayu
Pull request: no
Labels: name: Feature request

### Feature request

Similar to how model parameter and optimizer offload is supported using the [deepspeed library](https://github.com/huggingface/transformers/blob/de13a951b38b85195984164819f1ab05fe508677/docs/source/en/perf_train_gpu_one.md#deepspeed-zero), are there plans for natively supporting KV cache offloading as well? 

### Motivation

Apart from helping accommodate larger batch sizes on a single GPU, this can also significantly improve overall throughput, specially when batch sizes grow very large (resulting in a linear increase in KV cache size).

### Your contribution

I see there already exists an implementation of this: https://github.com/tjruwase/transformers/tree/kvcache-offload-cpu, so maybe this is simply about incorporating those changes in the main repo?

Comments:
cc @ArthurZucker 

I'm also interested in this! I would also be interested if there are plans to natively support performing other operations on tensors in the cache -- for example, as of now if I want to expand every tensor in the cache in a setting like this:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda:0"
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf").to(device)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")

embed_layer = model.get_input_embeddings()
input_ids = tokenizer(["Test"], return_tensors="pt", padding=False)["input_ids"].to(device)
inputs_embeds = embed_layer(input_ids)

output = model(inputs_embeds=inputs_embeds, use_cache=True)
kv_cache = output.past_key_values
```

then I don't see an alternative to iterating over the cache, as in:

```python
bs = 100
expanded_cache = []
cache_len = len(kv_cache)

for i in range(cache_len):
    keys, values = kv_cache[i]
    expanded_cache.append((keys.expand(bs, -1, -1, -1), values.expand(bs, -1, -1, -1)))

print(tuple(expanded_cache))
```

If you use the static cache, it will automatically create a new cache, which should not be much slower, but will take more ram indeed as you "clone" vs "expand". You want to only compute a system prompt and share the cache for all further batches?

Thanks for the response :) Essentially: sometimes prompts start with identical sequences of tokens after the system prompt, so we can cache a bit more -- but yes, the intent is to share the cache for future batches.

To clarify: this requires using torch.compile, but then I should be able to directly modify the cache tensors by accessing the cache's `key_cache` and `value_cache`?

yes sir!
OOM with mixed precision inference on OPT1.3B
Author: grigorn
Pull request: no

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-5.15.0-87-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am getting OOM error for this code on 40GB A100. 
```python
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').cuda()
inpt = torch.randint(0, 1000, (1, 2048))
with torch.autocast(device_type='cuda', dtype=torch.float16):
    out = model(inpt.cuda())
```
But usual inference works
```python
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').cuda()
inpt = torch.randint(0, 1000, (1, 2048))
out = model(inpt.cuda())
```


### Expected behavior

If I do model.half() inference works but I expected the first code work too without any OOM as full precision inference is ok.

Comments:
Hi @grigorn 
I suspect you are getting this because you did not wrapped your inference with `torch.no_grad()` context manager. Can you try again by adding `torch.no_grad()`

I wanted to use the model for training so I cant use the torch.no_grad(). Also, isn't it strange that inference is ok on fp32 but not in mixed precision? 

I agree it is strange, at the same time I feel this is quite orthogonoal to transformers, are you able to repro the same behaviour with a pure `torch.nn.Module` ?

No, I have not done that. What do you mean by "orthogonal to transformers"? And If I want to to train the model in mixed precision without Trainer, is this the only way?

I tried the same code with (2, 1664) input size on 49GB A6000 and I still see the same effect


I also tried running running 100 forward passes on input size of 1024. FP32 took 21 seconds using 23GB memory, fp16 4.74 seconds using 27GBs memory.

Probably a duplicate of #25419

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Schedule Free Optimizer (SGD, AdamW)
Author: vasqu
Pull request: no

### Feature request

See [schedule_free](https://github.com/facebookresearch/schedule_free): Schedule free optimizer alternatives for adamW and SGD. If possible, without needing it as a separate dependency.

### Motivation

Getting the right optimizers and even schedulers can be a very difficult and annoying task. Allowing scheduler-free approaches might help in reducing finding the right schedules. Quoting the meta repo:
"TLDR Faster training without schedules - no need to specify the stopping time/steps in advance!"

### Your contribution

Not sure if could help but I'd suggest looking at the [Caveats](https://github.com/facebookresearch/schedule_free?tab=readme-ov-file#caveats) especially (e.g. BatchNorm issue, default values, [minimum learning rate](https://github.com/facebookresearch/schedule_free/issues/5)..). 

Comments:
Closing this issue since #30079 will take care of it
Mixture of All Intelligence (MoAI)
Author: Dev-Khant
Pull request: no
Labels: name: New model

name: Multimodal

### Model description

A new large language and vision model (LLVM) that uses auxiliary visual information and natural language for prediction.

It uses 2 modules: 𝙈𝙤𝘼𝙄-𝘾𝙤𝙢𝙥𝙧𝙚𝙨𝙨𝙤𝙧 and 𝙈𝙤𝘼𝙄-𝙈𝙞𝙭𝙚𝙧. Here 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗼𝗿 condenses the verbalized outputs of the external CV models into auxiliary visual information and 𝗠𝗶𝘅𝗲𝗿 blends three types of intelligence — visual features, auxiliary features from external CV models and language features into a cohesive whole.

MoAI-7B surpasses both open-source and closed-source LLVMs in vision language tasks.

Model repo: https://github.com/ByungKwanLee/MoAI


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
@NielsRogge Can I work on this to add it to the library?

Sure! Free free to open a PR and let us know when it's ready for review or you need help integrating into the library. 

In general, we prioritise reviewing based on PRs opened rather than comments on issues, as we find this prevents issues from becoming stale. You're free to work on something if there's no active linked PRs open. 

Thanks @amyeroberts I'll start working on this as I don't see any open PR regarding this.

Waiting for this https://github.com/huggingface/transformers/pull/29667 to get merged, because internally model uses InternLM.
`device_map='auto' support for BertForSequenceClassification for inference using large models
Author: tnnandi
Pull request: no

### Feature request

Currently, from_pretrained() for BertForSequenceClassification does not support `device_map='auto'`, and is preventing inference using large models that don't fit into a single GPU. I'd also appreciate it if you could suggest the best alternative right now. 

### Motivation

Inference using large models that don't fit into a single GPU is a big issue. 

### Your contribution

N/A

Comments:
Closing, as this is a duplicate of https://github.com/huggingface/transformers/issues/25296.
Saving checkpoint fails when hugging face 503
Author: turtle0x1
Pull request: no
Labels: name: trainer

### System Info

Going to assume it doesn't matter what system your on

### Who can help?

@muellerz @pacman100 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

train a model with a more than average 50 line python file, then when hugging face http server 503's the training gets lost as it cant pull a `/company_xyz/model_xyz/resolve/main/config.json` file

### Expected behavior

Training time doesn't get blown away and the only thing left isn't a folder with a README.

The expected behaviour is this is all prefetched at the start (I cant see this changing that frequently or the desired behaviour to be changing a config.json between each checkpoint) ~or its made more obvious how to train offline first (for "some reason" I think not).~ (this is in getting started).  Still think the request is needless.

Comments:
It appears you can fix this (~it seems, with a missing vocab warning~ went away)  if you already have the tokenizer & model cached in the default location.

First make a directory, `model-name-xyz`

Then modify your train script to 

```py
tokenizer.save_pretrained("./model-name-xyz") # might wanna save in a different place to model
model.save_pretrained("./model-name-xyz")
```
then run the train script to save model + tokenizer.

Then once they are saved, open the train file and remove the above save comments.

Change the `from_pretrained("xyz/xyz")` to the `from_pretrained("./modal-name-xyz")` for both the model and the tokenizer.
On the tokenizer I also set `local_files_only=True`. For the model you'll need to add;

```py
config = AutoConfig.from_pretrained("./model-name-xyz/config.json")
# Might be a better way, XYZ could be any of the HF classes/modules/whatever
model = XYZ.from_pretrained("./model-name-xyz", config=config)
```

Also changed my train command to something like `HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python3 train.py`

Its late, some of these steps might not be required, or there might be a better way - but it going my training back going again, and I need sleep.

Still don't think the default behaviour should require an non-soft failing http request on saving a training step/epoch, which I is why I dont close this. Does pulling the `config.json` need to happen on every save?


Same issue, huggingface 503😭

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Whisper] Computing features on CPU slows down WhisperFeatureExtractor
Author: yashjogi
Pull request: no

### Feature request

In the WhisperFeatureExtractor class, to reduce the time it takes to compute audio features, an option should be added to compute the features on GPU in batch mode.

### Motivation

Currently, the Whisper Feature Extractor processes audio features on the CPU one by one for each batch, which slows down the process. Additionally, it doesn't utilize GPU operations despite using torch functions.

We ( + @vaibhavagg303 ) modified the code to enable batch feature extraction on CPU, cutting the average time for batches of 8 from 1.5 seconds to 0.25 seconds — depending on available CPU resources. Furthermore, we transferred all variables — mel filters and hanning window — in the function __torch_extract_fbank_features_  to GPU, further reducing **the average time to 0.02 seconds — a significant reduction from 1.5 seconds.**

Whisper Models, already with high latency, experience even greater latency due to processing input features on the CPU without batching. By doing batch-wise inference on GPU, the total inference time for Whisper-Small is reduced significantly, **bringing it down by whopping 25%.** 

@sanchit-gandhi 

The code used to measure the time taken is as follows. 
```
from transformers import WhisperProcessor, WhisperForConditionalGeneration
feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-small")
import librosa
import time

processor = WhisperProcessor.from_pretrained("openai/whisper-small", language='english')
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model.config.forced_decoder_ids = None
model = model.to('cuda')

total_time = 0
batch_size= 8

for i in range(0, len(df), batch_size):
  sample = df[i:i+ batch_size]
  files = sample['file'].tolist()
  sample_array = []
  for file in files:
    data, sample_rate = librosa.load(file, sr = 16000)
    sample_array.append(data)
 
 st = time.time()
  input_features = feature_extractor(sample_array, sampling_rate=16000).input_features
  input_features = torch.tensor(input_features).to('cuda')

  with torch.no_grad():
    predicted_ids = model.generate(input_features, language='english')
  transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

  total_time += time.time() - st
```


### Your contribution

This is the PR by @vaibhavagg303.
https://github.com/huggingface/transformers/pull/29900

Comments:
Change log level to warning for num_train_epochs override
Author: xu-song
Pull request: yes

# What does this PR do?

This log should be a `warning` instead of `info`, as `num_train_epochs` override is an very import message.



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- trainer: @muellerzr and @pacman100


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.