noamaanMulla-03/transformers-issues huggingface dataset:
Int4 transformers training
Author: NicolasMejiaPetit
Pull request: no
Labels: name: Feature request

### Feature request

There is a GitHub repo out with the necessary kernels and code (and a great paper) to train a transformer based models using int4. 

The authors use a couple of algorithms to get around the struggle of quantizing down to int4 including keeping non linear operators in fp16 to avoid certain quant issues, they solve the outlier problem by "propose a Hadamard quantizer (HQ) to solve the outlier problem. Its main idea is to quantize the matrices in another linear space which has fewer outliers."  The results they achieved were "We compare the training throughput of the FP16 PyTorch AMP and our INT4 training algorithm for training BERT [24] and GPT [37]-style language models on a system of 8 Nvidia A100 GPUs. We vary the hidden layer size, intermediate fully-connected layer size, and batch size, and plot the speedup of INT4 training in Fig. 5. Our INT4 training algorithm can achieve up to 35.1% speedup for BERT-style models and up to 26.5% speedup for GPT-style models." 

These results are without using Flash Attention which would increase gains further, and you could use the Galore 8bit optimizer, or better yet Deep speeds 1bit Adam optimizer, fully offloaded to the CPU. For optimized full fine tuning of large 7b models on consumer hardware.


This code and paper is for FFT but this same concept could apply directly for Lora and QLora.


Links:
[Paper](https://arxiv.org/pdf/2306.11987.pdf)
[Code](https://github.com/xijiu9/Train_Transformers_with_INT4)

### Motivation

Having int4 as a trainable dtype would provide a ton of utility. On 2 consumer 3090’s you get 1 TFLOP of compute according to Nvidia’s documentation. It would increase the training possibilities for the gpu poor. And significantly increase training speed on for server applications.

### Your contribution

Gathered information. I’m not very good at coding, at least not good enough to add to transformer repo. This might be too long of an endeavor, so if it is sorry for wasting your time, and we can close this feature request.

Comments:
Thanks for the detailed feature request! 
Might be better for the nanotron library. 
Just FYI @younesbelkada 

> Thanks for the detailed feature request!
> Might be better for the nanotron library.
> Just FYI @younesbelkada

Awesome thank you, I'll share the code with the folks over at the quanto repo, and the nanotron repo
WARNING: tokenization mismatch: 43 vs. 44. (ignored)
Author: lucasjinreal
Pull request: no

Recently there are many errors got either from fastchat or llava code base if using latest transfomers.

WARNING: tokenization mismatch: 43 vs. 44. (ignored)

What does this happen and how to dismiss it? Will it effect the final training result?

Comments:
Hi @lucasjinreal, 

Without a code sample to replicate, information about the running environment or more information about the error - including full trackback - there isn't much we can do to help you here.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
RWKV5 tokenizer truncation
Author: sedrick-keh-tri
Pull request: no

### System Info

Python 3.10. Transformers 4.37.2

### Who can help?

@arthur

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained("RWKV/HF_v5-Eagle-7B", trust_remote_code=True)

    long_str = ""
    for i in range(10000):
       long_str = long_str + " " + str(i)

    print(len(long_str))        # 48890
    long_str_tokenized = tokenizer(long_str, truncation=True, max_length=2048)
    print(len(long_str_tokenized['input_ids']))        # 19900



### Expected behavior

Proper truncation to max_length

Comments:
cc @ArthurZucker 

Hey! THanks for opening an issue. The problem is that you are using `trust_remote_code=True` and thus the `https://huggingface.co/RWKV/v5-Eagle-7B-HF/blob/main/tokenization_rwkv_world.py` file is used. The code is not on transformers yet! 

Use the tokenizer from #29095 should fix this already 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Cannot use time series transformer as encoder and gpt model as decoder using encoder decoder architecture from hugging face
Author: nikhilajoshy
Pull request: no

### System Info

-

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

EncoderDecoderModel.from_pretrained()

### Expected behavior

I want to use EncoderDecoderModel.from_pretrained() where huggingface/time-series-transformer-tourism-monthly is the encoder and openai-community/gpt2 as decoder but as mentioned in https://huggingface.co/blog/warm-starting-encoder-decoder, TS transformer does not have a tokenizer and hence not sure what to use as encoder input ids

Comments:
Hi @nikhilajoshy, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Mark `test_eager_matches_sdpa_generate` flaky for some models
Author: ydshieh
Pull request: yes

# What does this PR do?

cc @fxmarty 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29479). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Apart from qwen2 (that is bugged and I did not work on), other tests are simply flacky. In my experience (for inference) this had no impact on e.g. generation.

see https://github.com/pytorch/pytorch/issues/103462 for numerical differences (although for gradients there)

I will remove the one for `qwen2`.

For qwen2 I think we should find which one of eager/sdpa is correct and maybe disable the one that is not until fixed :/

> For qwen2 I think we should find which one of eager/sdpa is correct and maybe disable the one that is not until fixed :/

I can skip it for now.
Typo in the document of automodelforcausallm
Author: xianjiedai
Pull request: no
Labels: name: Documentation

name: Good First Documentation Issue

Documentation Link: <[https://huggingface.co/transformers/v3.5.1/model_doc/auto.html#automodelforcausallm>](https://huggingface.co/transformers/v3.5.1/model_doc/auto.html#automodelforcausallm)


Specific Section Title: "AutoModelForCausalLM"


Description of Typos:


Original Text: "This is a generic model class that will be instantiated as one of the model classes of the library—with a causal language modeling head—when created with the when created with the"


Correct Text: "This is a generic model class that will be instantiated as one of the model classes of the library—with a causal language modeling head—when created with the"

Comments:
@xianjiedai would you like to open a PR to fix? it would be in the transformers repo

This issue can be closed as it won't be fixed (see https://github.com/huggingface/transformers/pull/29830#issuecomment-2021330414).

Thanks for the update! 🤗
Error when loading pretrained OpenLlama model
Author: kevin-guimard-ext
Pull request: no
Labels: name: Good Second Issue

### System Info

- `transformers` version: 4.38.2
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.13
- Huggingface_hub version: 0.21.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu118 (True)
- Tensorflow version (GPU?): 2.10.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @younesbel

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When running this snippet:
```
from transformers import OpenLlamaForCausalLM
model = OpenLlamaForCausalLM.from_pretrained("openlm-research/open_llama_7b")
```

I get the following error:

> You are using a model of type llama to instantiate a model of type open-llama. This is not supported for all configurations of models and can yield errors.
> ...
> AttributeError: 'OpenLlamaAttention' object has no attribute 'rope_theta'

### Expected behavior

The code snippet has been taken from the documentation (https://huggingface.co/docs/transformers/model_doc/open-llama), it is expected to work.

Comments:
I can reproduce. This model is deprecated, feel free to open a PR for a fix, otherwise would recommend using Llama, we no longer maintain OpenLlama

Ok, I'll switch to Llama then. I wanted to try OpenLlama because I heard that Llama required a subscription, but I may be wrong.

No, OpenLlama is basically Llama but with some "optimisations". You should be able to load https://huggingface.co/NousResearch/Llama-2-7b-hf  🤫 

It's a pity that the "Run the model" section has not been written for this model, with a snippet explaining how to use it. Must the weights be downloaded from the Meta repo?
Change Device Allocation and Improve Shape Handling in Trainer evaluation loop
Author: Thytu
Pull request: no
Labels: name: Feature request

### Feature request

## Context
Currently when setting the [compute_metrics](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L360) parameter as non-null value it sets the [prediction_loss_only](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3402)  param as `False`.

As `prediction_loss_only` is `False`, [self.prediction_step](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3586) will return non-null value for the `logits` variable.

This variable will be used here to be [padded](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3608), then [concatenated](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3612) to previously stored logits.

## What bothers me

### Device Allocation
If the logits are tensors, a new tensor will be created using the same device as logits' one ([cf](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_pt_utils.py#L91)). Meaning that if `logits` is on `cuda:0`, the resulting tensor will also be on `cuda:0`. This unnecessarily consumes VRAM as after the tensor will be moved back to CPU ([cf](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3626)) without further operations.

IMO the resulting tensor should be on CPU from the beginning to avoid consuming VRAM unnecessarily. But let me know if I miss something :)

### Handling of Shapes

It seems that the concatenation is only done across `logits` ([cf](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3608)), not taking into account the shape of previously registered logits in `preds_host`. Meaning that at each iteration, the `logits` should have the same max length otherwise it will raise the following error : `RuntimeError: Sizes of tensors must match except in dimension 0. Expected size [SHAPE] but got size [ANOTHER SHAPE] for tensor number 1 in the list.`

This causes issue for `AnyToSeq` models (i.e `SeqToSeq`) where the length of the outputs sequences can vary from one iteration to another. The current workaround would be to pad the output to a fix max length (i.e by padding the input itself to a fix max length) however this greatly increase the necessary compute, thus making the training process slower.

A fix could be to directly cast the logits to numpy with a non-constant shapes across logits.

---

@muellerzr I would be happy to have your input on that one

### Motivation

For more context, I encountered those behaviors when working on [that issue](https://github.com/Thytu/SMIT/issues/6).

Fixing both behaviors would :
* Remove the spikes in VRAM usage generated when using `compute_metrics`
* Avoid having to pad the model's input unnecessarily thus reducing the compute required and making the training faster.

### Your contribution

It's kinda of a blocker on my side, so I would be happy to work on a PR :)

Comments:
Reset alarm signal when the function is ended
Author: coldnight
Pull request: yes



# What does this PR do?

Fixes #29690


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- tokenizers: @ArthurZucker


Comments:
> Might be hard to reproduce, but yes agree with you we should reset in a finally!

Yeah, it's not easy to reproduce, but I've reproduced it, please see the newest comment in the issue https://github.com/huggingface/transformers/issues/29690#issuecomment-2024309619
Add support for BLIP-2 multimodal feature extraction
Author: youssefadr
Pull request: yes
Labels: name: WIP

# What does this PR do?

This PR introduces the addition of `get_image_feature` and `get_text_feature` methods to the `Blip2ForConditionalGeneration` class. These changes align with the original Qformer implementation, which utilized both text and image inputs.

The current implementation in HuggingFace lacks support for multimodal embeddings, especially the capacity to extract embeddings by passing both text and image to the QFormer. This PR addresses this shortcoming.

<!-- Remove if not applicable -->

Fixes #25300 #25245 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

@NielsRogge @amyeroberts 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25474). All of your documentation changes will be reflected on that endpoint.

cc @amyeroberts and @younesbelkada 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @youssefadr, feel free to re-open this if you still plan on adding this feature

@ArthurZucker Thank you, I'll do it, I am sorry I couldn't find time lately for it. It is on my backlog!

@NielsRogge Hi, should I open a new PR for this, or is there a way to re-open this one? It seems to be requested by a lot of users.

Is this still being worked on? I am in need of this tool.

it's stale, feel free to take it over
[`BC`] Fix BC for AWQ quant
Author: TechxGenus
Pull request: yes

# What does this PR do?

Fixes #29783
The previous fix #29934 only solved the problem of inference. This PR solves the error reporting during quantization.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker


Comments:
Rework tests to compare trainer checkpoint args
Author: muellerzr
Pull request: yes

# What does this PR do?

Follow-up to https://github.com/huggingface/transformers/pull/29255, noticed that this fails on multi-GPU since there is logic involved on the batch size, which we need to bring back in because `train_batch_size` is the total batch size, or `train_batch_size // n_gpu`.

I also noticed it would print out a list of warnings, which can get messy, so condensed the logs down to a single one for users.

Fixes failing CI on accelerate main (tested locally on two GPUs)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29883). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Neuron Trainium --Gradient_Accumulation_Steps > 1
Author: mathephysicist
Pull request: no

### System Info

If I use Optimum Neuron on Trainium with --gradient_accumulation_steps > 1 and training failed, 

Then I modified line https://github.com/huggingface/transformers/blob/6d1f545665ac66420af9f6702d891a30c5d070ea/src/transformers/trainer.py#L1966C21-L1966C23

to include 

```
if is_torch_tpu_available() :
  xm.mark_step()
```

and then set gradient_accumulation_steps > 1 and it worked. 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Use any Neuron script that uses huggingface trainer and works, and set --gradient_accumulation_steps 2


### Expected behavior

Should do gradient_accumulation

Comments:
cc @muellerzr as it seems to cover trainer + TPU

Thanks for the flag @mathephysicist! Can you confirm this works (with no change in Trainer) if installing accelerate from main via `pip install git+https://github.com/huggingface/accelerate@grad-accum-tpu`?

Will try that out! 

That seems to uninstall/remove a lot of the Neuron packages, resulting in xla_model related issues. That may be due to environment issues? I am trying the optimum-neuron tag v0.0.18, do you think trying optimum-neuron master would resolve them?

`RuntimeError: Cannot replicate if number of devices (1) is different from 32`

This is because optimum is pinned to a much older version of accelerate sadly. We'll need to put in the fix here in transformers it looks like... not ideal... (though the same solution has been put into accelerate)

@mathephysicist if you want to open a PR with your one-liner, I think that'd be fine with me. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.