noamaanMulla-03/transformers-issues huggingface dataset:
fix fuyu device_map compatibility
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes the multi-gpu issue that users were experiencing here https://github.com/huggingface/transformers/issues/29848. This happened because this [PR](https://github.com/huggingface/transformers/pull/27007) kinda reverted the changes I made to make it compatible with `device_map` [here](https://github.com/huggingface/transformers/pull/26949). 
This is not very elegant since we need to modify the forward a bit, but this is an exception since we are dealing with the output of the vision model and the input of the language model. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29880). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fyi this seems to actually still be broken as image_patches_indices can be wrong device for gather_continuous_embeddings

@ArthurZucker and @SunMarc seems like it might be helpful to specify a test on this and what sort of GPU setup the test has?  Took me a few days to pinpoint but I believe I get an error when device_map='auto' for 8x GPUs but not on 2x (and smaller GPU's)? 

Seems with more GPU's for multimodal models theres likely to be less chance the embeddings are on the same device for many HF models and not sure if that would be considered a bug or is specific to how `infer_auto_device_map` from `PreTrainedModel`.  Have not looked too deeply into if other models are casting embeddings to same device or what the overhead on that is compared to having embeddings all on same device to start with (also seems this could be issue with FSDP as well from my understanding as transformer_auto_wrap_policy is used with decoder blocks?).
hqq - fix weight check in check_quantized_param
Author: mobicham
Pull request: yes

Fixing the following issue https://github.com/huggingface/transformers/issues/30727 

Fixes: https://github.com/huggingface/transformers/issues/30727

@younesbelkada




Comments:
Thanks @younesbelkada ! Yeah I forgot to run ruff

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30748). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
When using model.generate, it does not stop at eos_token, but instead continues until the maximum length.
Author: bestpredicts
Pull request: no

### System Info

ubuntu20.04
transformers==4.29.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

use my own model, infer by


```shell
        output_texts = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            pad_token_id= tokenizer.eos_token_id,
           eos_token_id= tokenizer.eos_token_id,
            max_new_tokens=500, do_sample=False,
            top_k=30, top_p=0.85, temperature=0.3, repetition_penalty=1.2)
```
output like:
```shell
因此需要进行额外的优化。


























```



### Expected behavior

should end by eos token but not



Comments:
Hey @bestpredicts 👋 

I agree that `
` should not be generated. However, I need a stand-alone short reproducer to understand what's truly going on. We have many Llama checkpoints that are not compatible with `transformers`.

@bestpredicts I have same question, do you solved it?

> Hey @bestpredicts 👋
> 
> I agree that `
` should not be generated. However, I need a stand-alone short reproducer to understand what's truly going on. We have many Llama checkpoints that are not compatible with `transformers`.

me too,same question.

You problem can come from the `model.eos_token_id` that is not the correct one (wild guess) but we need a minimal reproducer to help you. 

Edit: have deleted this comment, because think the issue I was seeing was just the EOS was outputted as very unlikely by the models for some reason. (Check the history for the code snippet.)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Manual seting for eos_token_id for stop sequence.

```python
prompt_system = "Sos un modelo de lenguaje de avanzada que habla español de manera fluida, clara y precisa. Te llamas Roberto el Robot."
prompt = "Como se siente ser un robot pensante Roberto?"
chat2 = [
    {"role": "system", "content": f"{prompt_system}"},
    {"role": "user", "content": f"{prompt}"},
]

chat2 = tokenizer.apply_chat_template(chat2, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(chat2, return_tensors="pt").to('cuda')
streamer = TextStreamer(tokenizer)
stop_token = "
"
stop_token_id = tokenizer.encode(stop_token)[0]

_ = model.generate(**inputs, streamer=streamer, max_new_tokens=512, do_sample=True, temperature=0.1, repetition_penalty=1.2, top_p=0.9, eos_token_id=stop_token_id)
```
Why AttentionMaskConverter to_casual_4d needs input_shape[-1] > 1?
Author: HackGiter
Pull request: no

### System Info

transformers 4.37.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

import torch
from transformers.modeling_attn_mask_utils import AttentionMaskConverter

attn_mask_converter = AttentionMaskConverter(is_causal=True, sliding_window=None)

attention_mask = attn_mask_converter.to_causal_4d(
    1, 1, 5, dtype=torch.float16, device='cpu'
)
print(attention_mask.shape)

### Expected behavior

torch.Size([1,1,1,5])

Comments:
Hey! Best recommendation is to transition to using the `update_causal_mask`: 
https://github.com/huggingface/transformers/blob/416711c3ea88109cf25a9c5f85b4aeee2cb831b5/src/transformers/models/llama/modeling_llama.py#L1058 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
(Docs) Fix object_detection.md: Do not convert from RGB to BGR for augmentations
Author: rootAvish
Pull request: yes

# What does this PR do?

The docs of Albumenations here say that they use RGB instead of BGR: https://albumentations.ai/docs/examples/example/#read-the-image-from-the-disk-and-convert-it-from-the-bgr-color-space-to-the-rgb-color-space

The object detection notebook/docs however reverse the color channel dimension after opening the image as RGB and therefore pass BGR. My PR removes the reversal of channel dimension to correct this.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?

Comments:
@NielsRogge tagging you for a review here as you're the most recent contributor to the doc and if I'm not wrong its original author.

cc @qubvel could you take a look here?

Saw this was fixed in a broader PR by @qubvel in #30683, closing. 

@rootAvish Thanks!
When use Trainer to train model with "report_to" is wandb
Author: Luoqiu76
Pull request: no

### System Info

When using the report_to parameter for training, it is convenient to observe the images of various parameters during the training process, such as gradnorm, loss, etc., where the horizontal axis is steps and the vertical axis is various monitored quantities. However, after training is completed, the Trainer seems to report a unified metric, such as train_runtime, train_loss, etc., resulting in only one point remaining in the original curve after training, such as train_loss. The horizontal axis is the last step, and the vertical axis is train_loss, which seems to be the mean of all steps, causing the original curve to disappear. Is there any way to solve this matter
before it is
![image](https://github.com/huggingface/transformers/assets/80216895/c117f546-650c-437d-b442-89b3bce6e483)
after train and eval，it becomes
![image](https://github.com/huggingface/transformers/assets/80216895/1d65d89a-e470-4b2a-af5b-44db744d3755)
Please ignore the color difference. I found another experimental plot because the original curve has been replaced by a point

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I Use examples/pytorch/language-modeling/run_clm.py to train a mini llama model from the scratch，use report_to wandb

### Expected behavior

I would like to know if the metric after the final training can be reported elsewhere, as it seems that reporting to Wandb will overwrite the previous curve graph

Comments:
Hey! According to the contribution guidelines, we need to have a reproducer (minimal) and an idea of which version of transformers you are using. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
https://github.com/huggingface/transformers/blob/main/examples/research_projects/rag/finetune_rag.py ==> Execution Problem
Author: ds-256
Pull request: no

While Executing this file with a custom Index on Pytorch based ddp it just executes and gives no error with CPU and GPU . The Script I am using is  
!python /kaggle/working/rag/rag/finetune_rag.py \
--data_dir /kaggle/input/ragfine/ResearchStuff/seq2seqdataset/ \
--output_dir /kaggle/working/new_for_everything \
--model_name_or_path facebook/rag-sequence-base \
--model_type rag_sequence \
--distributed_retriever pytorch \
--index_name custom \
--passages_path /kaggle/working/Passages/ \
--index_path /kaggle/working/my_knowledge_dataset_hnsw_index.faiss \
--gpus 1 

The Various library versions that I am using are datasets==2.18.0 pytorch-lightning==1.6.0 faiss-cpu==1.8.0





![image](https://github.com/huggingface/transformers/assets/62596204/be18f9c2-d62e-455e-b515-5c54c1eec546)


Comments:
The Snip given above is for the last Output obtained from the execution of the above script where it instantly terminates after 5-10 seconds


Hey, if there is no error it's gonna be a bit hard to understand what's wrong here? 

Exactly My Point . What I wanted to know has someone used this script in the recent times with the custom dataset so that atleast I could know if there is some issue on my part or is the script really not working


We don't maintain the examples. You might just have the process killed ? If you find the bug let us know, we can't debug in your stead !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Convert Helsinki-NLP model to huggingface
Author: nichellehouston
Pull request: no
Labels: name: New model

### Model description

how convert this model to huggingface
https://object.pouta.csc.fi/Tatoeba-MT-models/eng-fas/opus-2021-02-23.zip

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Video Classification Task Guide Using Undeclared Variables 
Author: astern21
Pull request: no
Labels: name: Documentation

name: Good First Issue

name: Vision

### System Info

- `transformers` version: 4.32.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.17.2
- Safetensors version: 0.3.2
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.13.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.4 (cpu)
- Jax version: 0.4.16
- JaxLib version: 0.4.16
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Follow the Video Classification example in the HuggingFace docs:
https://huggingface.co/docs/transformers/tasks/video_classification

2. When Reaching the step 

`class_labels = sorted({str(path).split("/")[2] for path in all_video_file_paths})
label2id = {label: i for i, label in enumerate(class_labels)}
id2label = {i: label for label, i in label2id.items()}

print(f"Unique classes: {list(label2id.keys())}.")
`

The variable `all_video_file_paths` is not previously defined in the example docs. I had to write a function to retrieve the video file paths myself, as seen below:

`import os

dataset_path = "UCF101_subset" 

all_video_file_paths = []

for split in ["train", "val", "test"]:
    split_path = os.path.join(dataset_path, split)
    for category in os.listdir(split_path):
        category_path = os.path.join(split_path, category)
        for video_file in os.listdir(category_path):
            video_file_path = os.path.join(category_path, video_file)
            all_video_file_paths.append(video_file_path)`

3. When preparing the dataset for training, the variable `dataset_root_path` is not previously defined. 


### Expected behavior

I assume that these variables should have been previously defined to not cause errors for people following the example docs. 

Comments:
Thanks for reporting, would you be able to open a PR to fix this?

Sure, I'd be happy to help. 

Hi @amyeroberts , the main version of video classification [notebook](https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb) seems to define the all_video_file_paths. Should this issue be considered fixed?

Hi @manju-rangam - the issue is referencing the examples in the docs, which should still be updated. 

@amyeroberts, I've created a PR related to this

updated based on review comments @amyeroberts 

Hey @amyeroberts new to open source community can i be considered to work on this issue if its not fixed?

The PR above has been merged, so will close this issue. @anirudh2781998 feel free to take a look at other issues labeled with "good first issue" or "good first documentation issue"
[awq] replace scale when we have GELU
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR replaces the scales by `ScaleActivation`  when we have GELU activation. Before this PR, the scales calculated during the quantization were not loaded properly. To have a quick fix that doesn't depend on each model, I just check if we have the following layers `layers = ["fc_in","dense_h_to_4h","up_proj","c_fc"]` in the module that contains the GELU. This is done in order to get the shape of the scales. We need that to be able to load the scales, otherwise we will get an error saying that the shapes don't match. 

Fixes https://github.com/huggingface/transformers/issues/29421
Fixes https://github.com/huggingface/transformers/issues/30225

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30074). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[BUG] Load StarCoder2 AWQ using Transformers
Author: TechxGenus
Pull request: no

### System Info

Name: transformers
Version: 4.39.3

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Same to https://github.com/casper-hansen/AutoAWQ/issues/441
When loading starcoder2-AWQ using transformers, I received a confusing error:
```py
model = AutoModelForCausalLM.from_pretrained(
    "TechxGenus/starcoder2-3b-AWQ",
    torch_dtype=torch.float16,
    device_map="auto",
)
```
get:
> Some weights of the model checkpoint at TechxGenus/starcoder2-3b-AWQ were not used when initializing Starcoder2ForCausalLM: ['model.layers.0.mlp.act.scales', 'model.layers.1.mlp.act.scales', 'model.layers.10.mlp.act.scales', 'model.layers.11.mlp.act.scales', 'model.layers.12.mlp.act.scales', 'model.layers.13.mlp.act.scales', 'model.layers.14.mlp.act.scales', 'model.layers.15.mlp.act.scales', 'model.layers.16.mlp.act.scales', 'model.layers.17.mlp.act.scales', 'model.layers.18.mlp.act.scales', 'model.layers.19.mlp.act.scales', 'model.layers.2.mlp.act.scales', 'model.layers.20.mlp.act.scales', 'model.layers.21.mlp.act.scales', 'model.layers.22.mlp.act.scales', 'model.layers.23.mlp.act.scales', 'model.layers.24.mlp.act.scales', 'model.layers.25.mlp.act.scales', 'model.layers.26.mlp.act.scales', 'model.layers.27.mlp.act.scales', 'model.layers.28.mlp.act.scales', 'model.layers.29.mlp.act.scales', 'model.layers.3.mlp.act.scales', 'model.layers.4.mlp.act.scales', 'model.layers.5.mlp.act.scales', 'model.layers.6.mlp.act.scales', 'model.layers.7.mlp.act.scales', 'model.layers.8.mlp.act.scales', 'model.layers.9.mlp.act.scales']

And it doesn't output properly. There is no error when using `AutoAWQForCausalLM` or `vllm.LLM` to load it, and they work well.

### Expected behavior

This error is not generated and the output is correct

Comments:
cc @younesbelkada 

Hi @TechxGenus 
Thanks ! 
I think @SunMarc recently made a PR: https://github.com/huggingface/transformers/pull/30074 that should fix your issue ! 
@amyeroberts I requested your review for whenever you have time 🙏 

@younesbelkada Thanks!
I test #30074, it pointed out the problem but didn't work. The main reason is that starcoder2 uses `PytorchGELUTanh` instead of `nn.GELU`. It works when replacing this.
Since there are currently many variations of `GELU`, I'm not sure if other models have similar issues.

thanks @TechxGenus ! Maybe in @SunMarc 's PR it is possible to extend it to support other modules in addition to `nn.GELU` - wdyt @SunMarc ?

Yes I will update the PR to support more modules ! 

related report:
https://huggingface.co/TechxGenus/starcoder2-7b-AWQ/discussions/1
Support for Falcon2-11B
Author: Nilabhra
Pull request: yes

This PR adds support for the Falcon2-11B architecture. 


Comments:
cc @ArthurZucker 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker The tests have passed.
[GroundingDino] Adding ms_deform_attn kernels
Author: EduardoPach
Pull request: yes

# What does this PR do?

This PR fixes https://github.com/huggingface/transformers/issues/30765#issuecomment-2106174617 by adding the missing  `ms_deformable_attn` kernels to `GroundingDino` 

## Who can review it?

@NielsRogge 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30768). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Grounding DINO missing custom kernels
Author: sam-ulrich1
Pull request: no
Labels: name: Vision

### System Info

Latest including master as of 5/11/24
Ubuntu, 4090 GPU

### Who can help?

@amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When running any GroundingDINO model you get the following. I have confirmed the kernels do not exist in the transformers repo:
```
Could not load the custom kernel for multi-scale deformable attention: [Errno 2] No such file or directory: '/.../transformers-main/src/transformers/kernels/grounding_dino/vision.cpp'".
Could not load the custom kernel for multi-scale deformable attention: [Errno 2] No such file or directory: '.../transformers-main/src/transformers/kernels/grounding_dino/vision.cpp'".
```

### Expected behavior

Either the kernels are not necessary and should be removed or they are necessary and causing performance issues. GroundingDINO does seem to under perform in transformers but this is purely anecdotal

Comments:
cc @EduardoPach 

Regarding

> GroundingDINO does seem to under perform in transformers but this is purely anecdotal

Could you clarify? Text queries need to be lowercased + end with a dot

Hey @sam-ulrich1 thanks for opening this Issue! `GroundingDino` was missing the `ms_deform_attn_kernels` this was something that we overlooked when we merged the model to the library. I have opened this PR https://github.com/huggingface/transformers/pull/30768 to fix it.
Blip dynamic input resolution
Author: zafstojano
Pull request: yes

# What does this PR do?

This PR adds ability to handle dynamic input image resolutions to the BLIP family of models.

Addresses #30579 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Ping: @amyeroberts 

Comments:
@amyeroberts good point!

the tests now also have a check for the textual content.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30722). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.