noamaanMulla-03/transformers-issues huggingface dataset:
Support for custom data_collator in Trainer.train() with datasets.Dataset
Author: kaletap
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Currently (transformers==3.3.1) Trainer removes unknown columns (not present in forward method of a model) from datasets.Dataset object. It prevents using custom DataCollator in `.train` method since it doesn't have columns that one would want to use. 

I would be interested in an option to not remove unknown columns and allow user to handle them in DataCollator (or provide their own DataLoaders to the train method).

## Motivation
The example code I was trying to use for training looks like this:
```
from typing import List

from datasets import load_dataset
from transformers import RobertaForSequenceClassification, RobertaTokenizer, Trainer


tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaForSequenceClassification.from_pretrained('roberta-base', return_dict=True)

train_dataset = load_dataset("yelp_polarity", split="train")
val_dataset = load_dataset("yelp_polarity", split="test")

class DataCollator:
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer
        
    def __call__(self, examples: List[dict]):
        labels = [example['label'] for example in examples]
        texts = [example['text'] for example in examples]
        tokenizer_output = self.tokenizer(texts, truncation=True, padding=True)
        tokenizer_output['input_ids'] = torch.tensor(tokenizer_output['input_ids'])
        tokenizer_output['attention_mask'] = torch.tensor(tokenizer_output['attention_mask'])
        output_dict = dict(labels=labels, **tokenizer_output)
        return output_dict
    
data_collator = DataCollator(tokenizer)

trainer = Trainer(
    model=model,                   # the instantiated 🤗 Transformers model to be trained
    train_dataset=train_dataset,   # training dataset
    eval_dataset=val_dataset,      # evaluation dataset
    data_collator=data_collator    # problem with removing unwanted columns by transformers.Trainer code. Doesn't really work, is it a bug?
)

trainer.train()
```
(with `trainer.train()` not working)

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
Before trying to fix things, I would be interested to learn if there is a suggested alternative way of doing that? 

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Just set `remove_unused_columns=False` in your `TrainingArguments` to disable that behavior.

Great, I completely missed that! Thanks.

hello,when I use my own data_collator, I got TypeError: Qwen2Model.forward() got an unexpected keyword argument 'labels', could you tell me why?
Fix Bug: Gemma2 the past_key_value.update() function has added a new parameter "sliding_window" to support the _sliding_update function.
Author: kkk935208447
Pull request: yes

# What does this PR do?
System Info
transformers 4.42.3

Now gemma2 model generates long text that exceeds the window size (>4096), it will report a CUDA error, which seems to be a problem with the failure of the _sliding_update function in HybridCache. The error is as follows:

```python
# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    device_map="auto",
    torch_dtype=torch.bfloat16
)

input_text = "Write me a poem about Machine Learning." * 800
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**input_ids,max_new_tokens = 1150)
print(tokenizer.decode(outputs[0]))
```

```python
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?


## Who can review?
@ArthurZucker

Comments:
Could you take a look when you have a minute @sanchit-gandhi ?
Cambricon MLUs support SDPA and flash_attn
Author: huismiling
Pull request: yes

# What does this PR do?

add features: MLUs support SDPA and flash_attn.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
cc @ArthurZucker @younesbelkada 

@younesbelkada Thanks, this is the commit. 
[Cambricon support SDPA and flash_attn](https://github.com/huggingface/transformers/pull/31102/commits/6adf3d9e7908a1ff03853f0ac339ac07fd9ad452)

Hi @huismiling 
This is not the commit from Flash Attention library, what I want to see is the commit in flash attention library that enables cambricon support to confirm it is included in the 2.3.3 version, so far I couldn't find it :/ 

@younesbelkada 
Hi, MLU imply private flash attn. It is not included in flash-attention. 
MLU's flash attn has not been open-sourced yet.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker @LysandreJik 
Hi, is this PR Okay to be merged ? Is there anything I can help? 

@ArthurZucker @younesbelkada @LysandreJik 
Hi, sorry to disturb. But this PR takes so long time to merge. May someone help this ? 
If there is anything needed to fixed, please let me know.

Sorry I was waiting for the tests and got lost in track!
[tests] fix deepspeed zero3 config for `test_stage3_nvme_offload`
Author: faaany
Pull request: yes

## What does this PR do?
Since we manually modified the original zero3 config value [here](https://github.com/huggingface/transformers/blob/main/tests/deepspeed/test_deepspeed.py#L402), we will end up with a ValueError in accelerate ([code](https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L3269)). 

For Zero3 Checkpointing, we need to turn this value to True. 

 @amyeroberts and @ydshieh 

Comments:
OK for me.

Hi @ydshieh , could you take a look at this PR? Thx!

Hi @faaany Works for me, but the link 

> we will end up with a ValueError in accelerate ([code](https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L3269)).

doesn't seem pointing to the desired line/block?

> Hi @faaany Works for me, but the link
> 
> > we will end up with a ValueError in accelerate ([code](https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L3269)).
> 
> doesn't seem pointing to the desired line/block?

Oh, that's a mis-click. This is the correct line: https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L3278

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31881). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix bad test about slower init
Author: muellerzr
Pull request: yes

# What does this PR do?

Fixes a test that seems *very* compute-dependent. On my machine running 5 times this passed, the CI showed passing, but the main CI showed failing. We need to eventually investigate some good bounds

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32002). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Tests: remove cuda versions when the result is the same 🧹🧹
Author: gante
Pull request: yes

# What does this PR do?

Removes the dictionary of possible test outcomes for each cuda version, when ALL outcomes are exactly the same.

Why?
1. No duplication of redundant information
2. Future-proofing: the dictionary form requires one entry per each type of device (new cuda compute capability, AMD devices, ...), it's bound to increase over time

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31955). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bug report update -- round 2
Author: gante
Pull request: yes

# What does this PR do?

The previous PR (#31983) had something wrong in its config file, which made the bug report option disappear. 

![Screenshot 2024-07-16 at 17 06 28](https://github.com/user-attachments/assets/fa5a18dd-5578-4eee-9cfb-aeecd1be7f01)

This PR adds a markdown section at the top of the bug report page. The markdown section was inspired by the [example](https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository#creating-issue-forms) in the GH docs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32006). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix gather when collecting 'num_input_tokens_seen'
Author: CodeCreator
Pull request: yes

# What does this PR do?

Following from #29099, the allgather for `num_input_tokens_seen` is still getting stuck with distributed training, since the tensors are still on CPU and have not been moved to device in `_prepare_inputs` yet. This still results in torch.distributed errors such as:
```
File ".../site-packages/torch/distributed/distributed_c10d.py", line 2948, in all_gather_into_tensor
     work = group._allgather_base(output_tensor, input_tensor, opts)
ValueError: Tensors must be CUDA and dense
```

This PR simply moves the token count to `self.args.device` and then moves them back to cpu after gathering.
This problem was also mentioned in the discussion of issue #28791, but the issue was closed when the padding was fixed.


## Who can review?

@pacman100
@muellerzr

Comments:
can you run `pip install -e .[quality]; make style; make quality`?

@muellerzr Thanks for the pointer! I ran the code formatter as suggested.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31974). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Meta FAIR Chameleon 7b and 30b
Author: AmericanPresidentJimmyCarter
Pull request: no
Labels: name: New model

### Model description

The model is a LLaMA style architecture with a VQGAN for image input and generation. It is also likely to be finetuned for patch input for images similar to Fuyu, so it would be a good idea to have the implementation flexible for different types of image input. The weights are available under a research license.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/facebookresearch/chameleon

https://ai.meta.com/blog/meta-fair-research-new-releases/

Comments:
Hi! Adding Chameleon to transformers is work in progress and should be finalized soon 🤗
Enable interleaved image-image generation w/ Chameleon & Anole
Author: leloykun
Pull request: yes

# What does this PR do?

- Adds modelling for the `VQVAE` decoder & also includes it in the conversion step.
  - I've also uploaded the converted model here: https://huggingface.co/leloy/Anole-7b-v0.1-hf
- Adds support for decoding the BPE tokens -> discrete image tokens ->  pixel values -> PIL object
- Reimplements Chameleon's FSM to be more compatible with Transformers and Outlines (for structured generation)


Links:
- [Official Anole repo by GAIR-NLP team](https://github.com/GAIR-NLP/anole)
- [Official Anole model by GAIR-NLP team](https://huggingface.co/GAIR/Anole-7b-v0.1)
- [Transformers-compatible Anole with image generation support](https://huggingface.co/leloy/Anole-7b-v0.1-hf)

(partially) Implements # (issue)
- https://github.com/GAIR-NLP/anole/issues/18

@ArthurZucker @zucchini-nlp @JoyBoy-Su

Comments:
@ArthurZucker Indeed - this would be a great addition to the library! 

> I didn't really go through the changes, but one thing to consider is to move image-postprocessing logic into processors

@zucchini-nlp This should be handled in a similar way to how we do post-processing with tokenizers: if it involves post processing of images, then the logic should be available with the image processor, but we can then post-process with the processor. For example, most processors will have a `batch_decode` method. 

Hi @zucchini-nlp & @amyeroberts! I've already moved some of the image-postprocessing step to the processor. But I left the token -> pixel value decoding step in the model cuz it uses the vqvae model. This makes the interface a bit weirder tho:

1. Feed prompt, pixel values, & other inputs to model and get a sequence of tokens
2. Preprocess tokens into text and image segments
3. Pass the image segments back to the model to get pixel values
4. Pass text segments and pixel values to processor to get final results

What do you guys think would be a better way to do this?

> However, we need to think about how to return image pixels in interleaved-mode, as the base GenerateDecoderOnlyOutput doesn't allow it. @gante here as well to, WDYT about it?

In these cases we usually overload the base class in the modeling file! [Example](https://github.com/gante/transformers/blob/a1a34657d41627b21dddf2bf9cc55941329a60b6/src/transformers/models/perceiver/modeling_perceiver.py#L56)

@zucchini-nlp 

@leloykun Oops, seems like this PR was into Chameleon branch. I merged chameleon PR and the branch was deleted.

Can you pls reopen in into `main`?
Apply the change deprecating `resume_download` param to unchanged lines to be consistent across the source code
Author: unscrew
Pull request: yes

Apply the change deprecating 'resume_download' param to unchanged lines to be consistent across the source code

# What does this PR do?

Apply the change made in [PR - Respect resume_download deprecation #30620](https://github.com/huggingface/transformers/pull/30620) to the rest of the source code where the param is used.

<!-- Remove if not applicable -->

Follow-up fixes to [issue #30618](https://github.com/huggingface/transformers/issues/30618)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Tagging the author of [PR - Respect resume_download deprecation #30620](https://github.com/huggingface/transformers/pull/30620) @Wauplin


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31333). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add prefix space ignored in llama #29625
Author: itazap
Pull request: yes

# What does this PR do?

- add param add_prefix_space to llama tokenizer

Fixes #29625

## Before submitting
- [x] modified test_no_prefix_space to test test_no_prefix_space=False
## Who can review?
 @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30964). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Only set max_length if max_new_tokens is None for TextGenerationPipeline
Author: TRT-MichaelO
Pull request: yes

# What does this PR do?

Get rid of the `Both max_new_tokens (=512) and max_length (=512) seem to have been set. max_new_tokens will take precedence. Please refer to the documentation for more information.` warning for `TextGenerationPipeline`. There is no change in behaviour, just makes it a bit clearer to the user what is going on.

No added tests or changes to documentation


Comments:
Will reopen in another PR.
Fixed `log messages` that are resulting in TypeError due to too many arguments
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
`FutureWarning` should be used inside `warnings.warn()`, but at some places i see that it is used inside logging statements. So, I fixed them and some of the log statements have comma (`,`) between strings, which leads to considering them as multiple positional arguments, resulting in `TypeErrors`.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
Minor typo in ImageClassificationPipeline
Author: moses
Pull request: no
Labels: name: bug

### System Info

Issue is on HEAD on the main branch, been around for a while

Transformers env not really relevant, but for completeness:
- `transformers` version: 4.32.1
- Platform: macOS-14.5-arm64-arm-64bit
- Python version: 3.11.5
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Narsil There's a typo in ImageClassificationPipeline, that I don't believe usually has any effect, but it's confusing when reading the code. I will provide a PR with a fix.

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This typo does not cause behavior change, because of the `or self.model.config.num_labels == 1`/`> 1` condition.

### Expected behavior

The `problem_type` condition should match the `num_labels` condition

Comments:
doc: fix broken BEiT and DiNAT model links on Backbone page
Author: dvrogozh
Pull request: yes

Fixing broken links for BEiT and DiNAT models on backbones page:
* https://huggingface.co/docs/transformers/main/en/main_classes/backbones

CC: @stevhliu @SunMarc

Comments:
Added a typo fix `s/ConvNet/ConvNext`. Link was good for this model.