noamaanMulla-03/transformers-issues huggingface dataset:
Set pad_token in run_glue_no_trainer.py #28534
Author: JINO-ROHIT
Pull request: yes

# What does this PR do?

This PR adds the pad tokenizer in the run_glue_no_trainer.py script under examples.

Fixes #28534 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @ArthurZucker 


Comments:
done! @amyeroberts 

Exception: Found the following copy inconsistencies:
- tests/models\roc_bert\test_tokenization_roc_bert.py: copy does not match models.bert.test_tokenization_bert.BertTokenizationTest.test_is_whitespace at line 167
Run `make fix-copies` or `python utils/check_copies.py --fix_and_overwrite` to fix them.
make: *** [Makefile:38: repo-consistency] Error 1

I get this error but i tried running both commands, still gives the same error, how to resolve this? @amyeroberts 

There's something funny going on with the formatting here - files like src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py shouldn't be modified when calling `make fixup`.
and you shouldn't need to run `make fix-copies` for this PR. 

In your environment, make sure you have all of the necessary formatting libraries with `pip install -e .[testing]`. Once that's done, I'd undo the formatting changes to the unrelated files and then try running `make fix up` again 



@amyeroberts ive recreated my env, can you help me out with undoing the formatting files?
BartForSequenceClassification: Use eos_token or cls_token?
Author: ekagra-ranjan
Pull request: no
Labels: name: bug

### System Info

NA

### Who can help?

@patil-suraj 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The [BartTokenizer doc](https://huggingface.co/docs/transformers/v4.21.1/en/model_doc/bart#transformers.BartTokenizer) mentions that `cls_token` is attached to the beginning of the input sentence and is used as the token for sequence classification purposes. However, in the HF code it is picking the last eos_token:
https://github.com/huggingface/transformers/blob/v4.21.1/src/transformers/models/bart/modeling_bart.py#L1514-L1521

### Expected behavior

The [Bart paper (Sec 3.1)](https://arxiv.org/pdf/1910.13461.pdf) matches with what the code does, i.e. the last token is to be used as the classification token.

```
3.1 Sequence Classification Tasks
For sequence classification tasks, the same input is fed
into the encoder and decoder, and the final hidden state
of the final decoder token is fed into new multi-class
linear classifier. This approach is related to the CLS
token in BERT; however we add the additional token
to the end so that representation for the token in the
decoder can attend to decoder states from the complete
input (Figure 3a).
```

Should I update the Bart doc with this method of classification?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Yes, I also think so. It should be fixed, otherwise it makes confusion for beginners.
Extra space in Llama2 Tokenizer default_chat_template
Author: chenjoya
Pull request: no

### System Info

`transformers` version: 4.38.2

### Problem

Just find this minor problem. There should not be an extra space between eos token and content: https://github.com/huggingface/transformers/blob/965cf677695dd363285831afca8cf479cf0c600c/src/transformers/models/llama/tokenization_llama_fast.py#L269

It should be 
```python
"{{ ' '  + content.strip() + eos_token }}"
```

### Reproduction

We can test Llama-2-Chat to know.

```python
import transformers
model = transformers.LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
tokenizer = transformers.AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
chat = [{'role': 'user', 'content': 'Hello, how are you? Please answer concisely.'}]
input_ids = tokenizer.apply_chat_template(chat, return_tensors='pt').cuda()
model.to('cuda')
generate_ids = model.generate(input_ids)
```

We can get
```
>>> generate_ids
tensor([[    1,   518, 25580, 29962, 15043, 29892,   920,   526,   366, 29973,
          3529,  1234,  3022,   275,   873, 29889,   518, 29914, 25580, 29962,
         29871, 15043, 29991,   306, 29915, 29885,   925,   385,   319, 29902,
         29892,   306,  1016, 29915, 29873,   505, 21737,   470, 23023,  1080,
         29892,   577,   306,   508, 29915, 29873,  1234,   376, 29902, 29915,
         29885,  2691, 29892,  6452,   366, 29908,   470,   738,   916,  3022,
           895,  2933, 29889,  2398, 29892,   306, 29915, 29885,  1244,   304,
          1371,   366,   411,   738,  5155,   470,  9595,   366,  1122,   505,
         29991,  1128,   508,   306,  6985,   366,  9826, 29973,     2]],
       device='cuda:0')
```

As you can see, before the EOS token 2, the token id 29973 is `?`, rather than space (which corresponds to token id 29871).

```python
tokenizer._convert_id_to_token(29973) # is ?
```

Comments:
Cc @Rocketknight1 

Hi @chenjoya, thanks for spotting it! I did some investigation and I think you're right. However, this would need to be fixed in the LLaMA model repos as well as in the codebase, because the default_chat_template used in `transformers` is only a legacy fallback for models that don't have a proper `tokenizer.chat_template`. Would you be willing to make the relevant PRs to `transformers` and the model repos?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[SegFormer] Add support for segmentation masks with one label
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR makes it possible to fine-tune SegFormer in case you have a mask containing only a single value, i.e. your mask could look like [[255, 0], [0, 255]]. In this case, config.num_labels = 1 and the ignore index is 255.

If this works fine, then we can add it to any other model supported by `AutoModelForSemanticSegmentation`.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

How soon will this PR be done? And why using BCEWithLogitsLoss not Dice loss?

when is this going to be closed. Our team would like to use it

when this becomes active, how would one use it.

thank you nielsrogge for sending for review.

hello @NielsRogge 

hope I didnt disturb

I tried to peel off the classifier in pytorch and change the output channels to one, then manually compute the loss instead of getting it from the segformer huggingface object, so basically I just got the output and then did a dice loss myself.

So i tried to write bin seg myself kinda, and I started to get a bunch of negative values.

Any idea why that happened, and how I could fix it? I mean, I replaced the last conv2d layer.

Is this now automatically using dice loss when we set `num_labels = 1`? Maybe I missed it but it seems the documentation doesn't explain it.

HHi @aegonwolf,

When config.num_labels = 1, the binary cross-entropy loss is used, as can be seen here: https://github.com/huggingface/transformers/blob/1689aea73346816b936b84932e12b774974e61a6/src/transformers/models/segformer/modeling_segformer.py#L813-L817.

What about a sigmoid function at the end or any way of scaling outputs to [0, 1] range?

@nikolaJovisic the BCE loss uses sigmoid.

@NielsRogge Thank you for your amazing work. However I am still struggling with this task, can you make an example notebook on how to finetune Segformer for 1 class only please? Thank you a lot! 

@NielsRogge modeling_segformer.py script is changed, but modeling_tf_segformer.py seems to be unchanged, I get an error there when I try to train model for 1 class only.

Any update on this task ? Any example notebook ?

> Any update on this task ? Any example notebook ?

Hi,

Back when I was working on this, the BinaryCrossentropy variant was merged and the issue was closed, but if I remember correctly it was done only for PyTorch version, so I implemented the Tensorflow version myself on a fork of this repository, you can find it on my profile, never made the effort to open a PR tho. Hope this might help. The fine-tuning itself was straightforward after that if I remember correctly.

Best wishes,
Nikola

@nikolaJovisic would be great if you could open a PR for it!
Set pad_token in run_glue_no_trainer.py #28534
Author: JINO-ROHIT
Pull request: yes

# What does this PR do?
This PR adds the pad tokenizer in the run_glue_no_trainer.py script under examples.

Fixes #28534

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @ArthurZucker 


Comments:
Couldn't connect to `https://huggingface.co`.
Author: chrysanthemum-boy
Pull request: no

### System Info

- `transformers` version: 4.38.1
- Platform: Linux-6.5.0-27-generic-x86_64-with-glibc2.38
- Python version: 3.10.14
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Error code can be found in `src/utils/hub.py`.

134 ` _default_endpoint = "https://hub-ci.huggingface.co" if _staging_mode else "https://huggingface.co"`


### Expected behavior

` _default_endpoint = "https://hub-ci.huggingface.com" if _staging_mode else "https://huggingface.com"`

Comments:
I'm sorry, it's my fault.
Refactor doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

Main changes:

  - run doctest in multiple independent jobs
    - motivation: to avoid GPU OOM (or other environment states) affect subsequent tests
    - this is done by using a map between directories and their files
    - The files in `docs/source/en/model_doc` or `docs/source/en/tasks` are **NOT** grouped together with other files in the
same directory: the objective is to run doctest against them in independent GitHub Actions jobs. 
  - Use the same trick as in #28773 to allow GitHub Actions to generate more than 256 jobs in matrix
    - (as now we have many more (smaller) jobs instead of a single big job)

A file `doc_test_results` is saved and uploaded as artifact so we can always have a way to check.

A (example) run page: see [here](https://github.com/huggingface/transformers/actions/runs/8659176586)
A full run page: see [here](https://github.com/huggingface/transformers/actions/runs/8659540805)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30210). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merge now :-)
Unused or unrecognized kwargs: padding, max_length, truncation.
Author: Excuses123
Pull request: no
Labels: name: bug

name: Multimodal

### System Info

- `transformers` version: 4.39.2
- Platform: Linux-3.10.0-1160.99.1.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Fine-tuning the CLIP model. Previously, I passed both text and images to the processor and specified some parameters for the text: padding, max_length, truncation. Everything was working fine before upgrading to version 4.39. However, after the upgrade, I encountered a warning that affects the training process. How should I resolve this? 
![image](https://github.com/huggingface/transformers/assets/22993056/2d8d8ece-f190-4432-bcfe-43ea3439c848)

### Expected behavior

Do I need to separate the text and images for processing and use processor.tokenizer and processor.image_processor separately?

Comments:
Same problem is occurring.
Downgrading to 4.38.2 will solve the problem.

Meet the same problem

Hi @Excuses123, thanks for raising this issue! 

Could you share a minimal reproducer? 

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在假期结束后，尽快给你回复。

> Hi @Excuses123, thanks for raising this issue!
> 
> Could you share a minimal reproducer?

Hello, here is a minimal reproducible example.

import torch
import requests
from PIL import Image
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("openai/clip-vit-large-patch14-336")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor(text="a photo of a cat", 
          images=image, 
          max_length=64, 
          padding='max_length', 
          truncation=True, 
          return_tensors='pt')

Thanks for sharing @Excuses123. The warnings have started because of new input validation on the image processor. This is good as it shows tokenizer kwargs were being passed to the image processor when they shouldn't be! I've opened a PR to address this 

just to Change code
for the /**your paht****/transformers/models/clip/processing_clip.py 
     function : def __call__(self, text=None, images=None, return_tensors=None, **kwargs):
     line:  104   image_features = self.image_processor(images, return_tensors=return_tensors, **kwargs)
    the new code :  ->  image_features = self.image_processor(images, return_tensors=return_tensors)  # delete the others kwargs
Separate out kwargs in processor
Author: amyeroberts
Pull request: yes

# What does this PR do?

Separates out the kwargs that are passed to the processor into tokenizer and image processor kwargs. 

This solves two issues:
* Removes warnings from validation checks that now happen in the image processor (nice proof they're working!)
* Prevents passing kwargs which have a name clash between the two processing classes


Fixes #30106


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30193). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix doctest more (for `docs/source/en`)
Author: ydshieh
Pull request: yes

# What does this PR do?

A continuation of #30216, but this time for files in `docs/source/en` instead of `src`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30247). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Awq`] Add peft support for AWQ
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, adds Trainer + AWQ + PEFT support.
Needs to be merged at the same time as: https://github.com/huggingface/peft/pull/1399
The tests are directly added on https://github.com/huggingface/peft/pull/1399

cc @casper-hansen @pacman100 @amyeroberts  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28987). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts - we just got a release from autoawq ! this is ready for review 🙏 
After merging this, I'll merge https://github.com/huggingface/peft/pull/1399 :) 

cool！
update github actions packages' version to suppress warnings
Author: ydshieh
Pull request: yes

# What does this PR do?

There are so many 

> Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20

warning in our GitHub Actions run pages, for example, [here](https://github.com/huggingface/transformers/actions/runs/8682815917)

This PR updates the following:

actions/upload-artifact
actions/download-artifact
actions/checkout

from `@v3` to `@v4` to suppress these warnings

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30249). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MPS memory leak?
Author: tonystratum
Pull request: no

### System Info

```
- `transformers` version: 4.39.3
- Platform: macOS-14.4.1-arm64-arm-64bit
- Python version: 3.11.8
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0a0+git42cbe02 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (MPS)
- Using distributed or parallel set-up in script?: No
```

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Trying to translate from eng_Latn to deu_Latn with facebook/nllb-200-distilled-600M. The inference code is pretty much the boilerplate code from the [model page (huggingface)](https://huggingface.co/facebook/nllb-200-distilled-600M), with some logic on top. The code and input data is in this [gist](https://gist.github.com/tonystratum/84b4b4a892b2217932f1b6080fe4c24a).

When the `prompt_works` is passed, the expected result is produced and no memory issues are observed. When `prompt_not_works` is passed, the inference slows down and memory usage rapidly increases.

The demo below runs code equivalent to the provided gist and produces equivalent behaviour.

https://github.com/huggingface/transformers/assets/61351346/914f20ce-c1fb-4b89-8455-f6bc58981e73

Thanks for your help in advance!

### Expected behavior

Inferencing the contents of `prompt_not_works` in the gist provided above runs successfully without memory issues.

Comments:
Add Idefics2
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds the Idefics 2 model into transformers

cc @VictorSanh @ArthurZucker 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30253). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Update recurrent_gemma.md for some minor nits
Author: sayakpaul
Pull request: yes

# What does this PR do?

Just some minor nits in the RecurrentGemma doc. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30238). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove incorrect arg in codellama doctest
Author: Rocketknight1
Pull request: yes

`return_type = 1` is not a valid kwarg for the text generation pipeline because `ReturnType` is an `Enum`, and directly passing integer values does not actually work for this! The doctest works fine without this arg.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30257). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.