noamaanMulla-03/transformers-issues huggingface dataset:
Unable to pass encoder_outputs to generate calls
Author: gabisurita
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: Github Main branch

### Who can help
 TextGeneration: @TevenLeScao 
 

## Information

Model I am using (Bert, XLNet ...): T5

I'm unable to pass preconputed encoder_outputs to the `.generate()` method.

I've tried defining:

```python
model_kwargs = {"encoder_outputs": encoder_outputs}
output = model.generate(model_kwargs)
```
But I've noticed some validation errors for `input_ids`. Even if I replace input_ids with dummy values, I've noticed the model_kwargs is always replaced here:

https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L448

I think it can be fixed (without optimization) by just replacing:

```python
model_kwargs["encoder_outputs"] = encoder_outputs
```

with:

```python
model_kwargs.setdefault("encoder_outputs", encoder_outputs)
```

If you agree I can try to open a PR to fix this.

Best,

Comments:
Hey @gabisurita - I understand that one might want to forward encoder_outputs in the generate function. However, adding such a possibility opens the door for many problems in case `beam_search` is chosen. We are currently working on a bigger refactor that should solve this problem by a better design choice of `generate()`. I'm afraid that this will still take ~3,4 weeks to complete though.

Hi @patrickvonplaten,

I've noticed that your PR refactoring generate was merged. It seems a big improvement, thank you! Still, `input_ids` is still required or overridden by an empty tensor. It's still not clear to me how can I use the new API with `encoder_outputs` or `imput_embeds`.

Hey @gabisurita - if you look at the tests, you can now directly use `beam_search` instead of generate for your use case I think :-). Here the tests: https://github.com/huggingface/transformers/blob/a7d73cfdd497d7bf6c9336452decacf540c46e20/tests/test_generation_utils.py#L295 

From the tests, it should be quite easy to understand how to use `beam_search` directly I think :-) 

Let me know if that helps!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I got a similar error "Make sure that `model_kwargs` include `encoder_outputs` of type `ModelOutput`." This worked for me:

downgrade transformers to 3.0.2
[tests] use the correct label column name  
Author: faaany
Pull request: yes

## What does this PR do?
```bash
pytest tests/deepspeed/test_model_zoo.py::TestDeepSpeedModelZoo::test_zero_to_fp32_zero2_img_clas_vit -rA
```
The test above fails with the following message: 
```bash
stderr: Traceback (most recent call last):
stderr:   File "/mnt/code/examples/pytorch/image-classification/run_image_classification.py", line 451, in <module>
stderr:     main()
stderr:   File "/mnt/code/examples/pytorch/image-classification/run_image_classification.py", line 282, in main
stderr:     raise ValueError(
stderr: ValueError: --label_column_name label not found in dataset 'hf-internal-testing/cats_vs_dogs_sample'. Make sure to set `--label_column_name` to the correct text column - one of image, labels.
```
The fact is that the test dataset `hf-internal-testing/cats_vs_dogs_sample` contains 2 columns: `image` and `labels`. Since the default label name is "label" for `run_image_classification.py`, we need to explicitly set the label column name for this task. 

After the fix, the test passes. Below is a code snippet from the successful test result:
```bash
stderr: [INFO|trainer.py:2085] 2024-06-12 16:04:51,440 >>
stderr: 
stderr: Training completed. Do not forget to share your model on huggingface.co/models =)
stderr: 
stderr: 
100%|██████████| 10/10 [00:06<00:00,  1.61it/s]
stderr: [INFO|trainer.py:3057] 2024-06-12 16:04:51,445 >> Saving model checkpoint to /tmp/tmp6hh60e8s
stderr: [INFO|configuration_utils.py:471] 2024-06-12 16:04:51,445 >> Configuration saved in /tmp/tmp6hh60e8s/config.json
stderr: [INFO|modeling_utils.py:2474] 2024-06-12 16:04:51,448 >> Model weights saved in /tmp/tmp6hh60e8s/model.safetensors
stderr: [INFO|image_processing_utils.py:257] 2024-06-12 16:04:51,448 >> Image processor saved in /tmp/tmp6hh60e8s/preprocessor_config.json
stderr: [INFO|modelcard.py:450] 2024-06-12 16:04:51,894 >> Dropping the following result as it does not have all the necessary fields:
stderr: {'task': {'name': 'Image Classification', 'type': 'image-classification'}, 'dataset': {'name': 'hf-internal-testing/cats_vs_dogs_sample', 'type': 'cats_vs_dogs_sample', 'config': 'default', 'split': 'train', 'args': 'default'}}
================================================= short test summary info =================================================
PASSED tests/deepspeed/test_model_zoo.py::TestDeepSpeedModelZoo::test_zero_to_fp32_zero2_img_clas_vit
========================================= 1 passed, 1 warning in 86.02s (0:01:26) =========================================
```



Comments:
@amyeroberts @ydshieh could you take a look at this PR? Thx! 

my fault, I am not using the latest master code.
Fix the error message for importing bitsandbytes when not installed.
Author: 46319943
Pull request: yes

# What does this PR do?
As illustrated in https://github.com/huggingface/transformers/issues/31273, the error message for importing the bitsandbytes when the package is not installed should be raised to notify the user, instead of showing the problem about `torch`.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/31273

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
Integrations:
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada


Comments:
I initially thought that `transformers.integrations.bitsandbyte`s should only be used with `bitsandbytes`, requiring `bitsandbytes` to be installed.

Thank you for your clarification. I now understand that it is not necessary.
Fix idefics cache
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31373 and fixes #31380. We have to update past kv length in all cases, not only when `return_legacy_cache=True`. Also removed `past_length = past_key_values.seen_tokens` as it's deprecated and shows same length as `get_seq_length`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31377). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use huggingface_hub helper function to split state dict
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR uses the helper function from huggingface-hub to split a state dict into shards instead of `shard_checkpoint`. This will make maintenance easier for all HF libraries. Similar PR have been created in [accelerate](https://github.com/huggingface/accelerate/pull/2795) and [diffusers](https://github.com/huggingface/diffusers/pull/7830). 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31091). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Could you make `get_storage_id` public/documented since it is used in multiple places (not only when cleaning the state dict) @Wauplin ? Thanks ! 

> If I'm not mistaken there are similar methods for tf / flax in transformers that could benefit from it. Can be done in separate PRs though.

I'll have a look and do it in another PR after this PR gets merged and tested ! 

Just a quick question @Wauplin , the current helper function in huggingface-hub introduced a regression in transformers (not able to handle kiB, miB). Is is fine ? If so, I can modify the failing test : `test_checkpoint_sharding_local_bin`

> Could you make get_storage_id public/documented since it is used in multiple places (not only when cleaning the state dict) @Wauplin ? 

@SunMarc Done in https://github.com/huggingface/huggingface_hub/pull/2304

> he current helper function in huggingface-hub introduced a regression in transformers (not able to handle kiB, miB). Is is fine ?

I'm not a `transformers`'s maintainer but I would assume it's fine. I'm quite convinced those parameters are never used (expect for the default value) and even less using kib, mib, etc. notation. If others feel strongly about it, we could somehow mitigate it directly in `transformers` (typically with a `max_size.lower().replace("mib", "mb").replace("gib", "gb")` + a deprecation warning). But I'd prefer not to update `huggingface_hub` for those. 

> I'm not a transformers's maintainer but I would assume it's fine. I'm quite convinced those parameters are never used (expect for the default value) and even less using kib, mib, etc. notation. If others feel strongly about it, we could somehow mitigate it directly in transformers (typically with a max_size.lower().replace("mib", "mb").replace("gib", "gb") + a deprecation warning). But I'd prefer not to update huggingface_hub for those.

WDYT @amyeroberts ? 

> but I would assume it's fine. I'm quite convinced those parameters are never used (expect for the default value) and even less using kib, mib, etc. notation. If others feel strongly about it, we could somehow mitigate it directly in transformers (typically with a max_size.lower().replace("mib", "mb").replace("gib", "gb") + a deprecation warning). But I'd prefer not to update huggingface_hub for those.

@Wauplin @SunMarc Sounds good to me! Agreed, I don't think we need to handle a deprecation cycle here
feat(ci): add trufflehog secrets detection
Author: McPatate
Pull request: yes

### What does this PR do?

Adding a GH action to scan for leaked secrets on each commit.

### Context

`trufflehog` will scan the commit that triggered the CI for any token leak. `trufflehog` works with a large number of what they call "detectors", each of which will read the text from the commit to see if there is match for a token. For example, the hugging face detector will check for hf tokens and then query our `/api/whoami{-v2}` endpoint to check if the token is valid. If it detects a valid token, the CI will fail, informing you that you need to rotate the token given it leaked.

### What to do if the CI fails

If the CI errors, you should check the logs to see which token was leaked. After that, depending on the origin of the token, you should go to the token issuer and delete it, recreate it or rotate it (depending on the available features).
Once this is done, remove the token from the code and use an environment variable to load the secret or any other external configuration management system.

### References

- https://github.com/trufflesecurity/trufflehog
- https://github.com/marketplace/actions/trufflehog-oss

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31344). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Would be good to have some guidelines added to let users know what to do if this pipeline fails e.g. in the [contributing guidelines](https://github.com/huggingface/transformers/blob/6b11f89c6b3d15655a2264df73e453cf9692c62e/CONTRIBUTING.md#tests)

Would you like me to address this in this PR?

> What's the chance of having false positives on these runs?

From my experience I would say pretty low, on some occasions a sha could be misinterpreted as a secret. But it will not be considered "verified", so in most cases safe to ignore.

> Would you like me to address this in this PR?

Ideally yes. At least, a comment within this PR which we can link on issues. As, tbh, I'm not 100% sure what I should tell people to do if this fails 😅 

> From my experience I would say pretty low, on some occasions a sha could be misinterpreted as a secret. But it will not be considered "verified", so in most cases safe to ignore.

OK! 

> At least, a comment within this PR which we can link on issues.

Updated the PR description, lmk what you think.
LLM Cache 
Author: mostafamdy
Pull request: no
Labels: name: Feature request

### Feature request

Hi,

I'm working with an LLM model that generates text in stages. It stops after each section, performs an action, integrates the action's result into the generated text, and then continues. While this approach works, I'm facing a problem in time it takes.

I've tried using a static cache, but it actually increased processing time.

Is there a way to optimize this process?


### Motivation

.

### Your contribution

.

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

cc @gante 

I posted it in the forums. Thanks
can't import phi3config etc.
Author: Murasame-sway
Pull request: no

### System Info

transformers       4.40.1
python3.8

### Who can help?

@ArthurZucker @youn

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import Phi3Model, Phi3Config

### Expected behavior

hello, In the Transformers source code, I found two classes, phi3config and phi3model, as shown in Figure 1. However, when I tried to import these two classes in another environment with the transformers library installed, it failed, as shown in Figure 2. My transformers version is 4.40.1. can you help me？
![57f7204c2db2d5bf9b3bd4055e68c73](https://github.com/huggingface/transformers/assets/67264710/d980a7f1-b6a1-4196-b5f9-3422b1f60136)
![b16dc47102ceac2a033365a54cdc1fd](https://github.com/huggingface/transformers/assets/67264710/bb11fbc6-21bb-4f17-a2a3-5792298aed57)

Comments:
the Transformers source code is clone from main branch

Could you try either:
- Installing from source (with `pip install git+https://github.com/huggingface/transformers -U)
- Installing as an editable with a path towards your clone (with `pip install -e path-to-clone/transformers`)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'CommitOperationAdd' from 'huggingface_hub' (C:\Users\46213\anaconda3\lib\site-packages\huggingface_hub\__init__.py)
Author: wccccp
Pull request: no

### System Info

windows
python=3.9
transformers              4.23.1
pytorch                   1.13.0          py3.9_cuda11.6_cudnn8_0    pytorch

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


>>> import transformers
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\46213\anaconda3\lib\site-packages\transformers\__init__.py", line 30, in <module>
    from . import dependency_versions_check
  File "C:\Users\46213\anaconda3\lib\site-packages\transformers\dependency_versions_check.py", line 17, in <module>
    from .utils.versions import require_version, require_version_core
  File "C:\Users\46213\anaconda3\lib\site-packages\transformers\utils\__init__.py", line 48, in <module>
    from .hub import (
  File "C:\Users\46213\anaconda3\lib\site-packages\transformers\utils\hub.py", line 32, in <module>
    from huggingface_hub import (
ImportError: cannot import name 'CommitOperationAdd' from 'huggingface_hub' (C:\Users\46213\anaconda3\lib\site-packages\huggingface_hub\__init__.py)
>>>


### Expected behavior

please help me to solve the problem！！！

Comments:
Could you do `pip show huggingface_hub`?

> 

(base) C:\Users\46213>pip show huggingface_hub
Name: huggingface-hub
Version: 0.10.1
Summary: Client library to download and publish models, datasets and other repos on the huggingface.co hub
Home-page: https://github.com/huggingface/huggingface_hub
Author: Hugging Face, Inc.
Author-email: julien@huggingface.co
License: Apache
Location: c:\users\46213\anaconda3\lib\site-packages
Requires: typing-extensions, requests, filelock, tqdm, packaging, pyyaml
Required-by: transformers, ltp, evaluate, datasets

For anyone finding this now, if you installed `transformers` and/or `huggingface_hub` with pip, try re-installing with conda. That solved it for me.

i.e. `conda install -c conda-forge transformers huggingface_hub`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

delete package "charser_normalizer" manually then reinstall package "charser_normalizer" may work.
Make chat templates part of ProcessorMixin
Author: Rocketknight1
Pull request: yes

Right now, IDEFICS2 supports chat templates, but in class-specific code. In this PR, we'll try to move that support to the generic `ProcessorMixin` so that we don't have to keep duplicating it as more VLMs (like LLaVA) start to need chat templates.

Draft PR for now until I make sure this doesn't break things!

Comments:
Hi @Rocketknight1, very nice! FYI, I'm working on standardizing the processors here https://github.com/huggingface/transformers/pull/30511 and I was thinking how to make this compatible with apply_chat_template - thought you might want to take a look at the design here, and I'll keep an eye here as well :) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30744). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@molbap I didn't see your PR, but we should definitely use your approach! Supporting chat templates should be simple enough - all they require is moving `apply_chat_template` to `ProcessorMixin`, and then supporting `chat_template` as an attribute for any processor that wants to use them. Do you want me to pause my PR until yours is merged?

I don't think you need to pause yours! I'll try and get mine merged soon - it's already approved, just need to make sure nothing breaks 

Cool! I'll probably wait until yours is merged, in that case, and then adding chat templates should be clean and straightforward.

This should be ready - cc @amyeroberts for core maintainer review! I'm speeding things up because the lack of this feature is causing some user issues, and we have a lot of VLMs coming.
logging format
Author: wwwbq
Pull request: no

Hello, I would like to change the format of the logger in the transformers  when using AutoTokenizer. from_pretrained(). May I ask where the enable_explicit_format function in the logging.py file was called?

Comments:
Hi @wwwbq, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

You can find examples of how to use `enable_explicit_format` in some of our examples e.g. here: https://github.com/huggingface/transformers/blob/15b3923d656cf3e1a0c20d3ddc356eb89a703ce0/examples/pytorch/question-answering/run_qa.py#L261
Add missing French translation of tutoriel_pipeline.md
Author: jadechoghari
Pull request: yes

# What does this PR do?

This pull request adds a French translation of the `pipeline_tutorial` from English to French.

## Changes made
- Translated `pipeline_tutorial.md` from English to French.
- Updated the `_toctree.yml` to include the French version of the pipeline tutorial.

## Why is this change important?
Providing documentation in multiple languages helps make the information accessible to a broader audience and supports users who are more comfortable reading in French. 🇫🇷
More docs to come. 🤗

## Review
Like last time I will tag:
@amyeroberts (for the merge ✅)
@younesbelkada (for the review 🇫🇷)



Comments:
Great :) !, will just leave the pipeline in masculine form.
Should I resolve?
Thanks !

Sounds great to leave that, thanks for explaining !

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31396). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[QoL fix] [Image processing] Add warning on assumption of channel dim
Author: aliencaocao
Pull request: yes

Sometimes the input image can have height or width 1 and in HWC format. One example is if you `np.asarray(PIL.Image)`. The current function to infer channel dim assumes the CHW silently which can cause confusion to users as the image will be wrongly processed as a grayscale image, which will raise further, harder to interpret errors when running through models.

This PR adds a warning when this happens. ~It also set the channel dim to last if inputs are PIL.Image as they are always in HWC after `np.asarray`, avoiding ambiguity in this case.~

The default behaviour of assuming first dim as the channel has not changed to prevent any breaking changes.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts 

Comments: