noamaanMulla-03/transformers-issues huggingface dataset:
[SDPA] Make sure attn mask creation is always done on CPU
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

Many SDPA tests currently fail. E.g. when running:
```
CUDA_VISIBLE_DEVICES="1" RUN_SLOW=1 pytest tests/models/whisper/test_modeling_whisper.py::WhisperStandaloneDecoderModelTest::test_eager_matches_sdpa_inference_0_float16 -sv
```

We get the following error message:
```
>       range_tensor[range_tensor >= indices] = 0
E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
```

Until now SDPA tests were not run on our Circle CI because we test on PyTorch 2.1.0, but SDPA needs PyTorch 2.1.1 (see [here](https://github.com/huggingface/transformers/blob/3b742ea84cfc32432d60c0b65c886576ef736833/src/transformers/utils/import_utils.py#L275))

This PR:
- 1. Makes sure the sdpa attention_mask is always created on CPU
- 2. That we test on PyTorch 2.1.1
- 3. It fixes a "FileNotFoundError" for a Whisper slow test (see [here](https://github.com/huggingface/transformers/actions/runs/7442488193/job/20246174017#step:9:1734))

Comments:
@ydshieh Also I don't fully understand why we use the `huggingface/transformers-pytorch-gpu` Docker image for the Pipeline tests here https://github.com/huggingface/transformers/blob/3b742ea84cfc32432d60c0b65c886576ef736833/.github/workflows/self-scheduled.yml#L248 , 
but we use the `huggingface/transformers-all-latest-gpu` image for all other tests. Should we maybe change the image for the pipeline tests? 

> why we use the huggingface/transformers-pytorch-gpu

IIRC, this image doesn't have `tensorflow` stuff, but `transformers-all-latest-gpu` has both torch/tf. In theory, we can try to use only `transformers-all-latest-gpu`. But it's not a bad idea if we have environments that has only torch or only tf to test.


(there is also a job `run_pipelines_tf_gpu` which uses `transformers-tensorflow-gpu`.)



Moved the fix of Whisper's slow test here: https://github.com/huggingface/transformers/pull/28407 (cc @ydshieh) 

@patrickvonplaten FYI The image fails to build due to `intel_extension_for_pytorch`, so we are still using the old image (therefore torch 2.1.0)

I will have to check what happens

```
ERROR: failed to solve: process "sh -lc python3 -m pip install --no-cache-dir intel_extension_for_pytorch==$INTEL_TORCH_EXT -f https://developer.intel.com/ipex-whl-stable-cpu" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to solve: process "sh -lc python3 -m pip install --no-cache-dir intel_extension_for_pytorch==$INTEL_TORCH_EXT -f https://developer.intel.com/ipex-whl-stable-cpu" did not complete successfully: exit code: 1
```
ValueError: Tokenizer class CPMAntTokenizer does not exist or is not currently imported.
Author: jq460494839
Pull request: no

### System Info

transformers==4.35.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. download cpm-ant-10b from huggingface
2. load cpm-ant tokenizer locally

### Expected behavior

Traceback (most recent call last):
  File "/opt/projects/FastChat/fastchat/train/train.py", line 301, in <module>
    train()
  File "/opt/projects/FastChat/fastchat/train/train.py", line 273, in train
    tokenizer = transformers.AutoTokenizer.from_pretrained(
  File "/root/miniconda/envs/torch_npu/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 766, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class CPMAntTokenizer does not exist or is not currently imported.

Comments:
```python
from transformers import CpmAntTokenizer
```

is what you are looking for

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Enable multi-label image classification in pipeline
Author: amyeroberts
Pull request: yes

# What does this PR do?

Enables multilabel image classification in the pipeline and explicity specifying the activation function applied to the model's logits - matching the logic for [text classification pipeline](https://github.com/huggingface/transformers/blob/ffd3710391c0700a3957f0cdf2c99bc5ae966c70/src/transformers/pipelines/text_classification.py#L195).

Fixes https://github.com/huggingface/huggingface.js/issues/429


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
@Narsil  - I've added applying sigmoid function to the model outputs for image classification matching the text classification pipeline, per your [suggestion here](https://github.com/huggingface/huggingface.js/issues/429#issuecomment-1880625080). 

Agree controlling through a config would be a good idea -- it is partially controlled through `model.config.problem_type` here. Is this enough or would you like the activation function to be more explicitly defined in the config?
Set `cache_dir` for `evaluate.load()` in example scripts
Author: aphedges
Pull request: yes

# What does this PR do?

While using `run_clm.py`,[^1] I noticed that some files were being added to my global cache, not the local cache. I set the `cache_dir` parameter for the one call to `evaluate.load()`, which partially solved the problem. I figured that while I was fixing the one script upstream, I might as well fix the problem in all other example scripts that I could.

There are still some files being added to my global cache, but this appears to be a bug in `evaluate` itself. This commit at least moves some of the files into the local cache, which is better than before.

To create this PR, I made the following regex-based transformation: `evaluate\.load\((.*?)\)` -> `evaluate\.load\($1,
cache_dir=model_args.cache_dir\)`. After using that, I manually fixed all modified files with `ruff` serving as useful guidance. During the process, I removed one existing usage of the `cache_dir` parameter in a script that did not have a corresponding `--cache-dir` argument declared.

[^1]: I specifically used `pytorch/language-modeling/run_clm.py` from v4.34.1 of the library. For the original code, see the following URL: https://github.com/huggingface/transformers/tree/acc394c4f5e1283c19783581790b3dc3105a3697/examples/pytorch/language-modeling/run_clm.py.

## Who can review?

Maintained examples:

- PyTorch:
  - text models: @ArthurZucker
- TensorFlow: @Rocketknight1

Comments:
I would appreciate if you could make sure to preserve my commit message (with any needed changes) when merging so it ends up in the Git log.
Correctly resolve trust_remote_code=None for AutoTokenizer
Author: Rocketknight1
Pull request: yes

If `trust_remote_code` is left at the default `None` in `AutoTokenizer.from_pretrained()`, an error is thrown if you try to load a repo that requires remote code, rather than the correct dialog box being displayed. This PR fixes that issue.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28419). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I tried, but I couldn't figure out a way to test it, and I don't think the previous test authors could either! The test would need to check that `input()` is called, but we can't really check that - we can only check for the error that's raised when it times out. This is what happens in [the existing tests](https://github.com/huggingface/transformers/blob/main/tests/models/auto/test_tokenization_auto.py#L306), and that's why the tests didn't catch the problem - the error is raised without ever calling `input()`, and the test only checks for the error, so it passed.
Share custom pipelines on Huggingface Hub
Author: fhamborg
Pull request: no

# 🚀 Feature request

Publish and share custom pipelines on Huggingface Hub.

## Motivation

It seems that the models section on Huggingface Hub is currently useful only for use cases supported out-of-the-box by the huggingface/transformers library, e.g., for sequence classification, masked pretraining, and sequence-to-sequence classification (e.g., translation). However, if one has a different use case, such as target-dependent sentiment classification (TSC), it seems the Hub is not the right platform to publish the model, since neither the Inference API will work not will users not be able to use AutoModel.from_pretrained("somehuggingfacehuburltoamodel") work out-of-the-box. 

For example, in the case of TSC, the input consists of two parts: a sentence and a target phrase within, whereas the latter is typically expressed as two char-based indexes. For example, an input might be ("I like Bert but I hate Robert", 7, 11), and the model should then identify the sentiment of this sentence towards the target "Bert" (defined by the indexes 7 and 11). 

It seems to me that [pipelines](https://huggingface.co/docs/transformers/master/add_new_pipeline) are a solution to this, where one could identify how to preprocess such inputs and then correspondingly postprocess the model's outputs. However, I could not find a way to publish one's pipelines. This in turn makes Huggingface Hub currently not really usable for any use case different to those supported also by the Huggingface Transformers library. 

## Your contribution

This issue seems to be more related to the huggingface hub, so it would be interesting to see your thoughts first. If there need to be changes done in the transformers library, I'm glad to talk about necessary changes and how I can help :)


Comments:
cc @osanseviero and @LysandreJik!

Hey @fhamborg! That would be a super cool complement to @sgugger's great work on making models/tokenizers/configurations accessible from the hub!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello, are there any updates on this issue?

Hello @ttranslit, yes this is supported now! Please read the following doc to know more about it: https://huggingface.co/docs/transformers/main/en/add_new_pipeline#how-to-create-a-custom-pipeline
README: install transformers from conda-forge channel
Author: kevherro
Pull request: yes

# What does this PR do?

Switch to the conda-forge channel for transformer installation, as the huggingface channel does not offer the latest version.

Fixes #28248 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@ArthurZucker I appreciate your review!

> we should probably link @LysandreJik comment from the issue

"In the future we'll likely remove the huggingface channel support for conda unless we see explicit demand here; if that's the case, please put a thumbs up here (we'll monitor), otherwise, the conda-forge channel is usually very up to date." ([here](https://github.com/huggingface/transformers/issues/28248#issuecomment-1869765891))

> mention that the huggingface channel is deprecated for transformers

Good callout. Should this go in the README as well?


If you're up for it, would you also mind updating this in the [Installation](https://huggingface.co/docs/transformers/installation#install-with-conda) docs? 🙂 

@stevhliu I think I did this correctly! Note that this also cleans up some whitespaces and newlines in the changed files.

My pleasure! 

Just seeing this now, but thanks for the PR @kevherro!
Doc
Author: jiqing-feng
Pull request: yes

Hi @amyeroberts 

We need to update some lib versions for CPU training in the docs, would you please help review it? Thx!

Comments:
@stevhliu please help review the doc

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28431). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @stevhliu . Thanks for your advice, I have fixed these comments. Please take a look, thx!
Fix docstring checker issues with PIL enums
Author: Rocketknight1
Pull request: yes

The docstring checker is called by `make repo-consistency` or `make fix-copies`, but it struggles with enums, and particularly struggles with the `PIL.Resampling` enum, as this moved in PIL 10 and we set some code in `image_utils` to always put it in the same place. This caused issues where, depending on the installed PIL version, the docstring checker would try to replace enum names like `Resampling.BICUBIC` with the enum int value for that entry.

After this fix, people should be able to upgrade to the latest version of `PIL` and run `make fixup` or `make fix-copies` without issues! The issue may persist on older versions of PIL, unfortunately, where the value is sometimes just a raw `int` rather than an `Enum`, but we can just ask users to upgrade if they encounter issues there.

Comments:
[`Awq`] Add llava fused modules support
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds the Llava + fused modules support for blazing fast text generation using Llava + AWQ! 

This PR also fixes the issue: https://github.com/huggingface/transformers/pull/28032#issuecomment-1868262141 pointed out by a user since a custom past key value is passed to the model, indeed filtering out indexes that are inside the range of `extended_attention_mask` fixes the issue.

Added also a slow test

Can also confirm all Llava slow tests pass!

cc @casper-hansen

Comments:
Thank you soooo much, this PR and #28032 helped me work well now!

Thanks for your reviews @ArthurZucker ! Merging ! I'll address the points you shared in https://github.com/huggingface/transformers/pull/28239#discussion_r1448621388 in another PR as stated in my reply
[VideoMAE] TensorFlow implementation
Author: sayakpaul
Pull request: yes
Labels: name: WIP

Closes #18641 

## TODO

- [x] modeling_tf_videomae.py
- [x] integration tests
- [x] rest of the tests
- [x] documentation

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20351). All of your documentation changes will be reflected on that endpoint.

I am hitting an issue and thought of asking for help.

The issue stems from the port of `VideoMAESelfAttention` i.e., `TFVideoMAESelfAttention`. Upon analyzing deeper, I think it's because of the mismatch between `nn.functional.linear` and my custom `linear_transformation()`. This is my investigative [notebook](https://colab.research.google.com/gist/sayakpaul/d50d013f59674b943ef7e2b6ed9d2f91/scratchpad.ipynb) where I have tried debugging the issue but still no luck so far. The assertion errors seem to be within low tolerance but can easily sum up.

Cc: @amyeroberts @gante @Rocketknight1 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Added WIP label to prevent PR from being closed

No point in extending it further.
Mark two logger tests as flaky
Author: amyeroberts
Pull request: yes

# What does this PR do?

Two tests which capture and check the logger's output occasionally fail. 

```
FAILED tests/test_modeling_utils.py::ModelUtilsTest::test_model_from_pretrained_with_different_pretrained_model_name - AssertionError: False is not true
FAILED tests/test_modeling_utils.py::ModelUtilsTest::test_unexpected_keys_warnings - AssertionError: "were not used when initializing ModelWithHead: ['added_key']" not found in ''
```

It looks like the logger isn't capturing the output. I have never been able to replicate the errors outside of circle CI, locally or on a VM. 

The reason for the failure is unclear: there are other tests in the same module which utilise `CaptureLogger`. However, it's always these two tests which fail. 

Example runs, where failing tests were unrelated to the PR:
* https://app.circleci.com/pipelines/github/huggingface/transformers/81004/workflows/4919e5c9-0ea2-457b-ad4f-65371f79e277/jobs/1038999
* https://app.circleci.com/pipelines/github/huggingface/transformers/82051/workflows/8674dab8-35ac-4336-8db2-24d90426554f/jobs/1054942

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
@ydshieh Sure, will add! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28458). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update metadata loading for oneformer
Author: amyeroberts
Pull request: yes

# What does this PR do?

Previously, the loading of the metadata file for oneformer was effectively hardcoded to download a file from the hub. This PR updates the `prepare_metadata` method to allow for loading of local files as well as model repos. 

```py
from transformers import OneformerImageProcessor

image_processor = OneformerImageProcessor(
    checkpoint, 
    repo_path="path/to/file", 
    class_info_file="local_metadata.json"
)
```

Fixes #23116 and partially addresses #27572 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
[`Mixtral` / `Awq`] Add mixtral fused modules for Awq 
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds Mixtral + AWQ fused modules for blazing fast text generation! 

```python
from transformers import MixtralForCausalLM, AwqConfig, AutoTokenizer

model_path = "casperhansen/mixtral-instruct-awq"

quantization_config = AwqConfig(
    do_fuse=True,
    fuse_max_seq_len=1024,
)

model = MixtralForCausalLM.from_pretrained(model_path, quantization_config=quantization_config, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_path)

tokenizer.pad_token = tokenizer.eos_token 

inputs = ["Here are the top 10 useful Hindi phrases for your upcoming trip to India:\n1. ", "Hello my name is"]

inputs = tokenizer(inputs, return_tensors="pt", padding=True).to(0)
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
```
I introduced the same changes in modeling_utils as https://github.com/huggingface/transformers/pull/28239 for a tiny issue with respect to `modules_to_not_convert` not being handled correctly for fused module. 
Users needs autoawq>=0.1.8 to use this feature

cc @casper-hansen

Comments:
Thanks for your review @amyeroberts ! I left few comments and open questions, let me know wdyt! 🙏 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28240). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @amyeroberts for all your reviews! I just added the more general test with a tiny model ! I will merge the PR and address potential comments in a follow up PR ! 🙏