noamaanMulla-03/transformers-issues huggingface dataset:
Mask2FormerImageProcessor - fails to process multichannel image
Author: maciej-adamiak
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.4
- Platform: Linux-6.5.0-1027-oem-x86_64-with-glibc2.35
- Python version: 3.11.9
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA RTX A4500 Laptop GPU


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

[Colab](https://colab.research.google.com/drive/1T_lGRfyRoQCycZbLJAVsa6c12K5NI7GD?usp=sharing)



### Expected behavior

The image processing should be handled correctly. 

It looks like the [get_max_height_width](https://github.com/huggingface/transformers/blob/174890280b340b89c5bfa092f6b4fb0e2dc2d7fc/src/transformers/models/mask2former/image_processing_mask2former.py#L938) is using a default parameter rather then a value `input_data_format` set in the class constructor.

Comments:
cc @qubvel or @zucchini-nlp in case you have some bandwidth!

Hi @maciej-adamiak , thanks for opening an issue and providing an example to reproduce!

Would you like to make a fix? We should make sure that `input_data_format` is passed to [get_max_height_width](https://github.com/huggingface/transformers/blob/174890280b340b89c5bfa092f6b4fb0e2dc2d7fc/src/transformers/models/mask2former/image_processing_mask2former.py#L938) and all other transforms correctly handle it.

I'm on it.
Support reading tiktoken tokenizer.model file
Author: itazap
Pull request: yes

Use existing TikTokenConverter to convert tiktoken tokenizer.model file.
depends on loading without `config.json` file https://github.com/huggingface/transformers/pull/32356

- [x] add case to convert_tiktoken_tokenizer
- [x] add internal model
- [x] add test

## Workflow changes

1. `tokenization_utils_base.py': when` loading a model, the slow tokenizer is loaded first. If the `tokenizer.model `file is not SPM, then an error of type `google.protobuf.message.DecodeError` is thrown, or a RunTime error on loading `ModelProto`. So, the first step is to catch these errors relating to SPM and set the `tokenizer=False` to indicate failure.
2. `tokenization_utils_fast.py`: check if `slow_tokenizer=False`, if so, try to convert from tiktoken. 
3. `convert_slow_tokenizer.py`: use `TikTokenConverter` to convert. 

* Note: the reason we catch errors is because there is no way to differentiate the tokenizer.model file as SPM or TikToken with the current standards for hub files. So, we always try to convert from SPM, if we fail, we try with TikToken.



@ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31656). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
how can calculate the predict score of every pixel use mask2former swin-l model?
Author: funny000
Pull request: no
Labels: name: Feature request

name: Vision

### Feature request

I has download the mask2former swin-l model from huggingface website,
and use example code get segmentation map of image,
the example code is:
```python
import requests
import torch
from PIL import Image
from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation


# load Mask2Former fine-tuned on COCO panoptic segmentation
processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-large-coco-panoptic")
model = Mask2FormerForUniversalSegmentation.from_pretrained("facebook/mask2former-swin-large-coco-panoptic")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# model predicts class_queries_logits of shape `(batch_size, num_queries)`
# and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
class_queries_logits = outputs.class_queries_logits
masks_queries_logits = outputs.masks_queries_logits

# you can pass them to processor for postprocessing
result = processor.post_process_panoptic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
# we refer to the demo notebooks for visualization (see "Resources" section in the Mask2Former docs)
predicted_panoptic_map = result["segmentation"]
```

the code can get the seg map, but not pred score of every pixel,
so how to add calculate code to get perd score of every pixel?



### Motivation

not motivation

### Your contribution

a littlecontribution

Comments:
Hi @funny000,

As far as I see the pred probability is not returned for each pixel, but you can have a look at postprocessing code, here is probably what you need

https://github.com/huggingface/transformers/blob/65bb28444849976f853063edb958b3ef3dd59d12/src/transformers/models/mask2former/image_processing_mask2former.py#L1207

thanks @qubvel,
I see the image_processing_mask2former of code, and get the mask prob as like as pred score, has change code to output mask probs to post process of my work.
but have a other problem, when i slider a big height width image to generate panotanic segmantation map,
every small image of big image generate different label_id in same object, just like a car or water.
it's so odd, Why model output different label_id of same object in every small image?

Hi @funny000, that is expected behavior, the model does not re-identify objects between images. On each image object is a unique instance with its own class label, nothing else. You might want to search and explore some techniques on how to re-identify / stitch objects from different images or frames.
Bug Fix: Update hub.py to fix NoneType error
Author: rishiraj
Pull request: yes

Bug:
TypeError: argument of type 'NoneType' is not iterable

Analysis:
The error `TypeError: argument of type 'NoneType' is not iterable` suggests that `model_card.data.tags` is `None`, and the code is trying to iterate through it using `not in`.

Fix:

1. **Check if `model_card.data.tags` is `None` before the loop**: Since you're checking the variable `tags` before the loop, you should also ensure that `model_card.data.tags` is not `None`. You can do this by initializing `model_card.data.tags` to an empty list if it's `None`.

2. **Updated code**: Add a check and initialize the `tags` if it is `None` before proceeding with the iteration.

This way, if `model_card.data.tags` is `None`, it gets converted to an empty list before checking the contents. This prevents the `TypeError`.

Comments:
Thanks for the PR!

Could you just run the code quality tool to ensure that the code quality passes? You can install them with the following, from the root of your clone:
```
pip install -e ".[quality]"
```
And then run them with:
```
make fixup
```

@LysandreJik done, it's passing now. Blank line had to be removed.

Awesome, thanks!
Make Cache statically configurable at model construction time
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

Be able to construct and load a model like:
```
model = AutoModelForCausalLM.from_pretrained(
    hf_model_repo,
    attn_implementation="sdpa",
    generation_config=GenerationConfig(
        use_cache=True,
        cache_implementation=cache_implementation,
        max_length=max_cache_len,
        cache_config={
            "batch_size": batch_size,
            "max_cache_len": max_cache_len,
        },
    ),
)
```
See additional context in #32253

### Motivation

This feature request is to support `torch.export()`, and ensure the model is exportable in a way that can be further lowered and run in ExecuTorch with performance out-of-the-box.

### Your contribution

TBD

Comments:
PR is published: https://github.com/huggingface/transformers/pull/32830
[Docs] How to build offline HTML or Docset files for other documentation viewers?
Author: ueoo
Pull request: no
Labels: name: Documentation

name: Feature request

### Feature request

How can I build the docs into HTML files for use with other documentation viewers like [Dash](https://www.kapeli.com/dash) , [Dash-User-Contributions](https://github.com/Kapeli/Dash-User-Contributions)?

I successfully built the PyTorch docs for Dash by working directly in their `docs/` directory. I’m wondering if a similar process exists for Hugging Face libraries.

### Motivation

The Dash docset viewer is very useful for viewing multiple documentation sets in one place, even offline. It would be great to support it and include all Hugging Face libraries.

### Your contribution

I’ve built the PyTorch docs for Dash, so I’m familiar with incorporating and generating docsets.

Comments:
Hey @hologerry, a guide is available here to build the docs: https://github.com/huggingface/transformers/tree/main/docs

@LysandreJik Thanks.
use diff internal model in tests
Author: itazap
Pull request: yes

Update to use internal model - @require_read_token only works for internal

other changes:
- need to pass `bos_token` and `eos_token` because renaming the internal repo to "llama-3" from "llama3" resulted in the AutoTokenizer instantiating with LlamaTokenizerFast as opposed to PreTrainedTokenizerFast. This is good but needs to have `bos_token` and `eos_token` updated to the `Llama-3` values instead of the default `
` and `
 ` cc @ydshieh 
(related to tiktoken support PR #31656)

@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33387). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi! Could you briefly explain what are the other changes for (except that of changing the repo id.)?

(for example, those are necessary changes for #31316, or it's for something else?)
A Trainer subclass for Decoder-Only LM with generation in evaluate()
Author: skpig
Pull request: no
Labels: name: trainer

name: Feature request

### Feature request

The main feature request involves a New Trainer Subclass, similar to Seq2SeqTrainer, but suitable for Decoder-Only LM.

### Motivation

`Seq2SeqTrainer` provides a great abstraction for Encoder-Decoder LM, when we need to conduct generation during `evaluate()`

But the current implementation of both `Trainer` and `Seq2SeqTrainer` seems to be not suitable for Decoder-Only LM due to the difference of `input_ids` and `labels` between teacher-forcing training and generation-involved evaluation.

For example in instruction tuning:
- During training (teacher-forcing)
```python
input_ids = 'Translation the following texts: {Text in Chinese...} {Text in English...}'
labels = 'Translation the following texts: {Text in Chinese...} {Text in English...}'
```
- During evaluation
```python
input_ids = 'Translation the following texts: {Text in Chinese...}'
labels = '{Text in English...}'
```

So we need to prepare two kinds of inputs_ids during evaluation for calculation of both `loss` and `bleu_metrics`. It leads to different columns in eval_dataset. However, `Trainer._remove_unused_columns()` will remove columns for both `eval_dataset` and `train_dataset` not accepted by `model.forward()`. During training, this behaviour is expected (we only need the teacher-forcing inputs). But it will make evaluation difficult.

This feature is nearly identical across all CausalLM models when performing generation during evaluation, making it highly reusable. Given the increasing number of Decoder-only LMs (CausalLMs) in the community, I strongly recommend implementing a dedicated CausalTrainer to simplify deployments.

I may have missed something. If there is already a simpler way to customize such a Trainer, please let me know.

### Your contribution

I'm willing to help submit a PR. But I'm not familiar with some integrations such as fsdp and deepspeed. I may need someone to help me finish this feature.

Comments:
cc @muellerzr regarding the `Trainer` question, but maybe @Rocketknight1 can shed light on the NLP/training aspect

Guess the feature requested is same as in https://github.com/huggingface/transformers/pull/32346

Looks that way to me - we can add a note to #32346 that it fixes this issue! @skpig is there anything missing in #32346 that you need for your use-cases?

Fantastic, that's exactly what I need! Look forward to the new feature! 😊
[build-ci-image] add tiktoken
Author: itazap
Pull request: yes

[build-ci-image] add tiktoken

update: CI works but local image fails, will be fixed after https://github.com/huggingface/transformers/pull/32537 (unrelated issue)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32152). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

dev consistency docker image is failing (unrelated to PR, reverted all changes)
Fix `FbgemmFp8Linear` not preserving tensor shape
Author: vgel
Pull request: yes

# What does this PR do?

Fixes #32868 : `FbgemmFp8Linear` (and models using it, like llama3-405b-fp8) would incorrectly squash higher-dim tensors:

```python
>>> emd = base_model.model.embed_tokens
>>> emd.shape
torch.Size([32, 23, 16384])
>>> type(base_model.model.model.layers[0].mlp.up_proj)
torch.nn.modules.linear.Linear
>>> base_model.model.model.layers[0].mlp(emd).shape
torch.Size([32, 23, 16384])
>>> type(base_model.model.model.layers[1].mlp.up_proj)
transformers.integrations.fbgemm_fp8.FbgemmFp8Linear
>>> base_model.model.model.layers[1].mlp(emd).shape
torch.Size([736, 16384]) # <-------------------------------- wrong!!
```

This fixes that (and adds tests). See the linked issue for more details.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@SunMarc 

Comments:
@SunMarc Pinging for addressed comment, let me know if there's anything else!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33239). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Usage of deprecated task will break CI after datasets-3.0
Author: albertvillanova
Pull request: no

Usage of deprecated `task` in `load_dataset`will break CI after datasets-3.0: https://app.circleci.com/pipelines/github/huggingface/transformers/103944/workflows/09136e5e-f232-4132-ba3b-c7b0d5723493/jobs/1382170?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary
```
FAILED examples/tensorflow/test_tensorflow_examples.py::ExamplesTests::test_run_image_classification - TypeError: BuilderConfig.__init__() got an unexpected keyword argument 'task'
```

Comments:
Fix: Cast prefetch_bucket_size to integer for deepspeed >= 0.15
Author: kiddj
Pull request: yes

# What does this PR do?

## Fix compatibility with deepspeed versions >= 0.15

Deepspeed >= 0.15 integration fails due to a validation error where `stage3_prefetch_bucket_size` expects an integer but receives a float.

Ensure `prefetch_bucket_size` is always passed as an integer to avoid errors when using deepspeed. This change resolves compatibility issues with deepspeed versions 0.15 and above.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr

Comments:
Using P-tuning On Whisper‘s decoder
Author: dengchengxifrank
Pull request: no
Labels: name: Ex: LM (Finetuning)

name: bug

name: Audio

### System Info

- `transformers` version: 4.44.2
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.0
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.2
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A40

### Who can help?

@sanchit-gandhi @ylacombe . Hi, I want to perform p-tuning on whisper's decoder, but I found that in the process of trainer.train(), I can get the embedding through nn.Parameters and concatenate these embeddings to the output embedding, but when evaluating, the model will call the forward function, and in the first forward step, the model stipulates the prefix token of the input as [startoftranscirpt,en,transcribe,notimestamps], and generates a character in each subsequent step. I wonder how to use the trained embedding for the decoding process during evaluation in this case.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Using nn.Paramters()

### Expected behavior

Using the trained parameters for decode

Comments:
Hi @dengchengxifrank, thanks for opening this issue! 
I'm not sure to follow what's the issue and what you're trying to accomplish here!
Do you think you could:
1. explain a bit more what's going on
2. share a small code snippet to reproduce your issue
3. share your logs if there's an issue
?

Many thanks

Hi @ylacombe ， My question can be summarized as follows: I have a tensor of shape [4, 1024]. I want to concatenate this tensor to the front of the initial token as a condition when whisper uses model.generate(). I want to know where I should add the codes, because there is only init_tokens in the model.generate function(), and no embedding. I'm sorry that I don't have any codes to provide.
Sentence-transformer: No such file or directory error
Author: Rolv-Arild
Pull request: no

### System Info

Using the sentence-transformer widget leads to the following error at https://huggingface.co/NbAiLab/nb-sbert:
`[Errno 2] No such file or directory: '/data/NbAiLab_nb-sbert/1_Pooling/config.json'`
I have checked all the config files, and can not find any references to this. 
I am able to load the model locally (from the HF repo), and do valid calculations. The only thing that does not work is the widget.
I found this on the discussion forum: https://discuss.huggingface.co/t/sentence-similarity-demo-not-working/8711
Any idea about what is happening here?

### Who can help?

@Narsil @LysandreJik

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Open https://huggingface.co/NbAiLab/nb-sbert
2. In the widget, use Example 1 or fill in sentences
3. Press "compute"

### Expected behavior

Widget providing sentence similarities

Comments:
Fixed now.

The culprit is either the way the api works or sentence-transformers, but `sentence-transformers` is not great at dealing with missing files (it looks only for 1 file and assumes the rest is there, leading to the failure you are seeing).

Since the API uses massive storage to handle all the models on the hub, there is periodical cleaning of unused files, and sometimes it happens that sentence-transformers doesn't access all the files at the same time, leading to only partial cleanup of sentence-transformers model files and to this error.

Hope you understand a bit better what happened.
For the fix we'll see about maybe adding something upstream, but can't make any promises.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I was also facing the same issue, so I changed the directory from "_Pooling" to "1_Pooling". That resolved the issue.

My folder structure does not have the 1_Pooling folder, but it is there in the cache folder path in the code, so i get the same error. The funny thing is that it suddenly started showing me this error! I had no problems with it so far.

I am having the same problem, did you find a solution to this?

Hi, is there any solution for this problem? I am facing the same issue

I solved this issue by removing get_cache_dir() from the HuggingFaceEmbedding package in the following line: 
cache_folder = cache_folder or get_cache_dir(). 
It seemed to use the wrong cache location, and when I checked, the specified file location didn’t exist