noamaanMulla-03/transformers-issues huggingface dataset:
CLIP conversion script - Change fairseq to OpenAI
Author: gau-nernst
Pull request: yes

# What does this PR do?

Change fairseq to OpenAI in the CLIP conversion script

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

@ArthurZucker

Comments:
Fix `model_accepts_loss_kwargs` for timm model
Author: qubvel
Pull request: yes

# What does this PR do?

`TimmWrapper` model training is broken due to the `num_items_in_batch` param passed to the model forward. `TimmWrapper` model has `**kwargs` in the `transformers` model forward, but it goes straight to the `timm` model and not to the loss. This PR introduces a fix to avoid passing loss_kwargs even if `**kwargs` are in model forward.

Relevant to:
 - https://github.com/huggingface/transformers/pull/35121
 - https://github.com/huggingface/transformers/pull/34915

cc @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35257). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker Do you mean to move it to the `PretrainedModel` level?

```python

class PreTrainedModel:

    _accept_loss_kwargs = None
    
    @property
    def accepts_loss_kwargs(self):
        if self._accept_loss_kwargs is not None:
            return self._accept_loss_kwargs
        # otherwise check for signature here
        ...
        return accept_loss_kwargs
```

And then in modeling:

```python
class MyModel(PreTrainedModel):
    _accept_loss_kwargs = False
```

No for now let's merge as is! Not sure we'll need the abstraction for more models!
Why there is no LlavaForSequenceClassification ?
Author: Zuhashaik
Pull request: no
Labels: name: Feature request

### Feature request

Need of LlavaForSequenceClassification class in '`src/transformers/models/llava/modeling_llava.py`'..

### Motivation

If there is a LlavaForSequenceClassification class we can use decoder only models for classification tasks. 

### Your contribution

Allow me to contribute to this as I am already working on this model for mental health meme classification for the upcoming AAAI conference.

Comments:
Hi @Zuhashaik, thanks for opening this issue! 

We only add specific task heads for models if there are official checkpoints available; the task is described in the original paper or there's a large community demand. 

Note that you could technically also use `LlavaForConditionalGeneration` by training the model to only spit out one classification token per image.

> Note that you could technically also use `LlavaForConditionalGeneration` by training the model to only spit out one classification token per image.

Yeah I've attached an score layer at the top of the model by passing those last layer hidden states. While doing this, some cuda related issues ("data in different devices cuda:0 and cuda:1") came up.

So I've changed my `LlavaForConditionalGeneration` class itself in modelling_llava.py file (added score layer) and used that for classification.

Thankyou.

I am facing lots of data type and quantization errors while defining classifier (score) separately please initiate `LlavaForSequenceClassification` and SequenceClassification for vision models..

> Hi @Zuhashaik, thanks for opening this issue!
> 
> We only add specific task heads for models if there are official checkpoints available; the task is described in the original paper or there's a large community demand.

Thank you!

I'm also looking forward to `LLavaForSequenceClassification`

Facing lots of issues without this class, @amyeroberts please do the needful thank-you 

+1




> +1

@NielsRogge Please do the needful. It will be very much handy dealing with sequence cls class rather than using conditional generation class.

Thank you


@NielsRogge Please add the class
Modernbert Release Fixes
Author: warner-benjamin
Pull request: yes

This PR resolves two issues with ModernBERT so it will be working for release in just under an hour.

First, ModernBERT has a final head layer `ModernBertPredictionHead` with a pretrained Linear layer and LayerNorm . In downstream heads, these two layers are loaded to `ModernBertPoolingHead`. I readded `ModernBertPoolingHead` to `ModernBertForTokenClassification` so TokenClassification models will load the pretrained layer weights.

Second, #35235 appears to have broken the SDPA/Eager paths by changing how the RoPE layer ModernBERT inherited from works. In the interest of expediency, I unmoduarized it so we can have a working ModernBERT implementation on all paths.

cc @ArthurZucker @tomaarsen 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35344). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi, thanks, is there a plan to support multiplechoice ?

Have you already road-tested the token classification implementation? E.g. using the ModernBERT Large model only yields to ~91% F-Score on CoNLL-2003 (Test set, I tried some batch size and learning rate configurations), in two different frameworks (Transformers and Flair). I am not sure if ModernBERT needs really special hyper-parameters or if there are fixes needed in the architecture :thinking: 

I tested it with b5a557e5fe2d015bd36214a95878370eaed51571.

Hi @stefan-it I've tried to test `ModernBertForTokenClassification` with [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) and [UD_English-EWT](https://github.com/UniversalDependencies/UD_English-EWT) and got 97% F1-Score on UPOS (see [my report](https://qiita.com/KoichiYasuoka/items/d1fbb04804079a4e495e) written in Japanese). During my test I found that `ModernBertForTokenClassification` lacked `get_input_embeddings()` and made PR #35422. And I seem the tokenizer of ModernBERT-base is slightly difficult to handle POS-tagging (than BERT's one) and `"classifier_dropout": 0.0` seems too small.
Add Flax activation functions and corresponding tests
Author: m-momeni
Pull request: yes
Labels: name: Flax

The PR implements Flax version of all activation functions introduced previously in torch to be selected in configs and models.
Also a test is written to compare torch version and Flax version outputs and asses if they are close enough.

Comments:
CircleCI shows "tests_generate - Failed", but I think it's not related to my commits.

Hi @sanchit-gandhi, I'd be happy if you check the commit and if it's fine I can go forward to implement torch models in Flax.
query_length not found in BLIP-2 config
Author: thisisiron
Pull request: no

In the code, there's a reference to `self.config.query_length` [[link]](https://github.com/huggingface/transformers/blob/30335093276212ce74938bdfd85bfd5df31a668a/src/transformers/models/blip_2/modeling_blip_2.py#L1362):

```python
past_key_values_length = (
    past_key_values[0][0].shape[2] - self.config.query_length if past_key_values is not None else 0
)
```

However, query_length doesn't exist in the Blip2Config. Instead, it seems that we should be using `num_query_tokens` which is already defined in Blip2Config. [[link]](https://github.com/huggingface/transformers/blob/30335093276212ce74938bdfd85bfd5df31a668a/src/transformers/models/blip_2/configuration_blip_2.py#L281)

Comments:
@thisisiron hey, indeed seems to be a bug as there is no such attribute in the config. From what I see it is triggered only when Qformer is called with `past_key_values` which is never the case when using Blip-2 model. Unless one wants to tweak around and call the Qformer directly. That should be the reason we never caught this bug

If you have any use-case when the Qformer is called with cache, feel free to share a small code snippet and open a PR. Or you can just share the code here, and I will help you fix it :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to specific customized force_token_ids in whisper
Author: MonolithFoundation
Pull request: no
Labels: name: Generation

name: Audio

```
ValueError: A custom logits processor of type <class 'transformers.generation.logits_process.ForceTokensLogitsProcessor'> with values <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f4230cfac50> has been passed to `.generate()`, but it has already been created with the values <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f422829c510>. <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f422829c510> has been created by passing the corresponding arguments to generate or by the model's config default values. If you just want to change the default values of logits processor consider passing them as arguments to `.generate()` instead of using a custom logits processor
```

this way don't work:

```
inputs = inputs.to(self.model.dtype)
        with torch.no_grad():
            if forced_decoder_ids is not None:
                generated_ids = self.model.generate(
                    inputs, forced_decoder_ids=forced_decoder_ids
                )
            else:
                generated_ids = self.model.generate(inputs)
```

Comments:
cc @gante/@ylacombe maybe

cc @eustlb , since you're working on Whisper fixes :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add LTG-BERT model
Author: davda54
Pull request: yes

# LTG-BERT

This pull request adds the custom LTG-BERT model into the repository. This optimized LM architecture was introduced [in this paper](https://arxiv.org/abs/2303.09859) and is currently also used by a new generation of Norwegian LMs. The architecture features multiple improvements to the standard transformer module and we unfortunately cannot use any existing HF model wrappers. 

@ArthurZucker and @younesbelkada

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23037). All of your documentation changes will be reflected on that endpoint.

Hey! Great that you want to share this model 🔥 Would you be open to put ot on the hub following [this tutorial](https://huggingface.co/docs/transformers/custom_models)! Will be easier as there won't be any CI issues and since it is very similar to an existing model, this makes more sense!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey if you added the model on the hub could you share the links to it? This will help us keep track of the models that we support on the hub 

Hi @davda54 I'm very fascinated with this PR, and I think we should re-open. But `modeling_ltgbert.py` and `configuration_ltgbert.py` in this PR are slightly different from those used in https://huggingface.co/KoichiYasuoka/ltgbert-base-japanese-upos https://huggingface.co/HPLT/hplt_bert_base_en or https://huggingface.co/ltg/ltg-bert-bnc . How do we "standardize" them @ArthurZucker ?
latest transformer has a tensorflow and numpy error
Author: sleepingcat4
Pull request: no
Labels: name: bug

### System Info

collab version of transformers (latest)

### Who can help?

@gante @rock

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Install latest transformers, numpy and tensorflow versions (more specifically colab versions)

You'll encounter an error specifically more tensorflow for this line:

```Python
from transformers.pipelines import PIPELINE_REGISTRY
```

How the error looks like:

```Bash
SystemError: initialization of _pywrap_checkpoint_reader raised unreported exception
```

### Expected behavior

It should not produce an error and pass clean like I did on the colab by downgrading versions. 

https://colab.research.google.com/drive/1482Nc7IfE-s5sl7dBjEbWfOjz59QFA1l?usp=sharing

Comments:
Hi @sleepingcat4, can you confirm if downgrading just Tensorflow fixes the issue?

@Rocketknight1 downgrading transformers won't work for my library but I can try in an hour. 

Right now, I get this error by default as Tensorflow's latest version won't let me import even the basic classes of transformers. 

Yes, understood - I know downgrading isn't a fix, it'd just help us to debug it if you can figure out exactly which combinations of versions causes the bug!

@Rocketknight1 I am not getting this error now. I guess KVPress changed something on my colab while it was installing. But, from my understanding that tensorflow module should be fixed otherwise it will create more pain if some library uses a tensorflow version that is not compatible with transformers. 

If I am not wrong this error was introduced in the October update of Tensorflow. 

@sleepingcat4 - yes, if you or any other user can identify the combination of versions that triggers the error, please let us know and we'll investigate. I'll leave this issue open until then

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add HHEMv2 Model
Author: Miaoranmmm
Pull request: yes

This PR adds the support of codes for the HHEMv2 model. For information about HHEMv2 , please visit [model card](https://huggingface.co/vectara/hallucination_evaluation_model)
cc @forrestbao

Comments:
> Hey! from scanning through the code I am not sure I understand what is the difference with T5 here? 😓 If you want to add the model the best is to isolate the changes copied from t5 (wrap with. # Copied from) from the specificities of this model!

Hi @ArthurZucker, thanks for your feedback. We have removed the duplicate content `modeling_hhemv2.py`. Please review the updated version.

hi @ArthurZucker There are two differences here:
1. We hacked the T5ForTokenClassification for sequence classification. We padded a `<pad>` token at the beginning of the sequence in lieu of the `[CLS]` token and use the hidden state from the `<pad>` token to predict the final label. In this way, we only use the encoder part of T5. 
2. We also added the prompt template and tokenization into the class to save users the hassle. Now users can send pair of text and get the hallucination score -- while under the hood  the prompting and tokenization is done automatically for them. 

Hey!
1. Changing the token should not induce changes in the modeling code. However if you change the way the forward pass (Not the token or the id being used) then alright! 
2. This is not really something we want to have in the model itlesf. The chat template should be in the documentation and directly saved in the model that is pushed to the hub! 
If you wait a bit for #33248 to be merged will be a lot better to use this approach!
Reporting a vulnerability
Author: igibek
Pull request: no

Hello! 

I hope you are doing well! 

We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly.  GitHub has a feature called **Private vulnerability reporting**, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository. 

Can you enable it, so that we can report it? 

Thanks in advance! 

PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

Comments:
We are using another platform for vulnerability reporting. @Michellehbn can tell you more.

Hi @igibek ! Thanks for reaching out to us! 🤗 We have a bug bounty program with HackerOne and would love for you to submit security vulnerability reports to our private program at https://hackerone.com/hugging_face. Will it be possible to send us your H1 username or email address so that we can invite you to our program please, either here or to security@huggingface.co? Thanks again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hello @Michellehbn !

I found a crticial vulnerability in hugging face. Could you invite me to the private program please? I swear you will not regret it.
Update docs for `sdpa_kernel`
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Addresses #34411 (issue)

Replace `torch.backends.cuda.sdp_kernel` (deprecated) with `torch.nn.attention.sdpa_kernel`

## Who can review?

@stevhliu

Comments:
[i18n-ar] Translated file: `docs/source/ar/tasks/question_answering.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/tasks/question_answering.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

### Who can review?
@stevhliu , @MKhalusova , @abodacs, may you please review this PR?



Comments:
أشكرك أخي عبدالله @abodacs  على جهودك العظيمة
تحياتي وتقديري

Thank you @abodacs  and @stevhliu
"Is it possible for Hugging Face to implement a chat model for quick information retrieval similar to vLLM?"
Author: BeastyZ
Pull request: no
Labels: name: Feature request

### Feature request

![screenshot-20241231-155528](https://github.com/user-attachments/assets/889e19bf-7d74-4486-ad98-0210203df42f)


### Motivation

Quick and convenient information retrieval.

### Your contribution

I don't have relevant code for this feature request.

Comments: