noamaanMulla-03/transformers-issues huggingface dataset:
fix: Fixed `type annotation` for compatability with python 3.8
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Using `list[...]` instead of `List[...]` without the `from __future__ import annotations` import will cause runtime errors on Python versions prior to **3.9**. Since, we are supporting python versions from `3.8`, this needs to be fixed.

I used the following [`ruff`](https://docs.astral.sh/ruff/rules/future-required-type-annotation/) check to catch these.

For reference: https://peps.python.org/pep-0585/


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @amyeroberts 

Comments:
@Sai-Suraj-27 Running `make fixup` should resolve the quality checks

> @Sai-Suraj-27 Running `make fixup` should resolve the quality checks

Thanks for the review @amyeroberts. I have updated the PR and all the checks are passing now.
fix: Replace deprecated `assertEquals` with `assertEqual`
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Replaces deprecated `assertEquals` with `assertEqual`.

References: 
https://docs.python.org/3/library/unittest.html#assert-methods
https://stackoverflow.com/questions/930995/assertequals-vs-assertequal-in-python

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts



Comments:
fix: Replaced deprecated `typing.Text`  with `str`
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
`typing.Text` is an alias for `str`, and only exists for Python 2 compatibility. As of Python 3.11, typing.Text is deprecated. 
Use `str` instead.

For Reference: 
https://docs.python.org/3/library/typing.html#typing.Text
https://docs.astral.sh/ruff/rules/typing-text-str-alias/

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts


Comments:
fix: Replaced deprecated `logger.warn` with `logger.warning`
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
`logging.warn` and `logging.Logger.warn` are deprecated in favor of `logging.warning` and `logging.Logger.warning`, which are functionally equivalent. (https://docs.python.org/3/library/logging.html#logging.warning)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments:
fix: Fixed `ruff` configuration to avoid deprecated configuration warning
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Currently, if we run `ruff`, we are getting the following warning

![image](https://github.com/huggingface/transformers/assets/87087741/a08486e1-8d8e-4c33-a813-d5fb0c74369d)

This is because, the configuration that is currently used for ruff in `pyproject.toml` is deprecated. So, I have updated the configuration according to the updated configuration rules of [`ruff`](https://docs.astral.sh/ruff/configuration/). The warning is no longer coming and all the `ruff` checks are also successfully passing.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [ ] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [ ] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/c/discussion-related-to-httpsgithubcomhuggingfacediffusers/63)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/diffusers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
From my observations through git blame of `pyproject.toml`
@ArthurZucker 

Comments:
fix: Fixed the documentation for `logging_first_step` by removing "evaluate"
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Document says that [logging_first_step](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.logging_first_step) will evaluate on the first global_step. But it only logs on the first step, not evaluate.

Fixes #27902


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Hey, @ArthurZucker. Can you look into this?
Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update masked_language_modeling.md
Author: jaguaryang
Pull request: yes

The definition of Trainer must be added with a tokenizer parameter.

Fixes # (issue)
Models without a passed tokenizer will not be usable for evaluation.

Comments:
cc @muellerzr 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31185). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> These tutorials cover inference outside of the `Trainer` using the pipeline later on, so I'm not sure if this is truly needed here or not, since it will work just fine without it. If we want to add this in, all of the docs would need to be updated.

There is no need to update all docs, the most tutorials working fine. Without the tokenizer parameter the model can be trained, but will not be able to loaded for evaluation.  Trained multiple time to validate this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MLFlowCallback MLFLOW_RUN_ID not used
Author: dibanez-emergence
Pull request: no
Labels: name: trainer

name: Integrations

### System Info

In the [MLFlow integration documentation](https://huggingface.co/docs/transformers/en/main_classes/callback#transformers.integrations.MLflowCallback) we can read that "When MLFLOW_RUN_ID environment variable is set, start_run attempts to resume a run with the specified run ID"

However, when I set that env variable to resume training, the result is a new run instead of reattaching to the provided one.

In `MLflowCallback` class the member `_run_id` is not used to resume the experiment from it but merely to start the run:

```Python
            if self._ml_flow.active_run() is None or self._nested_run or self._run_id:
                # Some more code here ...
                self._ml_flow.start_run(run_name=args.run_name, nested=self._nested_run)
```


### Who can help?

@muellerzr @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Set MLFlow env variable `MLFLOW_TRACKING_URI` with the URI of the remote server
2. Assign value "1" to `HF_MLFLOW_LOG_ARTIFACTS` env variable so we persist checkpoints in MLFlow
1. Launch a trainer using MLFlow by including `report_to=mlflow` in the training parameters
2. Wait till MLFlow persists a checkpoint
3. Stop training before it is finished
3. Get the run ID from the experiment run we just created
4. Set `MLFLOW_RUN_ID` env variable using the existing run ID
5. Launch a new trainer using MLFlow by including `report_to=mlflow` in the training parameters


### Expected behavior

Training should resume from the provided MLFlow run ID.
Instead, this is creating a new run.

Comments:
MLFLOW_RUN_ID is used inside MLFlow

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @daniel-ibanez-merlyn , sorry for the delay. Would you like to try your hand at this fix? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add CogVLM (cleaner)
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds CogVLM, in a cleaner way. Follow-up of #27718.

Debugging logits (for branch with matching logits see `add_cogvlm_cleaner_with_matching_logits`):

- [x] image features from CLIP backbone match in case xops.memory_efficient_attention is used. So this needs to be replaced by native PyTorch => 
- [x] debug LLM forward -> had to do with RMSNorm being copied from llama, which uses `self.weight * hidden_states.to(input_dtype)` instead of `(self.weight * hidden_states).to(input_dtype)`.
- [x] support `EagerAttention` besides `SdpaAttention` (only latter gives matching logits)
- [x] support key/value caching
- [x] Make sure logits match with 1e-4.

Comments:
I have a branch [add_cogvlm_cleaner_address_comments](https://github.com/NielsRogge/transformers/tree/add_cogvlm_cleaner_address_comments) which I can merge once I can verify the conversion

Thanks for your review, so 3 comments are about adding a `text_config` to `CogvlmConfig` and leverage `AutoModelForCausalLM` in `modeling_cogvlm.py`, however this is not really possible for this model (I've considered it). Unlike other models like BLIP-2 and Llava, the authors modify the text model to include "vision-expert attention". One would need to work with hooks etc. to make it work directly with `AutoModelForCausalLM`, to inject the vision expert attention layers inside the language model. Rather, the authors [defined](https://huggingface.co/THUDM/cogvlm-chat-hf/blob/main/modeling_cogvlm.py#L275) a new `CogvlmDecoderLayer`, which includes both text- and vision attention. I'm not sure the model is plug-and-play. Refer to [this figure](https://github.com/THUDM/CogVLM/raw/main/assets/method-min.png) for details.




The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28196). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @amyeroberts would be great if you could review this PR, quite some updates were made, making it ready for review.

Let's have a first round of reviews from the people already familiar with the model and PR :) cc @zucchini-nlp 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
remove duplicate words in msg
Author: yukionfire
Pull request: yes

# What does this PR do?
remove duplicate words in messages to improve code quality.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Add support for XTR
Author: mjeensung
Pull request: no
Labels: name: New model

### Model description

XTR (ConteXtualized Token Retriever) is a multi-vector retrieval model that improves efficiency by focusing on retrieving and ranking the most important document tokens. Details are described at https://arxiv.org/abs/2304.01982.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Implementation for PyTorch: https://github.com/mjeensung/xtr-pytorch
Weight: https://huggingface.co/google/xtr-base-en
Author: @jhyuklee

Comments:
cc @ArthurZucker 

I am a beginner, I will try my best to solve this issue, If possible, assign me to it. I will try my very best to read the research paper mentioned, and add model description for XTR

Hi @Souradip121, great that you want to contribute! We don't assign issues to people, as we find that this can lead to issues becoming stale, even if they're highly requested. Instead, we will prioritise based on PRs being open. Adding a model can be a big task - great if you want to tackle straight away, but if you'd like to start on something smaller, issues tagged with "[Good First Issue](https://github.com/huggingface/transformers/issues?q=is%3Aopen+is%3Aissue+label%3A%22Good+First+Issue%22)" are a great place to start 

@amyeroberts Okay I will focus on Good First Issues first. Thanks for the suggestion!!!
4bit Adam
Author: NicolasMejiaPetit
Pull request: no
Labels: name: Feature request

name: optimization

### Feature request

Is there any chance we coukd get this 4bit adam optimizer added to tranformers?
It has nearly the same performance as 32bit adam with significant drop in vram overhead. 
[repo](https://github.com/thu-ml/low-bit-optimizers)
[Paper](https://arxiv.org/abs/2309.01507)

With this added qlora would be even more memory efficient, and theoretically, you should be capable of FFT a 7b on a 24gb card.

### Motivation

The github repo has a paper which shows negligible difference between 32bit and 4bit adam, and they have the code for the adam optimizer here: [4bit code](https://github.com/thu-ml/low-bit-optimizers/blob/main/lpmm/optim/adamw.py)

Or one bit adamw from deep speed, I only didn't recommend it since digging through the deep speed code, it isn't as laid out as this one is with a whole dedicated script. While yeah you can always use deepspeed and transformers, but deepspeed comes with its own set of draw backs, such as windows compatibility, and unsloth compatibility. But either or a one bit adamw would be awesome. Aside from that there aren't too many other ways to save memory for qlora. I mean theoretically if someone had the will, they could make a bitnet adamnw, that runs on the cpu. Since bit net doesn't need matmuls, the entire computation could be offloaded to the cpu. It would be actually fast, so the training wont be bogged down.

### Your contribution

Submit feature request

Comments:
cc @younesbelkada 

#31865 👀
Fix file type checks in data splits for contrastive training example script
Author: npyoung
Pull request: yes

# What does this PR do?
After checking if the validation file is a json or csv, existing code checks again to make sure it is only a json. This PR checks if the test file is a json or csv, as with validation and training, and does not force the validation file to be json.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
@npyoung Could you rebase on main? This will include upstream changes which should resolve the failing errors on the CI
fix: Removed `duplicate` field definitions in some classes
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
In some classes, some of the fields/attributes are defined multiple times. I think the duplicates can be removed:
https://github.com/huggingface/transformers/blob/a0a3e2f4692e7eef31ab6303def799d8bdf8906e/examples/research_projects/jax-projects/hybrid_clip/run_hybrid_clip.py#L163-L168

https://github.com/huggingface/transformers/blob/a0a3e2f4692e7eef31ab6303def799d8bdf8906e/src/transformers/models/deformable_detr/modeling_deformable_detr.py#L1081-L1083

https://github.com/huggingface/transformers/blob/a0a3e2f4692e7eef31ab6303def799d8bdf8906e/examples/research_projects/jax-projects/model_parallel/run_clm_mp.py#L140-L142
https://github.com/huggingface/transformers/blob/a0a3e2f4692e7eef31ab6303def799d8bdf8906e/examples/research_projects/jax-projects/model_parallel/run_clm_mp.py#L159-L161


Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments: