noamaanMulla-03/transformers-issues huggingface dataset:
evaluation_strategy and eval_steps does not work
Author: nkjulia
Pull request: no

### System Info

- `transformers` version: 4.27.1
- Platform: Linux-4.15.0-200-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): 2.4.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

train LM use run_clm.py
paramters like below:
--model_name_or_path gpt2 \
--output_dir ${out_model_dir} \
--train_file ${train_file} \
--validation_file ${validation_file} \
--validation_split_percentage 5  \
--block_size 512 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--do_train --do_eval --line_by_line \
--evaluation_strategy steps \
--eval_steps 100

but no eval result printed when training.

### Expected behavior

print eval result every eval_steps

Comments:
Please provide a reproducer we can execute. We do not have your `train_file` and `validation_file`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I have the same problem after I update transformers to 4.33.2 and resuming interrupted training.  No matter how I change it, eval_steps and save_steps is always 500.


same comment, sorry but if we don't have a reproducer we cannot help you 😢
Feature Request: El-Attention
Author: suriyakode
Pull request: no
Labels: name: New model

# 🚀 Feature request
I've looked into the paper titled "[EL-Attention: Memory Efficient Lossless Attention for Generation](https://arxiv.org/abs/2105.04779)".
It proposes a method for calculating attention that forgoes creating multi-head attention from the hidden state. This saves computational time and frees memory. 

## Motivation
El-attention seems to have no downsides, and promises significant memory and performance gains during training and inference. 

## Your contribution
The main difficulty may be in that it requires being added directly in to each model's attention mechanism code, or requires a ton of new subclasses for each part of each model. Maybe an easier solution to this would be a pipeline to use custom  attention mechanism code. 

Comments:
Any updates on this one?
🌐 [i18n-KO] Translated `whisper.md` to Korean
Author: nuatmochoi
Pull request: yes

# What does this PR do?

Translated the `whisper.md.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

 May you please review this PR? 
@bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

May you please review this PR? @sgugger, @ArthurZucker, @eunseojo 

Comments:
위의 리뷰 내용을 제외하고는, glossary와 문장 요소 등 다 잘 번역되어 있는 것 같습니다! LGTM!!
좋은 번역 감사합니다!

Did you want to ping someone else for Korean review or should I merge this? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26002). All of your documentation changes will be reflected on that endpoint.

@ArthurZucker 
> Did you want to ping someone else for Korean review or should I merge this?

The review is complete and I just hope to merge this PR.


Thanks for the contribution!
[AutoBackbone] Add test
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds a test which verifies that out_indices and out_features get saved properly.

The test fails if a Backbone class first inherits from `PretrainedConfig`, then from `BackboneConfigMixin`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts feel free to take over, I see there's an issue regarding `TimmBackboneTester`

(She will be of for a while now, do you need this to be taken over?) 

I've fixed it, feel free to merge
Create the return value on device to avoid unnecessary copying from CPU
Author: mksit
Pull request: yes

# What does this PR do?

`router_tuple = (torch.tensor([0], device=hidden_states.device),)` introduces an unnecessary data copy from the CPU. I have changed it to create the return tensor on the device to avoid potential performance issues.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
I am sorry that I did not explain clearly. The tensor is created on `device=hidden_states.device`, but the current creation method has caused non-negligible overheads due to the data copying in my program, which can be seen in the following trace.

![Screenshot 2023-09-14 at 12 30 33](https://github.com/huggingface/transformers/assets/25057334/9f8d4adb-5e87-4d73-ae91-22f44ed96e71)

This overhead seems unnecessary, so I have suggested this change in this commit.




@mksit Are you sure the `aten::to` op is not there anymore when replacing `torch.Tensor` by `torch.zeros`? I generally don't trust pytorch profiler for the timing of `aten::to`. @NouamaneTazi may have more insights

@fxmarty The `aten::to` operation disappeared after the replacement in my case. What do you suggest for the profiling of `aten::to`?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26151). All of your documentation changes will be reflected on that endpoint.
[docs] Fix model reference in zero shot image classification example
Author: Aleksandar1932
Pull request: yes

# What does this PR do?

This PR fixes a typo in the docs, more specifically the model reference, since it it defined as `detector` and later referenced as predictor which is not defined. Settled on `detector`, since on example in other locale (`docs/source/ko/tasks/zero_shot_object_detection.md`) is `detector`.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26206). All of your documentation changes will be reflected on that endpoint.
Fix the gitlab user mention in issue templates to the correct user
Author: muellerz
Pull request: yes

# What does this PR do?
This fixes a wrong suggested tag of a Github user not acquainted with this project to the correct contributor. 

#### Personal Note
I can definitely see that his and my username seem switched given our clear names and I'm sorry for any confusion. That username was given to me by a teacher in school who didn't want to spell my full last name and it stuck.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu and @MKhalusova 

Comments:
It's all good :D. I got mentioned occasionally before, which I ignored, but checked that he was later tagged correctly. I don't really use Github so I just deleted the mails without thought, so no real mess there. It did get slightly annoying and I just had the spare time to fix it. And I'm sure having the intended user mentioned is in your interest, too.

All the best for the future 👍 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26237). All of your documentation changes will be reflected on that endpoint.
batching in pipeline gives single output
Author: znb899
Pull request: no

### System Info

```
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset
import datasets

dataset = datasets.load_dataset("imdb", name="plain_text", split="unsupervised")
pipe = pipeline("text-classification", device=0)
for out in pipe(KeyDataset(dataset, "text"), batch_size=8, truncation="only_first"):
    print(out)
    # [{'label': 'POSITIVE', 'score': 0.9998743534088135}]
    # Exactly the same output as before, but the content are passed
    # as batches to the model
```

Why is batching not giving a list (of 8 here) of outputs? Isn't it the point of batching or am I missing something? Eitherway, it's counter intuitive.

This code is from the [the pipeline doc page](https://huggingface.co/docs/transformers/main_classes/pipelines#pipeline-batching). I haven't tried it but I have this issue (?) with another pipeline task.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run the above code

### Expected behavior

[{'label': 'POSITIVE', 'score': 0.9998743534088135}, {'label': x, 'score': x}, {'label': x, 'score': x} ....]

Comments:
You are looping on the outputs of `pipe(KeyDataset(dataset, "text"), batch_size=8, truncation="only_first")`.
If you just print `pipe(KeyDataset(dataset, "text"), batch_size=8, truncation="only_first")` you'll see what you want. The example in the doc shows how you can create an iterator over these outputs, which sometimes is faster than a direct call on the batch.
[`Falcon`] Remove SDPA for falcon to support earlier versions of PyTorch (< 2.0)
Author: younesbelkada
Pull request: yes

# What does this PR do?

Removes the call to `torch.scaled_dot_product_attention` in the modeling code of Falcon. That method has been introduced only from PyTorch >= 2.0 thus it is not possible to run falcon on PT 1.13 for instance 

If one wants to use SDPA for using Flash Attention 1, one should use `BetterTransformer` that will route FalconAttention to use SDPA: https://github.com/huggingface/optimum/pull/1343 

```python
model.to_bettertransformer()
```

cc @Rocketknight1 @fxmarty 

All slow tests pass on my end


Comments:
_The documentation is not available anymore as the PR was closed or merged._

For torch > 2.0 users this will lead to an unexpected slow down and especially can lead to a huge increase in memory.

I wonder whether it might be better to go for full backwards compatibility here and instead do something like:

```py
if hasattr(F, "scaled_dot_product_attention"):
    # use SDPA
else:
    # compatible with PT 1.13
```

If we want to push better transformers we could also deprecate the `if hasattr(F, "scaled_dot_product_attention"):` branch

The code LGTM but like @patrickvonplaten says, I'm wary of causing a big slowdown + memory increase in a very important model right now, especially if users aren't aware of BetterTransformer and just see that their performance got worse without understanding why!

Can we do the `if hasattr` code he suggested to keep higher performance for newer PyTorch, or will that break compatibility with BetterTransformer?

With an attention_mask passed to SDPA, there is no dispatch to flash / mem-efficient attention in torch 2.0 anyway, so with the codebase in transformers there would be no slowdown nor memory increase for torch 2.0 users. For torch nightly users (with nighly ~> July-August) there would indeed be given that Driss added support of attention_mask with the memory-efficient attention backend.

Thanks for the review, I was not aware the modeling was already on Pypi so this is breaking, I have reverted that with respect to @patrickvonplaten & @Rocketknight1 's suggestion to keep backward compatibility!
I really think in the future we should centralize these optimisations in a single place (e.g. `BetterTransformer` API and/or Flash-Attention-2 integration), as transformers should in theory be usable also for previous PT version I think we should make sure for the new models to not use SDPA and redirect that to `BetterTransformer` for API consistency

After fixing a small nit, this is ready for a final review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25947). All of your documentation changes will be reflected on that endpoint.
Fix gated repo tests
Author: Wauplin
Pull request: yes

Related to this [slack thread](https://huggingface.slack.com/archives/C01NE71C4F7/p1692855942544019) (private).

Since end of August (https://github.com/huggingface/transformers/commit/68fa9a5937ae7aa707f5ff2639aa36a37a0a9928), gated repo tests were skipped because failing to pass. This was due to a server-side change that now makes the README files readable even on gated repo (since users have to be able to read a model card before requesting access). This PR fixes the tests -and unskip them- by trying to download [gated_file.txt](https://huggingface.co/hf-internal-testing/dummy-gated-model/blob/main/gated_file.txt) instead.

cc @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26257). All of your documentation changes will be reflected on that endpoint.
Add ViTMatte
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR supersedes #25051 and #25524. 

To do:

- [x] simplify image processor
- [x] fix image processor tests => @amyeroberts should I overwrite the following tests, since the image processor also requires trimaps to be passed?

```
FAILED tests/models/vitmatte/test_image_processing_vitmatte.py::VitMatteImageProcessingTest::test_call_numpy - TypeError: preprocess() missing 1 required positional argument: 'trimaps'
FAILED tests/models/vitmatte/test_image_processing_vitmatte.py::VitMatteImageProcessingTest::test_call_numpy_4_channels - TypeError: preprocess() missing 1 required positional argument: 'trimaps'
FAILED tests/models/vitmatte/test_image_processing_vitmatte.py::VitMatteImageProcessingTest::test_call_pil - TypeError: preprocess() missing 1 required positional argument: 'trimaps'
FAILED tests/models/vitmatte/test_image_processing_vitmatte.py::VitMatteImageProcessingTest::test_call_pytorch - TypeError: preprocess() missing 1 required positional argument: 'trimaps'
FAILED tests/models/vitmatte/test_image_processing_vitmatte.py::VitMatteImageProcessingTest::test_image_processor_from_dict_with_kwargs - AssertionError: 42 != {'height': 42, 'width': 42}
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts thanks for your review, I've addressed all comments. However regarding renaming the outputs from `alphas` to `logits` => are we sure we want to do this, given that image matting models output the alpha values, not really logits?

cc @rafaelpadilla for reference

Addressed all comments. Feel free to merge if approved

Failing tests are flaky/unrelated

@NielsRogge 
It seems to me that every comment was addressed and now all tests are passing.
I believe now it is ready to merge. May I?

Yes
<wip> Early draft of crossformer model
Author: raghavanone
Pull request: yes

Fixes #22852 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23130). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @raghavanone, thanks for opening this PR! 

The easiest, fastest and preferred way to add a new model is directly onto the hub: https://huggingface.co/docs/transformers/model_sharing

The bar for adding models into the transformers repo through a PR is a lot higher and will require all passing tests and approval from a maintainer. As such, adding this way will take a lot longer. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge  Request you to open this PR.
NLLB-CLIP model implementation
Author: visheratin
Pull request: no
Labels: name: New model

name: Model on the Hub

### Model description

Hi!

I recently trained a CLIP model with an NLLB text encoder to extend CLIP capabilities to 201 languages of the Flores-200 dataset. As far as the implementation goes, it is HF CLIP implementation with an M2M100 encoder from NLLB models. I'm wondering if you'd be interested in having NLLB-CLIP in the library? If yes, I can bring my implementation in accordance with other CLIP models and create a PR.

The link to the paper with description and results - https://arxiv.org/abs/2309.01859

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @visheratin, awesome work! 

We have recently been trying to push for `model on the hub` and have as much support as we can there. It will also be easier to integrate it. Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

Thank you, @amyeroberts!

[Here](https://github.com/visheratin/nllb-clip/tree/main/hf_model) is my current implementation that works with the hub. And I just uploaded the [base](https://huggingface.co/visheratin/nllb-clip-base) and [large](https://huggingface.co/visheratin/nllb-clip-large) variants of NLLB-CLIP to the hub.

The test code that I used as a sanity check is below:

```
from transformers import AutoTokenizer, CLIPProcessor
import requests
from PIL import Image

from modeling_nllb_clip import NLLBCLIPModel # for now, this is a local file from the repo

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
processor = processor.image_processor
tokenizer = AutoTokenizer.from_pretrained(
    "facebook/nllb-200-distilled-600M"
)
image_path = "https://huggingface.co/spaces/jjourney1125/swin2sr/resolve/main/samples/butterfly.jpg"
image = Image.open(requests.get(image_path, stream=True).raw)
image_inputs = processor(images=image, return_tensors="pt")
text_inputs = tokenizer(
    ["cat", "dog", "butterfly"],
    padding="longest",
    return_tensors="pt",
)

hf_model = NLLBCLIPModel.from_pretrained("visheratin/nllb-clip-base")

outputs = hf_model(input_ids = text_inputs.input_ids, attention_mask = text_inputs.attention_mask, pixel_values=image_inputs.pixel_values)

```

Let me know if I can do anything else!

@visheratin Awesome! The only thing to do would be adding the modeling files directly onto the hub alongside the checkpoint e.g. [like here for phi-1_5](https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_mixformer_sequential.py)

Added modeling files and a code sample to the README file. Over the weekend, I will add a proper description to the model card.
Fix Error not captured in PR doctesting
Author: ydshieh
Pull request: yes

# What does this PR do?

In https://app.circleci.com/pipelines/github/huggingface/transformers/72920/workflows/33795bcf-6b08-4c2f-b040-aa54bb44b12f/jobs/921247/artifacts, 

we have `errors.txt` but not `failures_short.txt`. But we only checked the file `failures_short.txt`, therefore the error is not detected, and showing all test passed.

This PR fixes this.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Error should have been reported in the console of https://app.circleci.com/pipelines/github/huggingface/transformers/72920/workflows/33795bcf-6b08-4c2f-b040-aa54bb44b12f/jobs/921247/steps, but it is not.

(The artifact has `errors.txt` file)
[DO NOT MERGE] Test docker + FA-2
Author: younesbelkada
Pull request: yes

# What does this PR do?

Simply tests if we can successfully build FA2 on docker

Addresses: https://github.com/huggingface/transformers/pull/25598/files#r1324800667

Comments:
Closing as FA2 is currently not supported on T4 GPUs (according to FA2 readme). As we use T4 GPUs in our workflow for slow tests we cannot test https://github.com/huggingface/transformers/pull/25598 automatically for now. 
I will test on an A100 if a PR touches a core component of FA2 modules manually

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26268). All of your documentation changes will be reflected on that endpoint.