noamaanMulla-03/transformers-issues huggingface dataset:
Make CLIP model could use new added tokens with meaningful pooling
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #24650

This is to address the feature request #24650.

Although we have default values of bos/eos been corrected in #24773, the existing config on the Hub still have the incorrect value `1` and `2`, which prevents CLIP model to use new added tokens when a user add them.

Although we can open mass PRs on the Hub, I want to decouple (slightly) this with the ability to support such feature.
With this PR, if a user want to use new added tokens, they has to specify/update the `eos_token_id`.

**We don't need to wait all Hub repo. to be updated to merge this PR.**



Comments:
I will fix the CI by using `fix-copies` later

_The documentation is not available anymore as the PR was closed or merged._

I will merge once the branch is cut tonight.

Sorry for the spam: @sgugger said the branch cut would be on next Monday. I think it's safer to wait until then.
Remove `tests/onnx`
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove `tests/onnx`, as mentioned, as discussed

https://github.com/huggingface/transformers/pull/24800#issuecomment-1634822781

Note there are still some tests like `TFGPT2ModelTest::test_onnx_runtime_optimize` which are not removed in this PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
deprecate no_cuda
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR deprecates the `no_cuda` arg because it is confusing for Mac users as their models get dispatched to `mps` device when `no_cuda=False`. If they want to train the model on cpu, they need to set `no_cuda=True` which is not intuitive. We rename it `use_cpu` instead. 

Related issue #24697 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add eval and test for the example run_ner_no_trainer.py
Author: fuzihaofzh
Pull request: no

### Feature request

Hi, thanks for the great work. I just go through the example "transformers/examples/pytorch/token-classification/", I found that run_ner.py has eval and test. However, for run_ner_no_trainer.py, there is no eval or test. I just wonder if it is possible to add these components?

### Motivation

Enhance the example code

### Your contribution

I can help to test the code.

Comments:
Evaluation is done at every epoch in this script, not sure what more you need. The `no_trainer` scripts are more barebone on purpose, to make it easy for users to customize them.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Evaluating NER performance with subword tokenization is complicated. It would be very instructive if you could include code that outputs one prediction for each token in the original corpus.  This would have to address the behavior of tokenizers, such as splitting words with hyphens and not prepending them with ##, etc. 

 As I understand it, the script currently performs evaluation on the subword tokens and not the restored versions.
Skip failing `ZeroShotAudioClassificationPipelineTests::test_small_model_pt` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Skip failing `ZeroShotAudioClassificationPipelineTests::test_small_model_pt` for now.

see [failing job](https://app.circleci.com/pipelines/github/huggingface/transformers/68367/workflows/0d616969-381a-4ce2-96f9-ec83b259df75/jobs/856192)

likely a `datasets` issue

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`Docs`] Clarify 4bit docs
Author: younesbelkada
Pull request: yes

# What does this PR do?

As discussed internally with @lewtun , this PR refactors a bit the 4bit docs by adding more clarifications about best practices and giving relevant pointers to users about advanced usage.

Also fixed the requirements instructions as 4bit is now part of the latest release

cc @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Enable `ZeroShotAudioClassificationPipelineTests::test_small_model_pt`
Author: ydshieh
Pull request: yes

# What does this PR do?

This test was failing due to a dev version of `datasets` (created in another branch) being used in `main`. It's now resolved. 

Thank you @lhoestq for the investigation.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add DINOv2 to Transformers
Author: EduardoPach
Pull request: no
Labels: name: New model

### Feature request

Add DINOv2 to the transformers library.

Weights are available in [DINOv2 repo](https://github.com/facebookresearch/dinov2)

### Motivation

Currently, DINOv2 can be used through `torch.hub.load`, but having it ported to transformers directly would be nice to have, and since DINOv1 is already in the library it might not be that difficult to do

### Your contribution

I would love to do a PR to make this addition in the case that it's actually not something hard to do and someone could point me the way that I should look at.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`InstructBlip`] Fix int8/fp4 issues
Author: younesbelkada
Pull request: yes

# What does this PR do?

FIxes: https://github.com/huggingface/transformers/issues/24884

To reproduce:

```python
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
import torch
from PIL import Image
import requests

device = "cuda" if torch.cuda.is_available() else "cpu"

MODEL_NAME = "Salesforce/instructblip-flan-t5-xl"
# Note: Here we no longer specify `torch.bfloat16`.
model = InstructBlipForConditionalGeneration.from_pretrained(MODEL_NAME, device_map={"":0}, load_in_4bit=True)

processor = InstructBlipProcessor.from_pretrained(MODEL_NAME)

url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = "What is unusual about this image?"

# Note: Here we no longer specify `torch.bfloat16`, but we use `torch.float16` as shown in the test code for Salesforce/instructlblup-vicuna-7b
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)

outputs = model.generate(
    **inputs,
    do_sample=False,
    num_beams=5,
    max_length=256,
    min_length=1,
    top_p=0.9,
    repetition_penalty=1.5,
    length_penalty=1.0,
    temperature=1,
)

generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)
```

Strangely I couldn't reproduce the issue with vicuna models but managed to reproduce with flan-t5 models. Also it is very strange that users never reported the same issue with Blip2

cc @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`Blip`] Fix blip output name
Author: younesbelkada
Pull request: yes

As suggested by @ydshieh offline and similarly as https://github.com/huggingface/transformers/pull/22893
In fact, there is no reason to call the output logits `decoder_logits`, as they always come from the decoder

cc @sgugger @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi thanks for this @younesbelkada ❤️ 

However, this is not enough, you have to add something like 

https://github.com/huggingface/transformers/blob/3ec10e6c76362191b61260300fe1d6173a8dd7e1/src/transformers/models/swin/modeling_swin.py#L171-L177

in that PR to make backward compatibility.

(unless @sgugger say this model is still recent and/or not high usage)

Thanks @ydshieh for double checking, I missed that, it should be now added! :D 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24889). All of your documentation changes will be reflected on that endpoint.
`trainer.evaluate` does throws an error when using multiple evaluation dataset 
Author: hwijeen
Pull request: no

### System Info

```
- `transformers` version: 4.31.0.dev0
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```

### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. use transformers [examples code](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) on summarization (or any other)
2. Pass multiple evaluation dataset as follows when running the code. This should be supported as [documented here](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.eval_dataset).
```
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --train_file "a.txt" "b.txt" \
    --validation_file "a_valid.txt" "b_valid.txt"
```

### Expected behavior

The example code should not meet an error when `trainer.evaluate` is called, which is either 1) inte·mittently during training and 2) [at the end of the training](https://github.com/huggingface/transformers/blob/5bb4430edc7df9f9950d412d98bbe505cc4d328b/examples/pytorch/summarization/run_summarization.py#L695). 

During the training, Trainer [checks if the the passed `eva_dataset` consists of multiple or not](https://github.com/huggingface/transformers/blob/5bb4430edc7df9f9950d412d98bbe505cc4d328b/src/transformers/trainer.py#L2216). Since this line is missing in the 2) at the end of training evaluation case, this meets an error. I'd be happy to make a PR on this :)

Comments:
Yes, please do suggest a PR to fix this, thanks!
Separate CircleCI cache between `main` and `pull` or other branches
Author: ydshieh
Pull request: yes

# What does this PR do?

Keep the CircleCI cache running on `main` branch not affected by other branches/PR.

Reading the following by keeping in mind:
 - @lhoestq created a branch yesterday using `datasets 2.13.2.dev0`
 - @sgugger changed `setup.py` yesterday in the commit `4.32.0.dev0`
 - the cache used by @lhoestq 's branch is used on `main` in/after @sgugger 's commit.

Also: the CI triggered on `main` is much less frequently, so keeping its own cache from the pull events is fine (in terms of the cost)

Assume
  - someone changes `setup.py` to use a dev. version of a library, say `datasets` in a PR or a HF non-main branch
    - CI triggered + [precise] cache not found + [partial] cache found + cache updated with `datasets` dev. version
  - shortly, another person change `setup.py` (not necessary the same library involved) in another PR/branch and being merged
    - CI triggered+ [precise] cache not found + [partial] cache found: 
      - this could be the above one (depending the time gap)
      - `datasets` version remains `dev` if it has `datasets>=XXX` in `setup.py` in the merged PR
        - (as `dev` is newer version, so requirement already satisfied)
        - we get failures on `main` due to the `datasets dev` version, which should be avoided.




Comments:
_The documentation is not available anymore as the PR was closed or merged._
Disable ipex env var if false
Author: muellerzr
Pull request: yes

# What does this PR do?

Properly sets the Accelerate env variable if apex is set to False (the default in training args)


Fixes # (issue)

Solves https://github.com/huggingface/transformers/issues/24871


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Check for accelerate env var when doing CPU only
Author: muellerzr
Pull request: yes

# What does this PR do?

Checks if `ACCELERATE_USE_CPU` is enabled to ensure trainer parts get set properly if so (and if `--use_cpu` isn't used)

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Avoid some pipeline tasks to use `use_cache=True`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix  #24873

For example, we can pass `use_cache=True` (or set this in config) to `BartForSequenceClassification` and it will return `past_key_values` (although not useful to the task).

This is to respond to #24873, despite the memory issue reported there is not 100% for sure yet.
However, avoid using `cache` and not to let model returning `past_key_values` can avoid overhead like CPU/GPU communication (huge/many tensors).

In the code snippet I provided in #24873, there is a gain of 16.5% of running time.

We could probably extend this PR to other pipeline task classes.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

LGTM!  Thanks for diving into this

Thanks for resolving this so quickly @ydshieh!
🌐 [i18n-KO] Translated `perf_train_tpu_tf.md` to Korean
Author: 0525hhgus
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `perf_train_tpu_tf.md` file of the documentation to Korean 😄 
Thank you in advance for your review!

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- Team PseudoLab, may you please review this PR?  -->
@kihoon71, @0525hhgus, @54data, @Sunmin0520, @seank021, @augustinLib

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? -->
<!-- @sgugger, @ArthurZucker, @eunseojo -->

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update the RWKV documentation with fixes to spelling and wording
Author: DevJake
Pull request: yes

# What does this PR do?
This PR fixes and performs the following in the RWKV documentation:
- Various words spelt incorrectly
- Grammatical issues
- Remove unnecessary verbosity in places

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24168). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
GPT2ForQuestionAnswering: how to use? 
Author: mkschreder
Pull request: no

https://github.com/huggingface/transformers/blob/8f093fb799246f7dd9104ff44728da0c53a9f67a/docs/source/en/model_doc/gpt2.mdx?plain=1#L116

The documentation generated for this question answering model is complete nonsense: https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2ForQuestionAnswering.forward.example

It does not produce any meaningful results and the tensors are hardcoded:

```
# target is "nice puppet"
target_start_index = torch.tensor([14])
target_end_index = torch.tensor([15])
```



Comments:
Hi @mkschreder, thanks for raising this issue. 

In terms of the issue title - how to use - there's a more in-depth guide about question-answering in the [task documentation](https://huggingface.co/docs/transformers/v4.30.0/en/tasks/question_answering#question-answering) and [NLP course](https://huggingface.co/learn/nlp-course/chapter7/7?fw=pt). 

The snippets in the documentation are meant to be minimal examples so that users can get started and understand the model's API. Sometimes values are hardcoded to keep the example short, however we certainly don't want them to be incorrect or confusing. If there's particular improvements or fixes you'd like to see, we're always happy to review a PR :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[doc] `image_processing_vilt.py` wrong default documented
Author: stas00
Pull request: yes

Sync the doc with reality.

https://github.com/huggingface/transformers/blob/ee4250a35f3bd5e9a4379b4907b3d8f9d5d9523f/src/transformers/models/vilt/image_processing_vilt.py#L299

https://github.com/huggingface/transformers/blob/ee4250a35f3bd5e9a4379b4907b3d8f9d5d9523f/src/transformers/models/vilt/image_processing_vilt.py#L310

Comments:
_The documentation is not available anymore as the PR was closed or merged._