noamaanMulla-03/transformers-issues huggingface dataset:
Bug in LlamaTokenizer when `return_token_type_ids=True`
Author: ry-sun
Pull request: no

### System Info

- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

tokenizers: @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I use LLama's tokenizer and pass `return_token_type_ids=True`, I found that the length of the return value `token_type_ids` is different from `input_ids` and `attention_mask`.

```python
In [2]: from transformers import AutoTokenizer

In [3]: tok = AutoTokenizer.from_pretrained('/mnt/checkpoint/llama_7B_hf', use_fast=False)

In [4]: inputs = 'I LOVE'

In [5]: outputs = 'huggingface'

In [6]: tok(inputs, outputs, return_token_type_ids=True)
Out[6]: {'input_ids': [1, 306, 11247, 12064, 1, 298, 688, 3460, 2161], 'token_type_ids': [0, 0, 0, 0, 0, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}

In [7]: list(map(len, _.values()))
Out[7]: [9, 10, 9]
```

### Expected behavior

```python
In [6]: tok(inputs, outputs, return_token_type_ids=True)
Out[6]: {'input_ids': [1, 306, 11247, 12064, 1, 298, 688, 3460, 2161], 'token_type_ids': [0, 0, 0, 0, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}

In [7]: list(map(len, _.values()))
Out[7]: [9, 9, 9]
```

Comments:
Hey yep that's indeed a bug thanks for reporting
Generate: handle text conditioning with multimodal encoder-decoder models
Author: gante
Pull request: yes

# What does this PR do?

Consolidates `decoder_input_ids` preparation changes in a single place, for all future multimodal encoder-decoder models on PT and TF.

In a nutshell, this PR generalizes the following use cases:
1. The user passes `decoder_input_ids`, but it is missing the BOS token (some tokenizers, like the T5 tokenizer, do not prepend a BOS token). In this case, a BOS token is prepended.
2. The user passes `input_ids`, but the encoder has no `input_ids` input. In this case, `input_ids` is handled just like `decoder_input_ids`.

Slow tests were run on T5, Pix2Struct, BLIP, and BLIP2.

Comments:
cc @younesbelkada @NielsRogge FYI -- this PR consolidates your recent changes regarding text conditioning on multimodal models. The next models should be easier to add :) 

_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot @gante! 🙏
Indexing fix for gpt_bigcode
Author: jlamypoirier
Pull request: yes

The `past_key_values` is a list of tensors in gpt_bigcode, and not a list of lists like with most models, so we don't want to index twice. The code works either way but it's safer to avoid the unnecessary indexing. I also fixed the associated type hint.
@younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix `serving_output` for TF composite models (encoder-decoder like models)
Author: ydshieh
Pull request: yes

# What does this PR do?

**[If the concept is approved, I will apply the same changes to other TF encoder-decoder family of models]**

The composite models use its components' configurations. See for example

https://github.com/huggingface/transformers/blob/95e7057507c9eca8e997abb98645eee2621a5aea/src/transformers/modeling_tf_utils.py#L426-L430

However, in some places, our codebase still try to access some attributes at the top level of the configuration (i.e. not inside the 2 components), like

https://github.com/huggingface/transformers/blob/95e7057507c9eca8e997abb98645eee2621a5aea/src/transformers/models/vision_encoder_decoder/modeling_tf_vision_encoder_decoder.py#L664-L669

In particular, `self.config` may not have `use_cache`, for example, for this checkpoint `"nlpconnect/vit-gpt2-image-captioning"`. We should instead look  `self.config.deocder.use_cache`.

**This PR try to follow the rule of  ` # Encoder Decoder models delegate the application of the configuration options to their inner models. `.**

**This PR is also (another) one necessary step to fix #22731.**

Comments:
_The documentation is not available anymore as the PR was closed or merged._
🌐[i18n-KO] Translate `autoclass_tutorial` to Korean and Fix the typo of `quicktour`
Author: gabrielwithappy
Pull request: yes

# What does this PR do?

Translated the `autoclass_tutorial.mdx` file of the documentation to Korean and fix the typo of `quicktour`
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
<!-- 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- @sgugger, @ArthurZucker, @eunseojo May you please review this PR? -->
Team PseudoLab, may you please review this PR?
@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

May you please review this PR?
@sgugger, @ArthurZucker, @eunseojo 

## Review result
- [x] fix a wrong source code link of functions in the document 
> Links for API documents are not activated. I checked other language documents have same problem. I think it will be fixed when API documents are translated.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

- need to add the english link of `Load pretrained instances with an AutoClass`
- keep the `AutoClass` as a english

squashed commit messages and check a final document result.


@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd
Please review thie PR. 
Thank you in advance.

[Korean]
음.. 좀 헷갈리네요. 링크가 안되는 이유가 뭔지 좀 알아봐야 될 것 같습니다.
소스코드에 링크가 안걸리는게 문제 같습니다. 좀 더 찾아볼께요
[English]
I will check why the hyperlink does not work.
I think I missed somthing on link code of source codes in the document

Thank you.
I found my source code links are wrong.
I updated review action items and fix it @HanNayeoniee 

May you please review this PR?
@sgugger, @ArthurZucker, @eunseojo

Thanks for your contribution!
🌐 [i18n-KO] Translated `sequence_classification.mdx` to Korean
Author: 0525hhgus
Pull request: yes

# What does this PR do?

Translated the `tasks/sequence_classification.mdx` file of the documentation to Korean.
- The file name is `sequence_classification.mdx`, but the document name is `text classification`.
- Currently, it is being revised to consistent vocabulary.

Thank you in advance for your review:)

Part of https://github.com/huggingface/transformers/issues/20179

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

<!-- 제출 전 체크리스트로, 가짜연구소만의 체크리스트도 <details>로 감싸서 만들어두면 더 좋을 것 같아요. -->

## Who can review?
<!-- 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR?

@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

May you please review this PR?
@sgugger, @ArthurZucker, @eunseojo

Comments:
_The documentation is not available anymore as the PR was closed or merged._

May you please review this PR?
@sgugger, @ArthurZucker, @eunseojo
Generate: pin number of beams in BART test
Author: gante
Pull request: yes

# What does this PR do?

A recent change added the `num_beams==1` requirement for contrastive search. One of the tests started failing because of that change -- BART has `num_beams=4` [in its config](https://huggingface.co/facebook/bart-large-cnn/blob/main/config.json#L49), so the test was now triggering beam search, and not contrastive search. This PR corrects it.

(The long-term solution is to add argument validation to detect these cases)



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix a mistake in Llama weight converter log output.
Author: aljungberg
Pull request: yes

# What does this PR do?

Fixes a mistake in Llama weight converter log output.

Before: `Saving a {tokenizer_class} to {tokenizer_path}`
After: `Saving a LlamaTokenizerFast to outdir.`


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada


Comments:
_The documentation is not available anymore as the PR was closed or merged._
TypeError: export() got an unexpected keyword argument 'preprocessor'
Author: susht3
Pull request: no
Labels: name: New model

### Model description

onnx_inputs, onnx_outputs = export(
            preprocessor=tokenizer, model=model, config=onnx_config, opset=10, output=onnx_model_path
        )

i got error: TypeError: export() got an unexpected keyword argument 'preprocessor'

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
@susht3, could you please follow the issue template and provide a minimal reproducible code snippet, information about the running environment (found using `transformers-cli env`) and a full trackback of the error?
Tweak ESM tokenizer for Nucleotide Transformer
Author: Rocketknight1
Pull request: yes

Nucleotide Transformer is a model that takes DNA inputs. It uses the same model architecture as the protein model ESM, but in addition to a different vocabulary it tokenizes inputs without a `<sep>` or `<eos>` token at the end. This PR makes a small tweak to the tokenization code for ESM, so that it doesn't try to add `self.eos_token_id` to sequences when the tokenizer does not have an `eos_token` set. With this change, we can fully support Nucleotide Transformer as an ESM checkpoint.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Report a hyperlink mistake
Author: PolarisRisingWar
Pull request: no

The website is https://huggingface.co/docs/transformers/tasks/token_classification
And the sentence is: Mapping all tokens to their corresponding word with the [word_ids](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html#tokenizers.Encoding.word_ids) method.
The hyperlink is mistake, it should be the tokenizer in transformers package (https://huggingface.co/docs/transformers/v4.27.2/en/main_classes/tokenizer#transformers.BatchEncoding.word_ids) but the hyperlink given is in tokenizers package.

Comments:
Thanks for reporting! Would you be interested in opening a PR to fix it? The exact file to be modified is https://github.com/huggingface/transformers/blob/main/docs/source/en/tasks/token_classification.mdx and then you would have the contribution :) Happy to open a PR otherwise.

Transfering to the transformers repo as well as that's the correct one

Happy to open a PR if needed

@mayankagarwals Great! Feel free to open a PR and ping me for review :) 

Sure @amyeroberts 

@amyeroberts https://github.com/huggingface/transformers/pull/22765/files . Please review :)
Seq2SeqTrainer: Evict decoder_input_ids only when it is created from labels
Author: gante
Pull request: yes

# What does this PR do?

Fixes #22634 (what remains of the issue, the [failing FSMT command in this comment](https://github.com/huggingface/transformers/issues/22634#issuecomment-1500919952))

A previous PR (#22108) expanded the capabilities of the trainer, by delegating input selection to `.generate()`. However, it did manually evict `decoder_input_ids` from the inputs to make the SQUAD test pass, causing the issue seen above.

This PR makes a finer-grained eviction decision -- we only want to evict `decoder_input_ids` when it is built from `labels`. In this particular case, `decoder_input_ids` will likely have right-padding, which is unsupported by `.generate()`.

Comments:
> Did you confirm the failing test now passes?

Yes :D Both the SQUAD test that made me add the previous eviction and the command that FSMT command that @stas00 shared are passing with this PR

_The documentation is not available anymore as the PR was closed or merged._
Indexing fix - CLIP checkpoint conversion
Author: amyeroberts
Pull request: yes

# What does this PR do?

In the conversion script, there was an indexing error, where the image and text logits where taken as the 2nd and 3rd outputs  of the HF model. However, this is only the case if the model returns a loss. 

This PR updates the script to explicitly take the named parameters. 

Fixes #22739


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
A minor change in the `decoder_config` of T5Model
Author: uakarsh
Pull request: no

In the line mentioned below, 

https://github.com/huggingface/transformers/blob/fb3aa06cb673d0e2774a2924747d3290135c09cc/src/transformers/models/t5/modeling_t5.py#L1339,

I believe there should be an update in the `decoder_config`

The below is the change, I want to suggest

```python
decoder_config = copy.deepcopy(config) 
decoder_config.update("is_decoder", True)
```

This is because, we are using the decoder configuration, and when loading from the `PreTrained` class, the same is suggested.

Not sure whom to tag, @sgugger @NielsRogge, could you address it?



Comments:
My mistake, sorry for the issue
Support `padding_side` in `Blip2Processor`
Author: jemmyshin
Pull request: no

### Feature request

Support different `padding_side` for `Blip2Processor`.

### Motivation

When I use BLIP2 LLM, I found that the padding style is different between decoder-only model (opt-2.7b for example) and encoder-decoder model (flan-t5-xl for example). So I assume that the paddings are different from `Salesforce/blip2-opt-2.7b` and `Salesforce/blip2-flan-t5-xl`. But actually I got the same padding results, the default is `padding_side`=right. 

Code example:
```
prompt = ["hello world", "Question: how many cats are there? Answer:"]
processor_1 = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
processor_2 = Blip2Processor.from_pretrained("Salesforce/blip2-flan-t5-xl")

inputs_1 = processor_1(text=prompt, return_tensors="pt", padding=True)
inputs_2 = processor_2(text=prompt, return_tensors="pt", padding=True)
```

Output (the same for inputs_1 and inputs_2):
```
{'input_ids': tensor([[    2, 42891,   232,     1,     1,     1,     1,     1,     1,     1,
             1],
        [    2, 45641,    35,   141,   171, 10017,    32,    89,   116, 31652,
            35]]), 'attention_mask': tensor([[1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
```

I believe that right padding for flan-t5 will give the wrong outputs when calling `generate`, correct me if I am wrong: (transformers/generation/utils.py)
<img width="940" alt="image" src="https://user-images.githubusercontent.com/16580382/223613908-3ca57575-5536-4296-a449-e49ad3b4fa90.png">


Expected outputs (when setting `padding_side`=left):
```
{'input_ids': tensor([[    1,     1,     1,     1,     1,     1,     1,
             1,     2, 42891,   232],
        [    2, 45641,    35,   141,   171, 10017,    32,    89,   116, 31652,
            35]]), 'attention_mask': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
```

### Your contribution

I found the this `padding_side` feature exists in `AutoTokenizer`, is that possible to move this feature in `Blip2Processor`? 

Comments:
Hi,

You can achieve that by simply updating the `padding_side` attribute of the processor's tokenizer:
```
processor.tokenizer.padding_side = "left"
```
Note that Blip2Processor is just a wrapper around both the image processor and the tokenizer.

Thanks a lot!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Dataset load problem when using own data to run run_mim.py
Author: chenbingxiayu
Pull request: no

### System Info

transformers 4.27.0
python 3.8.16
Ubuntu 20.04

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I directly use the script run_mim.py (examples/pytorch/image-pretraining) to fine-tune the VIT model on my own data, I got the error FileNotFoundError: Unable to find 'my dataset absolute path' at /. However, when I run the script run_image_classification_no_trainer.py (examples/pytorch/image-classification) to fine-tune the VIT model on the same data with the same path, everything is ok.

### Expected behavior

I compare the implementation of run_mim.py and run_image_classification_no_trainer.py. It seems that the previous one has some problems when sets train_dir. 
In run_image_classification_no_trainer.py, the implementation is data_files["train"] = os.path.join(args.train_dir, "**") seeing https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification_no_trainer.py#L266
In run_mim.py, the implementation is data_files["train"] = self.train_dir seeing https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-pretraining/run_mim.py#L109
The later one misses "**" in the file_path for loading dataset.
I changed the code in my local file, and the script run_mim.py runs well.

Comments:
Would you like to open a PR with your fix?

> Would you like to open a PR with your fix?
I am not sure whether my solution is correct, I hope your organization could check it. If the solution is ok,  it is my honor to open a new PR.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
DataCollatorForSpanPreTraining
Author: zanussbaum
Pull request: no

### Feature request

It seems that there's already a script existing for [T5 pretraining](https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_t5_mlm_flax.py) that has a DataCollator, but only available in Flax. Would we be able to add a Data Collator for the Span pretraining task that's implemented in the T5 papers?

### Motivation

Currently it's not super easy to run T5 pretraining in Pytorch with Transformers

### Your contribution

I can help with the PR!

Comments:
Transformers is a library of models, not data collators. You can adapt the code of this data collator to PyTorch in your code, but we won't have it in the main library (the same way the Flax one is just in an example script).

No worries, I thought maybe it would fit nicely within the same class as [DataCollatorForLanguageModeling](https://github.com/huggingface/transformers/blob/v4.26.1/src/transformers/data/data_collator.py#L609) but totally understand wanting to keep the scope contained!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add a newline here in the docstring
Author: stefanvasilev
Pull request: no

There should be a newline to separate prev_key_values from inputs_embeds.
https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/whisper/modeling_whisper.py#L827

Comments:
Seems like it indeed! Do you want to suggest a PR with the change?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.