noamaanMulla-03/transformers-issues huggingface dataset:
Fix `transformers`' DeepSpeed CI job
Author: ydshieh
Pull request: yes

# What does this PR do?

The new (cu118) base docker image has pre-installed `transformer-engine` (which wasn't the case in the previous base image). This causes DeepSpeed CI job fails from the beginning with
```bash
E   ImportError: /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
```

This PR uninstall `transformer-engine` so @ydshieh won't be the breaking bad.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Going to merge. cc @stas00 so you know now we are using CUDA 118 on CI (as you mentioned/requested once before)

super! thank you for the heads up, @ydshieh!
Appending mapped dataset to list changes previous elements of a list
Author: surya-narayanan
Pull request: no

### System Info

Hi, 

I've been trying to tokenize a dataset with different tokenizers and store it, but in doing so, am running into a bug. The general idea is that appending to a list of datasets, seems to modify previous elements.

A code notebook is here: https://colab.research.google.com/drive/1ljMwBqzCe1fHffBcPP2py9IJMrocoNIU



### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1ljMwBqzCe1fHffBcPP2py9IJMrocoNIU

### Expected behavior

Appending to the list of datasets shouldn't modify previous elements of that list.

Comments:
update- the error seems to be in the use of the lambda function, no idea why

@surya-narayanan This isn't an issue specific to `datasets` - it's a Python behaviour due to binding of the `x` var when then `lambda` function is first defined:
https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result

For future issues experienced when using `datasets`, could you make sure to open them under the [datasets repo](https://github.com/huggingface/datasets)? 

Great, thanks :)
[`Blip`] Remove redundant shift right
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/23000

In fact `_shift_right` does not need to be called inside `BlipForQuestionAnswering` as the right shifting of the tokens is done directly on the text decoder as mentioned by the user. Therefore, that class will be trained to perform next-next token prediction instead of next-token prediction.

The fix is to simply remove that shift method and the call to it.

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23153). All of your documentation changes will be reflected on that endpoint.
Fix DeepSpeed stuff in the nightly CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Similar to #23463, but for the nightly CI (i.e. the nightly version of torch + deepspeed instead of the stable release).

This should be the last piece to make all the CI workflow to run (🤞)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix confusing `transformers` installation in CI
Author: ydshieh
Pull request: yes

# What does this PR do?

As mentioned in [this comment](https://github.com/huggingface/transformers/pull/23277#issuecomment-1544080975), the `transformers` in CI runs is not the same one installed during docker image build.

Furthermore, for past CI, we didn't update the docker image daily (there is no need to update the fixed 3rd-packages environment), but the `transformers` code is the latest one being test against. We get 
```bash
E   ImportError: cannot import name 'HfDoctestModule' from 'transformers.testing_utils'
```
as the `transformers` is the one during docker image build (months ago) which has no `HfDoctestModule` at the time.

To avoid and fix all such super confusing failures in the future, it's better to install the `transformers` again in the edit mode in the workflow files.

I don't change the daily CI workflow file yet in this PR -  I would wait this PR showing the change actually fix issues.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23465). All of your documentation changes will be reflected on that endpoint.
🌐 [i18n-KO] Translated `tasks/zero_shot_object_detection.mdx` to Korean
Author: HanNayeoniee
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `tasks/zero_shot_object_detection.mdx` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sgugger, @ArthurZucker, @eunseojo May you please review this PR? 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Mismatch between config.vocab_size and len(tokenizer) in Flan-T5
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.15.0-1023-azure-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer,AutoConfig
models = [
    "google/flan-t5-small",
    "google/flan-t5-base",
    "google/flan-t5-large",
    "google/flan-t5-xl",
    "google/flan-t5-xxl",
]
for model in models:
    config = AutoConfig.from_pretrained(model)
    tokenizer = AutoTokenizer.from_pretrained(model)
    print(f"{model}\n\tlen(tokenizer)={len(tokenizer)},tokenizer.vocab_size={tokenizer.vocab_size},config.vocab_size={config.vocab_size}")
```
![draft ipynb — LaMP  SSH: v100node  2023-05-08 13-41-06](https://user-images.githubusercontent.com/38466901/236742905-813afcaa-e2ae-46fe-a6dd-612c45f04ae4.png)


### Expected behavior

The two are matched.

Comments:
See #4875.

Got it. Thanks! Voting for clarification.

I gather from the thread that it shouldn't be a problem. The size was increased for the ease of GPU usage. Is this creating any issues in inferencing or training the model? Just want to know for better understanding!

Ok, here is my usecase.
I usually calculate loss myself rather than pass `labels` into a model for automatic loss calculation.
For example, when calculating commonly used **Cross Entropy Loss**, i would have to figure out vocab size myself.
```python
loss = F.cross_entropy(
                        outputs.logits.view(-1,VOCAB_SIZE),
                        labels.view(-1),
                        label_smoothing=cfg.trainer.label_smoothing_factor
                        )
```
So i think in cases where `vocab_size` matters, the value from from `config.vocab_size` and `len(tokenzier)` should be consistent.

The pre-trained model which is provided by google had its vocab_size manually set to 32128 by them. Here's what I found in their github:
<img width="809" alt="Screenshot 2023-05-20 at 9 54 23 AM" src="https://github.com/huggingface/transformers/assets/118152679/86b3bbf1-8d72-4b9f-807b-3ab6dfb3a2ef">
Did you try setting VOCAB_SIZE manually?

Hi, thanks for the response! I just mean when i need to know the value of `vocab_size`, imo it should be consistent with `len(tokenzier)` and `config.vocab_size` in huggingface.
Fix `tests/repo_utils/test_get_test_info.py`
Author: ydshieh
Pull request: yes

# What does this PR do?

3 tests break on `main` after #23153 (see [here](https://app.circleci.com/pipelines/github/huggingface/transformers/64922/workflows/d53f3351-a1d1-4df5-9c8b-28c45fff4f09/jobs/805142)), but can't blame @younesbelkada as the tests are not triggered on PR CI, neither after being merged. But on nightly CircleCI run, failures are detected.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23485). All of your documentation changes will be reflected on that endpoint.

Thanks so much for taking care of this @ydshieh 🔥 !
Codeparrot Humaneval metric error?
Author: Keysmis
Pull request: no

Hi~, I tried to reproduce the metrics you reported by running transformers/examples/research_projects/codeparrot/scripts/human_eval.py, and the model is codeparrot-small, but the results have significant deviations. Does anyone could reproduce the results?

Comments:
cc @lvwerra 

Hi @Keysmis can you report the arguments you used for the script? And what results did you get? We updated some models and maybe we didn't update the reported metrics everywhere. cc @loubnabnl 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
CodeGenAttention does not work with defaults in forward pass
Author: sgunasekar
Pull request: no

### System Info


- `transformers` version: 4.28.1
- Platform: Linux-5.15.0-1034-azure-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0a0+1767026 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?


The current version of models.codegen.modeling_codegen.CodeGenAttention forward throws error on line 193 when position_ids are not specified and defaults to None. This can be fixed by defining default position_ids as self.position_ids in the init. The issue  was an issue introduced in commit 4e94c6c. 

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers.models.codegen.modeling_codegen import CodeGenAttention
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("Salesforce/codegen-350M-nl")
model = CodeGenAttention(config)
x= torch.randn(4, config.n_ctx, config.n_embd)
model(x)
```


### Expected behavior

The block should instantiate the codegenattention with default position ids as torch.arange(seq_offset:seqlen)

Comments:
Hi @sgunasekar 
Thanks for the issue
As per my understanding, since the class `CodeGenAttention` is not a public class, it should be only used by `CodeGenModel`. In the modeling script if position ids is `None` we indeed manually create `position_ids` based on past length and sequence length. I personally don't think we should do this inside `CodeGenAttention`, but if you want to use that class as a standalone class you should manually create position ids and pass it in the forward pass.
I also want to hear from @ArthurZucker, @sgugger @amyeroberts to make sure we are aligned on this

👍🏻 on @younesbelkada 's answer, on almost all of our attention modules, the attention should be passed and there are not reason to give them a default value because this is handled in the modelling file. 

Completely in line with the comments above.

+1 - As @younesbelkada says, `CodeGenAttention` isn't a public class and this is easily resolved by passing in `position_ids` directly to the layer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix tensor device while attention_mask is not None
Author: zspo
Pull request: yes

# What does this PR do?

The PR while fix the tensor device while attention_mask is not None, the tensor like `torch.tensor(torch.finfo(attn_weights.dtype).min)`.

1. I don't use the `os.environ[“CUDA_VISIBLE_DEVICES”]` because i want to load other models in the same script.
2. Use `torch.cuda.set_device(6)` while other gpu device has already occupied.
3. The model is loaded in device `cuda:6`, but the new tensor will be load in `cuda:0` while attention_mask is not None.

```
{
    'input_ids': tensor([[...]], device='cuda:6'), 
    'attention_mask': tensor([[...]], device='cuda:6')
}

input_ids device:  cuda:6
model device:  cuda:6

print('attn_weights device', torch.tensor(torch.finfo(attn_weights.dtype).min).device)
attn_weights device cuda:0
```

```
 File "/data1/env/miniconda3/envs/llmdev/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 231, inforward
    attn_weights = torch.max(attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min))
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:6 and cuda:0!
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23534). All of your documentation changes will be reflected on that endpoint.
🌟 Mirostat decoding algorithm
Author: astariul
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm

## Description

Paper : https://arxiv.org/pdf/2007.14966.pdf

Abstract : 

> [...] We use this analysis to design a feedbackbased adaptive top-k text decoding algorithm called mirostat that generates text (of any length) with a redetermined value of perplexity, and thereby high-quality text without any tuning. [...] Mirostat avoids both traps: experiments show that cross-entropy has a near-linear relation with repetition in generated text. This relation is almost independent of the sampling method but slightly dependent on the model used. Hence, for a given language model, control over perplexity also gives control over repetitions.

## Open source status

* [x] the model implementation is available: https://github.com/basusourya/mirostat
* [x] the model weights are available: _Not applicable_
* [x] who are the authors: @basusourya 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RobertaTokenizerFast produces a different output than RobertaTokenizer
Author: HHousen
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

`RobertaTokenizerFast.tokenize()` produces a different output than `RobertaTokenizer.tokenize()`. I am not sure if this is an issue that will impact model performance. Is this intended? I assumed the fast tokenizers should be consistent with the normal ones in terms of outputs.

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
from transformers import RobertaTokenizer, RobertaTokenizerFast

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
tokens = tokenizer.tokenize("This is a test. 
 
 Another one. 
 
 Yet another one.")
print("Normal Tokens: " + str(tokens))

ids = tokenizer.convert_tokens_to_ids(tokens)
print("Normal IDs: " + str(ids))


tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
tokens = tokenizer.tokenize("This is a test. 
 
 Another one. 
 
 Yet another one.")
print("Fast Tokens: " + str(tokens))

ids = tokenizer.convert_tokens_to_ids(tokens)
print("Fast IDs: " + str(ids))
```

Output:

```
Normal Tokens: ['This', 'Ġis', 'Ġa', 'Ġtest', '.', '
', '
', 'ĠAnother', 'Ġone', '.', '
', '
', 'ĠYet', 'Ġanother', 'Ġone', '.']
Normal IDs: [713, 16, 10, 1296, 4, 2, 0, 2044, 65, 4, 2, 0, 3507, 277, 65, 4]
Fast Tokens: ['ĠThis', 'Ġis', 'Ġa', 'Ġtest', '.', 'Ġ', '
', 'Ġ', '
', 'ĠAnother', 'Ġone', '.', 'Ġ', '
', 'Ġ', '
', 'ĠYet', 'Ġanother', 'Ġone', '.']
Fast IDs: [152, 16, 10, 1296, 4, 1437, 2, 1437, 0, 2044, 65, 4, 1437, 2, 1437, 0, 3507, 277, 65, 4]
```

Using `tokenizer.enocde()` instead of `tokenizer.convert_tokens_to_ids(tokenizer.tokenize())` solves the discrepancy with the first token but still inserts token id `1437` between `
` and `
`.

## Expected behavior

`RobertaTokenizerFast` produces the same output as `RobertaTokenizer`.

## Environment info
     
- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
For records for the new users: Now, it seemed to have been resolved. I tried to reproduce this in my notebook but getting same results for both of them:

<img width="1383" alt="image" src="https://github.com/huggingface/transformers/assets/104596164/bde951ac-8b9c-4996-966b-1d11f4c12d35">
About Whisper finetuning on a out-of-vocabulary language datset
Author: LYPinASR
Pull request: no

### Feature request

Hello!
I am working on Whisper, and I want to finetune the model on _Amharic_ in OpenASR20 dataset. As far as I know,  _Amharic_ is not one of the languages participating in Whisper training. If I want to finetune the model on _Amharic_, what should I do?
Looking forward to your reply! Thank you!

### Motivation

Whisper finetuning on a out-of-vocabulary language datset

### Your contribution

I can do as needed.

Comments:
Hi @LYPinASR, thanks for raising an issue! 

Questions like this are best placed [in the forum](https://discuss.huggingface.co/), as we try to reserve github issues for bug reporting and feature requests. Here's a relevant thread about a previous finetuning event which might help: https://discuss.huggingface.co/t/open-to-the-community-whisper-fine-tuning-event/26681

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix passing kwargs to the tokenizer in FillMaskPipeline preprocess method
Author: fxmarty
Pull request: yes

As per title, the kwargs was not passed.

The modification follows https://github.com/huggingface/transformers/blob/fe1f5a639d93c9272856c670cff3b0e1a10d5b2b/src/transformers/pipelines/text_classification.py#L91-L179

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22709). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.