noamaanMulla-03/transformers-issues huggingface dataset:
Explanation of the default "auto" values for DeepSpeed stage 3?
Author: garrett361
Pull request: no

Hi, I would like to know how the default values of the various [default DeepSpeed stage 3 parameters](https://huggingface.co/docs/transformers/main_classes/deepspeed?_sm_vck=D7sD5KLsJ4kfrtW4NKbQJqJWqH4bkFR7JkHK7fQtHk65Kst1nq7r#zero3-config) were determined when using `"auto"` fields. They seem to work quite well, but I can't find any documentation of their origins.

What experiments or computations were done to land on the `reduce_bucket_size`, `stage3_prefetch_bucket_size `, and `stage3_param_persistence_threshold ` formulas below?  

Maybe @stas00 ? I see that you have written most of this code. Thank you in advance!

https://github.com/huggingface/transformers/blob/fdd81aea12f06e24ab5cf5ba3c7316df3ab1a779/src/transformers/deepspeed.py#L208-L212

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Ah, thank you! Will ask there, I wasn't aware of the forums.

@garrett361,

- The only real magical `auto` configs are documented here: https://huggingface.co/docs/transformers/main/en/main_classes/deepspeed#zero3-config - the document explains why those values were chosen and the know how came via oral recommendations from Deepspeed developers.
- The rest `auto` values are just there to pass through `TrainingArguments` so that they match on both sides.

Thanks @stas00 , please ignore the similar thread I started in the forums.

> the know how came via oral recommendations from Deepspeed developers

As I feared. Appreciate the confirmation, though. 

Please don't hesitate to file an Issue with Deepspeed to ask for documenting such essential nuances in the Deepspeed core documentation. I'm pretty sure there are other such tune up nuances that I don't think are documented anywhere on their website/repo.
🌐 [i18n-KO] Translated `perf_train_tpu_tf.md` to Korean
Author: 0525hhgus
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `perf_train_tpu_tf.md` file of the documentation to Korean. 
Thank you in advance for your review!
(I closed https://github.com/huggingface/transformers/pull/24896 and opened a new one.)

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR?
@kihoon71, @0525hhgus, @54data, @Sunmin0520, @seank021, @augustinLib

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
May you please review this PR?
@sgugger, @ArthurZucker, @eunseojo

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25433). All of your documentation changes will be reflected on that endpoint.

LGTM!

@0525hhgus May you please update the status to `Open` and tag our wonderful reviewers at Hugging Face? Thank you.

Documentation privew throws a 500 error. 
May you please review this PR? It looks fine in my github codespace. 
Thank you!
@sgugger, @ArthurZucker, @eunseojo

https://moon-ci-docs.huggingface.co/docs/transformers/pr_25433/ko/perf_train_tpu_tf
<img width="1280" alt="image" src="https://github.com/huggingface/transformers/assets/47289574/c1843192-d45d-4033-be43-c281c860886f">
Xformers is not installed correctly.
Author: david-waterworth
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

``` python 
from transformers import pipeline
pipe = pipeline("text-classification", model="roberta-base", device=0)
```

Edit: I know this model isn't trained for the "text-classification" task, I get the same problem with a private model I fine tuned.

Results in the message

```
...
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
```

But I'm using torch==2.0.1 and [memory-efficient-attention](https://huggingface.co/docs/diffusers/optimization/fp16#memory-efficient-attention ) states "If you have PyTorch 2.0 installed, you shouldn’t use xFormers!"

The message is confusing - I have torch 2.0 installed and pipeline is for inference. This message doesn't occur if I use `AutoModelForSequenceClassification.from_pretrained`

### Expected behavior

The documentation or the warning message are inconsistent.

Comments:
It looks like the `pipeline` is back to importing every model (this message comes from trying to access an unrelated model). I'll have a look later this week. You can ignore that warning in the meantime, it's irrelevant.

Should be fixed by the PR linked above.

same issue for me on this basic example:

```
import argparse
from transformers import pipeline

# Create the parser
parser = argparse.ArgumentParser(description="Perform sentiment analysis")

# Add an argument
parser.add_argument('Text', type=str, help="the text to analyze")

# Parse the argument
args = parser.parse_args()

# Load the classifier
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

# Perform sentiment analysis
res = classifier(args.Text)

# Print the result
print(res)
```

Reinstalled transformers, using v.4.31.0

The fix is not in v4.31.0, you will need to use a source install.
"Xformers is not installed correctly." error
Author: engageintellect
Pull request: no

### System Info

System info:
* Transformers 4.31.0
* Python 3.11.3

Error:
```
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
```

Code:
```
import argparse
from transformers import pipeline

# Create the parser
parser = argparse.ArgumentParser(description="Perform sentiment analysis")

# Add an argument
parser.add_argument('Text', type=str, help="the text to analyze")

# Parse the argument
args = parser.parse_args()

# Load the classifier
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

# Perform sentiment analysis
res = classifier(args.Text)

# Print the result
print(res)
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce this behavior:
1. run my code
2. receive error

### Expected behavior

no error

Comments:
This is a duplicate of #24903
reattach hooks when using `resize_token_embeddings`
Author: SunMarc
Pull request: yes

# What does this PR do ?
Solves #25554. 
This PR fix the case where one uses `resize_token_embeddings` on a model with that has been dispatched with `device_map`. We reattach the hook to the new modules. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Unable to import tokenizers
Author: jeffyjeff2893
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using python 3.9 on colab I'm unable to import anything from transformers
https://colab.research.google.com/drive/1KYBHdjLphk0L7ZFOPAaw6Xr-v5TuL55u?usp=sharing

### Expected behavior

I expect to be able to import AutoTokenizer

Comments:
Hey, you manually changed the path using `sys`, if you move your call to `transformers` above your code, it works as expected. 

After changing the python version I have to append the install location to sys path though or I get no module named transformers
Sorting FAISS scores for similarity search  
Author: NamburiSrinath
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.10.157-139.675.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.15
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger, @stevhliu, @MKhalusova 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import pandas as pd

samples_df = pd.DataFrame.from_dict(samples)
samples_df["scores"] = scores
samples_df.sort_values("scores", ascending=False, inplace=True)
```

Please refer [https://huggingface.co/learn/nlp-course/chapter5/6?fw=pt#using-faiss-for-efficient-similarity-search](https://huggingface.co/learn/nlp-course/chapter5/6?fw=pt#using-faiss-for-efficient-similarity-search)

### Expected behavior

I think the sorting of scores should be in ascending and not descending; because the default index is IndexFlatL2 which is L2/Euclidean distance.

It will be great if these two changes are made to relevant documentation

1. Change `ascending=False` to `ascending=True` 
2. There can be a reference that the default scores returned is the Euclidean distances (by digging sourcecode, I understood it's IndexFlatL2), but it will be easy to include this in documentation

**Refer:** [https://discuss.huggingface.co/t/chapter-5-questions/11744/58?u=namburisrinath](https://discuss.huggingface.co/t/chapter-5-questions/11744/58?u=namburisrinath)

**P.S:** I am sorry if this is the correct place to create the bug as the documentation needs to be changed accordingly. People consume Huggingface documentation a lot, so it needs to be fool-proof, so please correct if I am wrong! 

Comments:
Hi! Thank you for opening the issue.

I tag someone in the team on the course chapter discussion page. Let's wait a reply first 🤗 .

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TF Whisper export: Can't use caching?
Author: DevinTDHa
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-6.2.6-76060206-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help?

@sanch @gante @Rocketknight1 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Colab Notebook:

https://colab.research.google.com/drive/1AZRoVs5aK9kM2nKNZ6D98ubJOCGOnQep?usp=sharing

### Expected behavior

Hi!

I am working on deploying a Whisper model and want to export it as a TF saved model and I am having issues with it.

The default export behavior for the saved model will result in a serving that does not separate the encoder and the decoder. I wanted to do this manually, for which I wrote a class (see the colab notebook.).

The custom class separates the decoder and encoder to different servings. Additionally, tensors for caching (`past_key_values`) are also split and ordered by decoder and encoder, so it can be traced.

However, when trying to export the decoder/encoder states for caching, an error occurs. 

```python
        File "python3.8/site-packages/transformers/models/whisper/modeling_tf_whisper.py", line 99, in _make_causal_mask  *
            if past_key_values_length > 0:
    
        TypeError: '>' not supported between instances of 'NoneType' and 'int'
```

I am not sure, but I checked the code and I suspect that this error is due to the cache tensors having different sizes (for encoder/decoder). As we pass the cache tensors as a tuple to the model, there should be multiple sizes for these cache tensors (generated sequence length for the decoder, `1500` for the encoder). I am not sure, how else I could define the servings definition to cover this case.

Thanks in advance!





Comments:
Good spot - this is actually a bug in the TF-Whisper code! I've opened a PR to fix it at #25523.

@DevinTDHa The fix has been merged! You can try it by installing from main with `pip install --upgrade https://github.com/huggingface/transformers.git`. It fixes the issue for me when I run your demo notebook, but please let me know if you encounter any other issues with TF-Whisper.

Thanks again for the bug report and the clean reproduction notebook!

@Rocketknight1 Thanks for the very quick fix, it works!
different results shown each time when I run the example code for BertForMultipleChoice
Author: yuchuang1979
Pull request: no
Labels: name: wontfix

When I ran the following example provided for BertForMultipleChoice in the documentation, I've got different results each time when I run it. Does it mean that BertForMultipleChoice is only provided to fine-tune the BERT model with RocStories/SWAG like datasets, and no pretrained models (after the fine-tuning) are provided?

________________________
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertConfig
from pytorch_transformers import BertForNextSentencePrediction, BertForMultipleChoice
import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMultipleChoice.from_pretrained('bert-base-uncased')
model.eval()
choices = ["Hello, my dog is cute", "Hello, my cat is amazing"]
input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
labels = torch.tensor(1).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, classification_scores = outputs[:2]
print(classification_scores)

Comments:
Yes!

You have to fine-tune BertForMultipleChoice to be able to use it.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Does Bert add extra hidden layers that are randomly initialized on top of the pre trained network when using BertForMultipleChoice? Would these added hidden layers be the only hidden layers that are adjusted during the learning process?

> Does Bert add extra hidden layers that are randomly initialized on top of the pre trained network when using BertForMultipleChoice? Would these added hidden layers be the only hidden layers that are adjusted during the learning process?

Overall, the parameters of pretrained mode starts from saved states, and the parameters of the classifier are initialize randomly as it can be checked from transformers package. During the fine tuning all of the parameters are fine tuned.
Trn1 LoRA finetuning with HF reaches RuntimeError: Invalid device format: cpu
Author: kct22aws
Pull request: no

### System Info

optimum 1.9.0
optimum-neuron 0.0.7
transformers 4.30.2
DLAMI: https://aws.amazon.com/marketplace/pp/prodview-gr3e6yiscria2
AWS instance: trn1.2xl

`
  File "<string>", line 111, in __init__
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 1341, in __post_init__
    and (get_xla_device_type(self.device) != "GPU")
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 127, in get_xla_device_type
    return xm.xla_real_devices([device])[0].split(":")[0]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 268, in xla_real_devices
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 268, in <listcomp>
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 263, in _xla_real_device
    raise RuntimeError('Invalid device format: {}'.format(device_str))
RuntimeError: Invalid device format: cpu
`

Python training script attached as txt format
[train-vit.txt](https://github.com/huggingface/transformers/files/12034074/train-vit.txt)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Create an EC2 Trn1 instance with [Hugging Face DLAMI ](https://aws.amazon.com/marketplace/pp/prodview-gr3e6yiscria2)
2. `pip install peft`
3. install optimum-neuron form source: `pip install git+https://github.com/huggingface/optimum-neuron.git`
4. run python3 train-vit.py (script attached)

### Expected behavior

To run training script to completion.

Comments:
cc @pacman100 

Issue reported to https://github.com/huggingface/optimum-neuron/issues/134

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Best aproach to fine tune a GPT model for feature extraction 
Author: Luke-4
Pull request: no

Hi All,

I am trying to use BioGPT as a feature encoder and I want to compare if fine-tuning is going to improve the quality of the embeddings.

So I have two options the first is to fine-tune BioGPT without passing the labels and then use the last token of the last hidden state for classification using a separate machine-learning model. (Is it possible to fine-tune BioGPT as an encoder with the labels? Do the labels make any difference since the model is not attempting to classify?)

The second option would be to use BioGptForSequenceClassification which has a sequence classification head on top (linear layer) and fine-tune this by passing the labels to the model, I can then use this fine-tuned model for the classification or use the last token of the last hidden state for classification using a separate machine learning classifier.

Comments:
Hi! This is better on [HF forums](https://huggingface.co/).

This github repository is mainly for issues and feature requests 🙏 Thank you for your comprehension.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.