noamaanMulla-03/transformers-issues huggingface dataset:
This line prevent us from using "std" scaling any more.
Author: zhentao-xu
Pull request: no

https://github.com/huggingface/transformers/blob/b29e2dcaff114762e65eaea739ba1076fc5d1c84/src/transformers/models/time_series_transformer/modeling_time_series_transformer.py#L1549

Comments:
Hi,

Std scaling wasn't supported until #21020 was merged (only mean scaling is currently supported on the latest PyPi install). So if you install Transformers from source, you can use std scaling.

Cc @kashif 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@zhentaoxuttup were you able to use "std" scaling?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Unable to convert BioGpt slow tokenizer to fast: token out of vocabulary
Author: seantaud
Pull request: no

### System Info

I was trying to use BioGpt model in my QA task for fine-tuning. I would like to construct a fast tokenizer class based on the BioGptTokenizer, so that I could use the offsets_mapping to know from which words the tokens do origin. But unfortunately, when creating a BiogptTokenizerFast from the PreTrainedTokenizerFast by `convert_slow_tokenizer`,  following error occurs:   Error while initializing BPE: Token `-@</w>` out of vocabulary. 

#### Error trace
```
Traceback (most recent call last):
  File "run.py", line 124, in <module>
    trainer, predict_dataset = get_trainer(args)
  File "***/tasks/qa/get_trainer.py", line 31, in get_trainer
    tokenizer = BioGptTokenizerFast.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 1801, in from_pretrained
    return cls._from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 1956, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "***/model/biogpt/tokenization_biogpt_fast.py", line 117, in __init__
    super().__init__(
  File "***/model/biogpt/tokenization_utils_fast.py", line 114, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "***/model/biogpt/convert_slow_tokenizer.py", line 1198, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "***/model/biogpt/convert_slow_tokenizer.py", line 273, in converted
    BPE(
Exception: Error while initializing BPE: Token `-@</w>` out of vocabulary
```

### Who can help?

@ArthurZucker @younesbelkada @kamalkraj 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I copy the code related to colab.This is the link : https://colab.research.google.com/drive/1IMhiDz45GiarBLgXG9B2rA_u0ZOmmjJS?usp=sharing

### Expected behavior

According to this issue [https://github.com/huggingface/transformers/issues/9290](url), this problem might be caused by some missing tokens in `vocab.json` or `merge.txt`. Could you please check it? Thank you very much!

Comments:
Hey! It would be a bit difficult to say that this is a bug as we do not have an implementation of the `BioGPTConverter` which would be cool btw. In order to properly create a fast tokenizer you need to have the `pre_normalizer` the `decoder` etc. Take a look at `convert_slow_tokenizers` for more details! Feel free to open a PR and ping me 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix ConvNext V2 parameter naming issue
Author: alaradirik
Pull request: yes

# What does this PR do?
Renames gamma and beta parameters of the `ConvNextV2GRN` module, which caused the `save_pretrained` method to rename these parameters to weight and bias.

Existing checkpoints on the hub can be loaded without any warnings once the PR is merged.

Fixes #23090

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._
[doc] Try a few ≠ ways of linking to Papers, users, and org profiles
Author: julien-c
Pull request: yes

would love to hear others' thoughts.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

for some reason i can't see the model pages in the PR's generated doc

I can't preview the doc for some reason (getting an error on the [distilbert preview page](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22611/en/model_doc/distilbert)). Also there seems to be an issue with your CircleCI permissions, the tests won't run.
Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

Otherwise the changes look good to me in preview. Would just love to preview the badge for papers!

Weird, the doc build ran successfully and uploaded a zip file, but it does not contain the modeling files. Will see what's up in a bit.

It's correctly displayed on the docs now and on the link you shared @sgugger, there was an error with the backend sync.

ok i kinda like it, WDYT?

<img width="1160" alt="image" src="https://user-images.githubusercontent.com/326577/230628465-f3850483-8338-4956-a969-f4f9ffe6b3ea.png">

Link: https://moon-ci-docs.huggingface.co/docs/transformers/pr_22611/en/model_doc/t5
Add focalnet backbone
Author: alaradirik
Pull request: yes

# What does this PR do?
Adds `FocalNetBackbone` class to be used by X-Decoder and possibly other frameworks as FocalNet was published fairly recently.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Tidy Pytorch GLUE benchmark example
Author: tlby
Pull request: yes

Migration to Evaluate for metric is not quite complete

# What does this PR do?

#18369 left the Pytorch GLUE Benchmark example a bit rough, still hand implementing some metrics, and leaving metric objects constructed but unused in some cases.  This work completes the migration to Evaluate.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @atturaioe 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
GPTNeoForQuestionAnswering
Author: peter-sk
Pull request: yes

# What does this PR do?

Adds QA support for GPT Neo.

Includes PR #23030.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@ArthurZucker
@younesbelkada


Comments:
_The documentation is not available anymore as the PR was closed or merged._

One more to go (for now).

@sgugger @younesbelkada  - as Arthur is on holdiays 👍 

@younesbelkada 
I merged with main to isolate the GPT Neo specific parts. Now everything seems to work fine.

rebased - let's see whether it helps

@younesbelkada merging helped - ready to merge 👍
🌐 [i18n-KO] Translated `tasks/question_answering.mdx` to Korean
Author: jungnerd
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `tasks/question_answering.mdx` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sgugger, @ArthurZucker, @eunseojo May you please review this PR?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

May you please review this PR? 😄
@sgugger, @ArthurZucker, @eunseojo
num_noise_spans should be <= num_items #22246
Author: alexcpn
Pull request: yes

# What does this PR do?



<!-- Remove if not applicable -->

Fixes #22246

When `mean_noise_span_length` is set to 1  there are cases (for example `noise_density=.55` when the `num_noise_spans` becomes  greater than num_nonnoise_tokens

So the correction seems to be to consider also the `num_nonnoise_tokens` in calculation of `num_noise_spans`

num_noise_spans = int(np.round(min(num_noise_tokens,num_nonnoise_tokens) / self.mean_noise_span_length))

Demonstration of the buggy behaviour 
https://gist.github.com/alexcpn/b9bb2b0f01833d1bb862502faf99bab8#file-t5_denoising-py

Demonstration of the possible correction  
https://gist.github.com/alexcpn/b9bb2b0f01833d1bb862502faf99bab8#file-t5_denoising_corrected-py

## Who can review?

@sanchit-gandhi 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks- Done - Did `make style` and pushed the changes to this branch

Thanks @alexcpn - unfortunately the CI is still unhappy about the code style! Could you try rebasing onto main, run style fix, and then force pushing?
```
git rebase upstream/main
make style
git push -f issue-22246
```

I have rebased and checked and force-pushed. It is actually fine locally.

```
git branch
* issue-22246
  main
$ make style
black examples tests src utils setup.py
All done! ✨ 🍰 ✨
2380 files left unchanged.
ruff examples tests src utils setup.py --fix
make: ruff: No such file or directory
make: *** [Makefile:69: style] Error 127
```
Understood the problem - I was having black 22.x and CircleCI is using 23.x - https://github.com/huggingface/transformers/pull/21480

@sanchit-gandhi  CI is green

Awesome, nice find @alexcpn 🙌 Let's get a final review and get the PR merged!
resume checkpoint and continue training using deepspeed integration while changing the number of gpus
Author: hao-xyz
Pull request: no

### System Info

transformers version: 4.28.1
torch version: 1.13+cu116

Can trainer support resuming checkpoint while using different number of gpus?
I saw optimizer states are saved per rank while saving checkpoints, how can i resuming successfully while changing the numer of gpus ?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

trainer.train(resume_from_checkpoint=checkpoint) 

### Expected behavior

resuming successfully 

Comments:
Cannot resume FSDP optimizer state 
Author: qywu
Pull request: no

This line does not save optimizer state correctly when using FSDP.

https://github.com/huggingface/transformers/blob/88399476c3892435395618ed37993176dbb0de73/src/transformers/trainer.py#L2383

It should use FSDP's full_optim_state_dict to collect optimizer states from different processes.
```python
FSDP.full_optim_state_dict(self.model, self.optimizer)
```

Comments:
cc @pacman100 

Hello @qywu, indeed, that seems to be the case, as you already have the fix, it would be great if you could raise the PR with the fixes, Thank you!
gpt2 multi-gpu fix
Author: peter-sk
Pull request: yes

# What does this PR do?

Move tensors to same device.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@younesbelkada
@amyeroberts


Comments:
_The documentation is not available anymore as the PR was closed or merged._
GPTNeoXForQuestionAnswering
Author: peter-sk
Pull request: yes

# What does this PR do?

Adds GPTNeoXForQuestionAnswering.

Includes #23030 and #23057.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@ArthurZucker
@younesbelkada


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @younesbelkada as Arthur is on holdiays 👍 

@younesbelkada @amyeroberts this one is ready for review :-)
Fixed default config for `Pix2Struct` model to set `Pix2StructTextModel` to `is_decoder=True`
Author: gbarello-uipath
Pull request: yes

Previously, the `Pix2StructTextModel` was configured with `is_decoder=False` by default causing the attention mask used for self-attention to be non-causal and causing fine-tuning to fail.

As a fix, this PR adds `is_decoder=True` default kwarg to the `Pix2StructTextConfig` class in order to correctly configure the text model as a decoder. 

Fixes # 22903

@younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Just updated all pix2struct checkpoints that are under `google` org! Thanks again @gbarello-uipath for flagging this

Shouldn't the matcha and deplot checkpoints be updated as well?

Good point, will update them now

Just updated them! Thanks for flagging @RainbowMan1
[`GPT-J`] Fix causal mask dtype
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/23136

When going for `low_cpu_mem_usage` each parameter is force-casted to the expected dtype, which is force-set to `torch.float16` for 8bit models.

Therefore, for 8bit models (and also half-precision models) the causal mask is always force casted to float16 as it is part of the model's state dict, hence expected to be loaded from the Hub if the mask is available on the state dict.

The fix is to add `persistant=False` and add a field `_keys_to_ignore_on_unexpected` (for removing the warnings) to avoid loading that causal mask from the state dict and assign it to the buffer, and all causal masks that are saved as buffers should do the same to avoid unexpected behaviors.

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
ImportError: cannot import name 'hf_bucket_url' from 'transformers.file_utils'
Author: Andy824
Pull request: no

### System Info

`from transformers.file_utils import default_cache_path, hf_bucket_url`
I want to import hf_bucket_url on Colab, but I got the error
"ImportError: cannot import name 'hf_bucket_url' from 'transformers.file_utils' (/usr/local/lib/python3.9/dist-packages/transformers/file_utils.py)"

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

ImportError: cannot import name 'hf_bucket_url' from 'transformers.file_utils' (/usr/local/lib/python3.9/dist-packages/transformers/file_utils.py)

### Expected behavior

Please tell me am I doing something wrong?

Comments:
Yes, this function was removed several versions ago. It was only relevant for downloading files before the model Hub was properly setup. You should now use the `huggingface_hub` library to manage downloads of models from the Hub.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to use the method model.generate() correctly?
Author: zt991211
Pull request: no

### System Info

- `transformers` version: 4.19.4
- Platform: Linux-4.9.93-010.ali3000.alios7.x86_64-x86_64-with-redhat-7.2-Paladin
- Python version: 3.7.11
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

## This is my code
self.decoder.generate(inputs=A, do_sample=True,
                                                 max_length=80, min_length=1,
                                                 top_k=50, top_p=0.95)

## self.decoder here is the BART model and A here is the input_features not the input_ids.

## The official document says that for the model of the encoder-decoder architecture, the generate method can input input_features, but an error occurs. According to the error log, input_features input is not supported, and only input_ids can be used.

## error log: RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)

## I want to know whether my usage is wrong or there is a bug in the source code

### Expected behavior

I hope my code will return output_ids

Comments:
cc @gante 

@zt991211, thanks for raising an issue! 

Could you provide a more detailed snippet which is reproducible i.e. can be directly copied and run as well as a full traceback of the error encountered? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.