noamaanMulla-03/transformers-issues huggingface dataset:
Fix stale bot for locked issues
Author: LysandreJik
Pull request: yes

The stalebot has crashed once again due to the following error:

```
Traceback (most recent call last):
  File "scripts/stale.py", line 67, in <module>
    main()
  File "scripts/stale.py", line 57, in main
    issue.create_comment(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Issue.py", line 290, in create_comment
    headers, data = self._requester.requestJsonAndCheck("POST", f"{self.url}/comments", input=post_parameters)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 494, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers, input, self.__customConnection(url)))
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 629, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 726, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 760, in __requestRaw
    response = cnx.getresponse()
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 174, in getresponse
    r = verb(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/urllib3/connectionpool.py", line 931, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/GithubRetry.py", line 179, in increment
    raise Requester.createException(response.status, response.headers, content)  # type: ignore
github.GithubException.GithubException: 403 {"message": "Unable to create comment because issue is locked.", "documentation_url": "https://docs.github.com/articles/locking-conversations/"}
```

I believe it's the first time it's encountering a locked issue, hence the failure. I tested locally that it ran fine on this specific issue.


Comments:
Just the locked issues, I believe there's a single one here but unsure
Resume from checkpoint functionality of the Pytorch example has some bugs and need to fixed.
Author: MingxuanZhangPurdue
Pull request: no

### System Info

This example, i.e., [transformers/examples/pytorch/text-classification/run_glue_no_trainer.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py) has some bugs related to resume from check point, to be more specific, below are the original code adopted from line 520-521 and line 530-534

```
accelerator.print(f"Resumed from checkpoint: {checkpoint_path}")
accelerator.load_state(path)
```


```
# need to multiply `gradient_accumulation_steps` to reflect real steps
resume_step = int(training_difference.replace("step_", "")) * args.gradient_accumulation_steps
starting_epoch = resume_step // len(train_dataloader)
resume_step -= starting_epoch * len(train_dataloader)
completed_steps = resume_step // args.gradient_accumulation_step
```

However, it should be,

```
accelerator.print(f"Resumed from checkpoint: {checkpoint_path}")
accelerator.load_state(checkpoint_path)
```

where we  should load from ``checkpoint_path`` instead of ``path``, and

```
# need to multiply `gradient_accumulation_steps` to reflect real steps
resume_step = int(training_difference.replace("step_", "")) * args.gradient_accumulation_steps
starting_epoch = resume_step // len(train_dataloader)
completed_steps = resume_step // args.gradient_accumulation_steps
resume_step -= starting_epoch * len(train_dataloader)
```

we should move ``completed_steps = resume_step // args.gradient_accumulation_steps`` up,  before we change the ``resume_step``.


### Who can help?

@ArthurZucker @younesbelkada

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See above

### Expected behavior

I have provided the bugs and how to fix them, thanks a lot!

Comments:
This looks like an accelerate problem rather than a `transformers` problem! Could you take a look @muellerzr @pacman100 ?

The latter of which is already done, but I'll open a PR to fix the first part of printing the right path
Fix checkpoint path in `no_trainer` scripts
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR changes the print statement to print the right path in the `no_trainer` scripts when loading in a checkpoint

Fixes # (issue)

Solves https://github.com/huggingface/transformers/issues/25998

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
SwinModel docstring fix
Author: shivanandmn
Pull request: yes

# What does this PR do?
Docstring for `SwinModel` is added.

Fixes # (issue)


## Before submitting
- [x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. https://github.com/huggingface/transformers/issues/26638
- [ x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ydshieh 


Comments:
@ydshieh resolved merge conflicts - ready for ✅ Thanks!

@ydshieh 
I have updated changes. This is my first open-source contribution in the life.
I happy to learn from community.
Thanks 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26679). All of your documentation changes will be reflected on that endpoint.
Tranformers Entire documentation translation to Japanese 🇯🇵
Author: rajveer43
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the Japanese-speaking community 🌐 

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).


Comments:
Translated the accelerate.md file of the documentation to Chinese
Author: liteli1987gmail
Pull request: yes

What does this PR do?

🍅Add zh (Chinese) translation  accelerate.md
😊Add accelerate to _toctree.yml

- issue Transformers documentation translation to Chinese (Simplified) #20095

Who can review?
Documentation: @stevhliu and @MKhalusova
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26161). All of your documentation changes will be reflected on that endpoint.

Hi @liteli1987gmail, hope you don't mind me committing Arthur's suggestion for you so we can merge this PR! Thanks again for the translation :)
BLOOM past_key_values issue
Author: Kowsher
Pull request: no

### System Info

I used a custom plast_key_values by following this description
https://huggingface.co/docs/transformers/model_doc/bloom#transformers.BloomModel.forward.past_key_values

for example for batch size 2 and seq  length 3
i get plast_key_values[0][0] shape as orch.Size([32, 128, 3])  and plast_key_values[0][1] as torch.Size([32, 3, 128])

But getting this error
RuntimeError: The expanded size of the tensor (4) must match the existing size (8) at non-singleton dimension 2.  Target sizes: [32, 1, 4].  Tensor sizes: [32, 1, 8]

I'm using transformers==4.34.0 in google colab

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

N/A

### Expected behavior

N/A

Comments:
[docstring] Fix docstring for `LlamaTokenizer` and `LlamaTokenizerFast`
Author: minhoryang
Pull request: yes

# What does this PR do?

Fixes #26638 only for `LlamaTokenizer` and `LlamaTokenizerFast`.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26669). All of your documentation changes will be reflected on that endpoint.

@ydshieh ready for ✅ thanks!

@LysandreJik could you assign this to @ydshieh? Thank you.

`$ python3 -m pytest -v --make-reports doc_tests_gpu --doctest-modules src/transformers/models/llama/tokenization_llama_fast.py -sv`
> **[PASSED src/transformers/models/llama/tokenization_llama_fast.py::transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast](https://app.circleci.com/pipelines/github/huggingface/transformers/75029/workflows/2bc31261-2db7-4bb0-a657-fe0e69138af8/jobs/951332/artifacts)**
[i18n-KO] Translate `tokenizer.md` to Korean
Author: Sunmin0520
Pull request: yes

Translated the `tokenizer.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Lora adapter weights not getting loaded
Author: jaideep11061982
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.30.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
mymodel = AutoModelForMultipleChoice.from_pretrained(model_path  ) # original model
mymodel=PeftModel.from_pretrained(mymodel,'/kaggle/input/llm-best/best_ckp1') # best weights
```
`trainable params: 2050 || all params: 435850242 || trainable%: 0.00`

when I do get CV it gets me 
getScore(trainer,test_df_val,tokenized_test_ds_val)

i get No trained results as if not trained..

### Expected behavior

i should get same score as i got during the training followed by inferenec when it ends up with loading the best weights as I enable best load at the end  of training  option

Comments:
Hey, could you explain your problem properly? We need a full reproducer and traceback to be able to help you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[i18n-KO] Translate `pipelines.md` to Korean
Author: Sunmin0520
Pull request: yes

# What does this PR do?

Translated the `pipelines.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
 @bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
May you please review this PR? @sgugger, @ArthurZucker, @eunseojo

Comments:
@0525hhgus 님, 혹시 위의 CI 테스트에서 첫번째 에러 부분에 대해 확인해주실 수 있을까요? 
에러 메세지를 보면 
`ValueError: There was an error when converting ../transformers/docs/source/ko/main_classes/pipelines.md to the MDX format.
Unable to find 파이프라인 in transformers. Make sure the path to that object is correct.` 라고 나옵니다.
API 문서에 대해서는 toctree.yml에 작성해야하는 방법이 조금 다른걸까요..?

> @0525hhgus 님, 혹시 위의 CI 테스트에서 첫번째 에러 부분에 대해 확인해주실 수 있을까요? 에러 메세지를 보면 `ValueError: There was an error when converting ../transformers/docs/source/ko/main_classes/pipelines.md to the MDX format. Unable to find 파이프라인 in transformers. Make sure the path to that object is correct.` 라고 나옵니다. API 문서에 대해서는 toctree.yml에 작성해야하는 방법이 조금 다른걸까요..?

안녕하세요, @Sunmin0520 님!
pipelines.md 문서가 main_classes 폴더 아래 있다는 것이 나타나지 않아서 발생한 것으로 추정됩니다.
_toctree.yml에서 209 라인의 `pipelines`를 `main_classes/pipelines`로 수정하시면 될 것 같습니다! :smile: 

@0525hhgus 님, 감사합니다! 알려주신 것처럼 수정 후 문제없이 CI 테스트 완료된 것 확인했습니다 :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Padding is included in the computation of loss
Author: guyang3532
Pull request: no

### Description

I found the padding part in the logits (which is from embedding padding) is included in computation of loss in question answering bert:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L1876

Is it reasonable?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Reproduced with:
`torchrun --nproc_per_node=1 examples/onnxruntime/training/question-answering/run_qa.py --model_name_or_path deepset/minilm-uncased-squad2 --do_train --do_eval --per_device_train_batch_size 16 --num_train_epochs 3 --overwrite_output_dir --output_dir ./outputs/ --per_device_eval_batch_size 16 --seed 1337 --fp16 False --skip_memory_metrics False --dataset_name squad_v2 --learning_rate 2e-5 --remove_unused_columns False --version_2_with_negative --optim adamw_ort_fused --report_to none`

### Expected behavior

Is it more reasonable to pass only valid sequence length part of logits to loss compuation with something like:
`start_loss = loss_fct(start_logits[:valid_sequence_length], start_positions)`

Comments:
Hey! You should ask this question on [the forum](https://discuss.huggingface.co/), as this is more related to your understanding of the code 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Ko perf train gpu one
Author: HongB1
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다! -->
# What does this PR do?

Translated the `<your_file>.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [ ] Check for missing / redundant translations (번역 누락/중복 검사)
- [ ] Grammar Check (맞춤법 검사)
- [ ] Review or Add new terms to glossary (용어 확인 및 추가)
- [ ] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->
<!-- May you please review this PR? @keonju2 @harheem @junejae @wonhyeongseo  ... -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->

Comments:
번역 문서가 새롭게 업데이트 되었습니다! 아직 번역이 완료되지 않으셨다면, 문서 업데이트 부탁드립니다 😄 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.