noamaanMulla-03/transformers-issues huggingface dataset:
[i18n-KO] Translated `big_models.md` to Korean
Author: bolizabeth
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다! -->
# What does this PR do?

Translated the `big_models.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [ ] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->
May you please review this PR? @hyunhp @nuatmochoi @heuristicwave @mjk0618 @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? @sgugger, @ArthurZucker, @eunseojo -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24985). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🌐 [i18n-KO] Translated perf_train_gpu_many.md to Korean
Author: hyunhp
Pull request: yes

# What does this PR do?

Translated the `perf_train_gpu_many.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

May you please review this PR? @nuatmochoi, @bolizabeth, @heuristicwave, @mjk0618, @jungnerd, @hyunhp

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? @sgugger, @ArthurZucker, @eunseojo -->

Comments:
Please do not open and close several PRs for the same file. You can update a PR by pushing more commits to it.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24983). All of your documentation changes will be reflected on that endpoint.

> Please do not open and close several PRs for the same file. You can update a PR by pushing more commits to it.

Duly noted

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🌐[i18n-KO] Translated `<debugging>.md`to Korean
Author: kj021
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `<debugging>.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @sronger, @TaeYupNoh, @kj021, @HanNayeoniee, @eenzeenee, @sim-so

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- @sgugger, @ArthurZucker, @eunseojo May you please review this PR? -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24869). All of your documentation changes will be reflected on that endpoint.

소현님이 리뷰를 꼼꼼하게 해주셨네요!
LGTM 👍

안녕하세요 @kj021 님, 혹시 시간이 나실때 위의 수정사항들을 반영해주시길 바랍니다! 감사합니다.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🌐 [i18n-KO] Translated `tokenizer_summary.md` to Korean
Author: HanNayeoniee
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `<tokenizer_summary>.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sronger, @TaeYupNoh, @kj021, @HanNayeoniee, @eenzeenee, @sim-so

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- @sgugger, @ArthurZucker, @eunseojo May you please review this PR? -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25023). All of your documentation changes will be reflected on that endpoint.

리뷰를 남기고 submit을 안 했었네요 .. ㅠ

> 리뷰를 남기고 submit을 안 했었네요 .. ㅠ

ㅋㅋㅋㅋㅋㅋ 다행히 지금 번역 수정중이라서 확인했습니다!

> 나연님 항상 알기 쉬운 말로 번역해주셔서 좋아요! 게다가 이번 문서에서 토크나이저를 쭉 둘러볼 수 있어서 유익했습니다 😊
> 
> 리뷰 하면서 glossary 관련한 수정 제안을 몇 가지 드렸습니다. 참고 부탁 드립니다!

제가 번역을 오랜만에 해서 그런지 glossary 관련 수정사항이 많군요.. 꼼꼼한 리뷰 감사합니다!!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Porting the torchaudio kaldi fbank implementation to audio_utils
Author: ylacombe
Pull request: yes

# What does this PR do?

> [Kaldi](https://kaldi-asr.org/doc/) is a toolkit for speech recognition, intended for use by speech recognition researchers and professionals. 

Its [torchaudio `fbank` implementation](https://pytorch.org/audio/stable/compliance.kaldi.html) is sometimes used in the library, notably in the [Audio Spectrogram Transformer](https://github.com/huggingface/transformers/blob/d70fab8b2062526e9c2c60196421a8bc96c7df03/src/transformers/models/audio_spectrogram_transformer/feature_extraction_audio_spectrogram_transformer.py#L97) and [SpeechToText](https://github.com/huggingface/transformers/blob/d70fab8b2062526e9c2c60196421a8bc96c7df03/src/transformers/models/speech_to_text/feature_extraction_speech_to_text.py#L90) models.

It will also be used in the future [SeamlessM4T model](https://github.com/huggingface/transformers/pull/25693#pullrequestreview-1624417597).

This PR aims to port [the implementation](https://pytorch.org/audio/stable/_modules/torchaudio/compliance/kaldi.html#fbank) to `numpy`, directly in `audio_utils.py`.

Why is this important? It will reduce `transformers` dependence on `torchaudio` for some models.

Atm, I enriched some of `audio_utils` methods and added tests to make sure it has the same results as `torchaudio`.



## Before submitting
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

cc @ArthurZucker and @sanchit-gandhi , what do you think of this feature ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I've applied your suggestions @sanchit-gandhi and corrected the `use_torchaudio_version` naming to `triangularize_in_mel_space` with a more appropriate description, thanks for the review!

@ArthurZucker, would it be possible to review this PR when you have time ? thanks!

Thanks for your review @ArthurZucker ! I'm gonna merge as soon as tests pass.

BTW, `np.exp(...,base=10)` doesn't work. I used `np.power(10,...)` instead :hugs:
[TTA Pipeline] Test MusicGen and VITS
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Add tiny-random pipeline tests for MusicGen and VITS

Comments:
_The documentation is not available anymore as the PR was closed or merged._

In order to make pipeline tests (those with tiny models) work, the tiny model should be in the file `tests/utils/tiny_model_summary.json`.

So far `MusicgenForConditionalGeneration` is on the Hub but not in this file. Same for `hf-internal-testing/tiny-random-VitsModel`.

I will update this file later. 

Thanks @ydshieh! Is there a place where the tiny pipeline tests are documented? Or a way we can run this check automatically? I wasn't aware we needed to add them to `tiny_model_summary.json` too!

Hi @sanchit-gandhi 

It is mentioned in [our internal notion doc](https://www.notion.so/huggingface2/Pipeline-Testing-in-transformers-87c5f26a2c3847ada94935878b77cc34#e2be051c13784bf8a459fd95674b1724)

> - The file [[**tests/utils/tiny_model_summary.json](https://github.com/huggingface/transformers/blob/main/tests/utils/tiny_model_summary.json)](https://github.com/huggingface/transformers/blob/main/tests/utils/tiny_model_summary.json)** needs to be updated with the information about the new tiny models created/uploaded in the previous step.

~ 1 year ago, we decided to make pipeline testing to use tiny models on `hf-internal-testing` (rather than creating them on the fly during testing). For some technical reasons, CI needs to access to `tiny_model_summary.json` which should be updated whenever some new architectures are available. Currently I check the results on a GitHub Action workflow once every 1 or 2 months and update this file.

Amazing! Thank you for the information 🤗
How does one preprocess a pdf file before passing it into pipeline for summarization?
Author: pythonvijay
Pull request: no

Hi,

I think I am unable to read the pdf file in the required format as it runs into an error when I try to read it through PdfReader

here is my code:

```
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# Download a file based on its file ID.
#
# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz
file_id = "1RjF9CRj8xTQakH5jex1ppBrTw1heHwPS"
downloaded = drive.CreateFile({'id': file_id})
```
it works upto this point...

next, I pass this - 

`print('Downloaded content "{}"'.format(downloaded.GetContentString()))`

and I get this error:

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 11: invalid start byte`

and when I try to read the variable 'downloaded' into PdfReader, I again get an error:

`reader = PdfReader(downloaded)`

`AttributeError: 'GoogleDriveFile' object has no attribute 'seek'`

Please advise what I need to correct to make this work!

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests, and less for custom code decoding. Seems like you are not even using transformers here. 
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This is a pre step to feed the PDF file into a huggingface hosted transformer **T5-base** for text summarization
Token classification example (run_ner.py) should work without fast tokenizers
Author: ruanchaves
Pull request: no

# 🚀 Feature request

The token classification example ( [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py) ) calls the Tokenizer with `return_offsets_mapping=True` (line 279).

This is not allowed for Python tokenizers and raises the error `NotImplementedError: return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.`

run_ner.py should align tokens and labels even if a fast tokenizer is not available.

## Motivation

There isn't a fast tokenizer available for `vinai/bertweet-base`, and I guess this may apply to a few other models as well.  
Passing  `vinai/bertweet-base` as `model_name_or_path` to `run_ner.py` instantly raises `NotImplementedError`.

Comments:
Yes, this script only supports models that have a fast tokenizer (there is now a clear assert of that after the tokenizer is loaded). The old script will work with models that only have a slow tokenizer.

> Yes, this script only supports models that have a fast tokenizer (there is no a clear assert of that after the tokenizer is loaded). The old script will work with models that only have a slow tokenizer.

Somebody wrote an assert four days ago. There are certain inconveniences to the old script, e.g. it doesn't utilize the `datasets` library. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I am facing same problem in BioGPT


I also have the same problem, has anyone found any solution for that? 

No solution found yet, I also have this error in BioGPT

Can you try `pip install tokenizers`?
[QUICK FIX LINK] Update trainer.py
Author: SoyGema
Pull request: yes

Fix link

# What does this PR do?


Fix broken link.
Change `https://huggingface.co/docs/transformers/model_doc/auto.` 
for `https://huggingface.co/docs/transformers/model_doc/auto`

<img width="1093" alt="Captura de pantalla 2023-09-20 a las 16 10 06" src="https://github.com/huggingface/transformers/assets/24204714/c1d801bf-691e-4591-9071-1e3584e26ada">





<!-- Remove if not applicable -->


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hey @ArthurZucker ! It seems the point character broke things
Now it sohuld work. 

indeed sorry haha! merging 😉 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26293). All of your documentation changes will be reflected on that endpoint.
Use CircleCI `store_test_results`
Author: ydshieh
Pull request: yes

# What does this PR do?

@ArthurZucker loves it.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Very cool! Thanks @ydshieh and @ArthurZucker, this looks cool!

> More than me loving it, it gives us [access to the slow tests, the flaky tests, the tests that usually fail](https://app.circleci.com/insights/github/huggingface/transformers/workflows/run_tests/tests?branch=make_arthur_eye_happy) etc. Let's add this to all CI jobs!

OK!

@ArthurZucker I think all pytest jobs have this feature enabled. `.circleci/config.yml` doesn't have pytest but just some usual python scripts. Could you elaborate a bit more what you suggest? Thanks!

I mean `check_repository_consistency`, `check_code_quality`, `pr_documentation_test` if possible! 

Hi,

`pr_documentation_test` is already included in the current change.

The other 2 jobs are not `pytest`, and I am not sure if there is anything we can do similar to `pytest --junitxml=test-results/junit.xml`

We could make them use pytest, but only if you think it’s relevant! 

IMO, those are not really tests but checks of formatting :-). They won't be flaky, and those could not be splitted into individual test **methods** but each script as a whole.

I would rather move forward and take the new feature already available (with this PR) for us even only for the pytest jobs.

Thank you for asking to add this, Sir!
Fix doctest CI
Author: ydshieh
Pull request: yes

# What does this PR do?

The daily doctest CI is killed due to memory issue (from `PersimmonForCausalLM.forward`'s docstring). It's not only about GPU, even running with the 60G CPU memory, the job is still killed.

If I  run that test only - it could pass (with running on CPU), but the process is killed when the whole suite is run. 
(There might be some memory (leak) issue to check as a whole).

This PR put `src/transformers/models/persimmon/modeling_persimmon.py` to `not_doctested.txt`.

We probably better to further separate what are not doctested yet and what are ignored intentionally, so we won't forget to try to put them back to doctests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._