noamaanMulla-03/transformers-issues huggingface dataset:
🌐 [i18n-KO] Translated `add_tensorflow_model.md` to Korean
Author: keonju2
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `add_tensorflow_model.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `your_file.md` file of the documentation to Korean 😄 
Thank you in advance for your review!

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 OSSCA 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- Team OSSCA, may you please review this PR?  -->
<!-- @wonhyeongseo, @keonju2, @harheem, @HongB1, @junejae -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? -->
<!-- @sgugger, @ArthurZucker, @eunseojo -->

Comments:
🌐 [i18n-KO] Translated `add_tensorflow_model.md` to Korean
Author: keonju2
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `add_tensorflow_model.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `your_file.md` file of the documentation to Korean 😄 
Thank you in advance for your review!

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@keonju2
<!-- 1. 위 체크가 모두 완료된 뒤에만 OSSCA 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- Team OSSCA, may you please review this PR?  -->
<!-- @wonhyeongseo, @keonju2, @harheem, @HongB1, @junejae -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? -->
<!-- @sgugger, @ArthurZucker, @eunseojo -->

Comments:
🌐 [i18n-KO] Translated `add_tensorflow_model.md` to Korean
Author: keonju2
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `add_tensorflow_model.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `add_tensorflow_model.md` file of the documentation to Korean 😄 
Thank you in advance for your review!

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@keonju2
<!-- 1. 위 체크가 모두 완료된 뒤에만 OSSCA 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- Team OSSCA, may you please review this PR?  -->
<!-- @wonhyeongseo, @keonju2, @harheem, @HongB1, @junejae -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? -->
<!-- @sgugger, @ArthurZucker, @eunseojo -->

Comments:
DFFT
Author: soma2000-lang
Pull request: yes

 #18004

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22952). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Working

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Working


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
uploading saffu language model to the transformers library
Author: haoranzhao419
Pull request: yes

# What does this PR do?

In this PR, we upload a new language model **SAFFU** (Self-Attention-Feed-Forward-Unit) to the HuggingFace transformers library. This model is super efficient to use and fast to train that we derive a super efficient way of computing the self-attention matrix with an explicit mathematical solution.


Comments:
Need to fix some issues of the code
PoC for a ProcessorMixin class
Author: sgugger
Pull request: yes

# What does this PR do?

This PR refactors some common saving/loading functionality of `Processor`s behind a `ProcessorMixin` class which implements the `save_pretrained` and `from_pretrained` method.

It's designed to work with main attributes (by default `feature_extractor` and `tokenizer`, but that list can be changed or extended in subclasses) that are then required by the init and on which `save_pretrained`/`from_pretrained` is called when saving or loading the processor.

It also handles having several classes for a given tokenizer (fast/not fast) so we can have that feature automatically handled. I showed how this works in refactoring two processors (I did not put the tokenizer fast in CLIP because there are some problems with it, but adding it is as easy as changing the `tokenizer_class`).
I will refactor the other processors if the design suits everyone.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
🌐 [i18n-KO] Translated `perplexity.mdx` to Korean
Author: HanNayeoniee
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated `perplexity.mdx` file of the documentation to Korean.
Added draft of `model_summary.mdx` file because it's referenced.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sgugger, @ArthurZucker, @eunseojo May you please review this PR? 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

좋은 번역 감사합니다! 앞서 올려주신 수정 제안 이외에 추가 의견 없습니다!

@sgugger, @ArthurZucker, @eunseojo May you please review this PR?
Does Fine tuning of already fine tuned model forgets the previous features like weights and bias?
Author: akesh1235
Pull request: no

Is it possible for fine tuning the already fine tuned model without losing previous features??

Suppose i fine-tune a model on "squad" dataset then I want to Incremental fine-tune the same model on some other dataset having same/different data formate and hyperparameters, does this means now model is fine-tuned on 2 datasets??? or it forgets the "squad" dataset when I fine tune on the second dataset.

Please acknowledge me over this
@vanpelt 
@pvl 
@arfon 

Comments:
Hi @akesh1235, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Thankyou @amyeroberts mam , 
I have posted on forum [Link for My topic on forum ](https://discuss.huggingface.co/t/fine-tuning-the-existing-fine-tuned-model/43113?u=akesh1235) 

Please acknowledge me over this
@vanpelt
@pvl
@arfon

@akesh1235 Great, thank you. 

In future, please only @ relevant people in issues (transformers topics and the HF maintainers to ask are listed in our issues template). The people you tagged are incredibly busy people and if everyone did this it would be impossible for anyone to meaningfully address issues, PRs and notifications on github. 

Okay mam I apologize,
I'll take care of this next time



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
LION optimizer calling error
Author: lucasjinreal
Pull request: no

TypeError: Lion.__init__() got an unexpected keyword argument 'is_paged'

Comments:
As said previously, there is nothing we can do if you do not follow the issue template and post a code reproducer.

@sgugger hello, this is what I done:

```
trainer = transformers.Trainer(
        model=model,
        train_dataset=train_data,
        eval_dataset=val_data,
        args=transformers.TrainingArguments(
            deepspeed=deepspeed,
            per_device_train_batch_size=micro_batch_size,
            gradient_accumulation_steps=gradient_accumulation_steps,
            warmup_ratio=0.1,
            num_train_epochs=num_epochs,
            learning_rate=learning_rate,
            # fp16=True,
            fp16=not int8_train,
            logging_steps=10,
            # optim="adamw_torch",
            optim="paged_lion_32bit",
            evaluation_strategy="steps" if val_set_size > 0 else "no",
            save_strategy="steps",
            eval_steps=50 if val_set_size > 0 else None,
            save_steps=50,
            output_dir=output_dir,
            save_total_limit=5,
            load_best_model_at_end=True if val_set_size > 0 else False,
            ddp_find_unused_parameters=False if ddp else None,
            group_by_length=group_by_length,
            report_to="wandb" if use_wandb else None,
            run_name=wandb_run_name if use_wandb else None,
        ),
        data_collator=transformers.DataCollatorForSeq2Seq(
            tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True
        ),
    )
```

the newly commited LION seems can not load properly, please have a test

This is not something I can reproduce as many of the objects you use are undefined.

I believe this is because of you're using previous version of `bitsandbytes`. In the latest version, there's an additional arguments called `is_paged`

https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L9

@louisowen6  Hi, does there any minimal examples on how to enable LION optimizer with llama along with training with deepspeed?
Just can't found such a detailed describe on this, it could be better if a experiment result compare with AdamW and LION on llama model

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FLAVA tensors are masked twice, forward pass fails
Author: amariucaitheodor
Pull request: no

https://github.com/huggingface/transformers/blob/cf11493dce0a1d22446efe0d6c4ade02fd928e50/src/transformers/models/flava/modeling_flava.py#L1965

The line above is the second time this tensor is masked if the previous ITM logic happens (line 1950), resulting e.g. `IndexError: The shape of the mask [14] at index 0 does not match the shape of the indexed tensor [13, 196, 768] at index 0`

The fix could be something like `if pos_mask is not None and sequence_for_image.size(0) == pos_mask.size(0)` on line 1964.

Comments:
A similar thing happens at line 1969 because of line 1956 (`mim_labels` changes shape):
```python
    mim_labels[bool_masked_pos.ne(True)] = self.ce_ignore_index
IndexError: The shape of the mask [14, 196] at index 0 does not match the shape of the indexed tensor [13, 196] at index 0
```

The fix could be adding `bool_masked_pos = bool_masked_pos[pos_mask]` between lines 1969 and 1968.

Same for line 1988, fix could be `if pos_mask is not None and sequence_for_text.size(0) == pos_mask.size(0):`

Tried the fixes and FLAVA runs. I wonder how no one else noticed 🤔

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Metas MMS speech recognition
Author: LYPinASR
Pull request: no

### Feature request

Our request a simpler and more convenient inference process for a speech recognition model based on MMS just like wav2vec 2.0 in Transformers. 

### Motivation

We aim to encapsulate the various subroutines called by Facebook’s official model into a direct speech recognition model that is as easy to use as other transformer-based models like wav2vec 2.0. But we also know that the Hugging face team has been among the industry leaders in this area of work.

### Your contribution

We recognize that it may not be feasible for us to directly assist the Hugging Face technical team in this task. We believe that such an effort would be forward-looking given the popularity of MMS in current speech recognition research. The resulting model would be ideal for quickly transcribing our meeting notes.


Comments:
Duplicate of #23665

@NielsRogge 
Hi, Can you tell me how far along we are and about how long it will be ready for us to use? Thank you!



PR merged.

Also see:
- https://huggingface.co/docs/transformers/main/en/model_doc/mms
- https://github.com/huggingface/transformers/pull/23813
- https://huggingface.co/facebook/mms-1b-all

> PR merged.
> 
> Also see:
> 
> * https://huggingface.co/docs/transformers/main/en/model_doc/mms
> * [[MMS] Scaling Speech Technology to 1,000+ Languages | Add attention adapter to Wav2Vec2 #23813](https://github.com/huggingface/transformers/pull/23813)
> * https://huggingface.co/facebook/mms-1b-all

Can I use my own dataset instead of the dataset "mozilla_foundation_common voice_6.1", which you have shown in tutorial [https://huggingface.co/blog/mms_adapters](url) ? If so then , how. Thanks

Sure, you just need to load your own dataset, maybe this helps: https://huggingface.co/docs/datasets/v2.13.1/en/audio_load

> Sure, you just need to load your own dataset, maybe this helps: https://huggingface.co/docs/datasets/v2.13.1/en/audio_load
Thank you for your kind reply, although I have gone through the suggested tutorial but it didn't help, and ...
I have just uploaded some demo dataset here u can check it: [https://huggingface.co/datasets/rashmi035/MKB_Hindi_2023](url) ,As you can see the audio is visible in the dataset viewer but the corresponding ngram is not visible, can you help me with this?
[check_config_docstrings.py] improve diagnostics
Author: stas00
Pull request: yes

It's difficult to know what to do when one gets an error:

```
python utils/check_config_docstrings.py

Traceback (most recent call last):
  File "utils/check_config_docstrings.py", line 92, in <module>
    check_config_docstrings_have_checkpoints()
  File "utils/check_config_docstrings.py", line 88, in check_config_docstrings_have_checkpoints
    raise ValueError(f"The following configurations don't contain any valid checkpoint:\n{message}")
ValueError: The following configurations don't contain any valid checkpoint:
IdeficsConfig

Exited with code exit status 1
```

After figuring out what it wants, proposing a better assert message.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Calling the tokenizer modifies the tokenizer object
Author: vikigenius
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-6.1.31_1-x86_64-with-glibc2.36
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Very simple reproduction here

```python
from transformers import AutoTokenizer
from datasets.utils.py_utils import dumps # Huggingface datasets

t = AutoTokenizer.from_pretrained('bert-base-uncased')
t.save_pretrained("tok1")
th1 = hash(dumps(t))
text = "This is an example text"
ttext = t(text, max_length=512, padding="max_length", truncation=True)
t.save_pretrained("tok2")
th2 = hash(dumps(t))

assert th1 == th2 # Assertion Error
```

The actual difference can be found if you try to save the tokenizer after calling it. Diff the tokenizer.json and you can see that the keys "padding" and "truncation" got updated.

`diff tok1/tokenizer.json tok2/tokenizer.json` produces an actual diff

### Expected behavior

The tokenizer object should not change just because it was called with the padding parameters.

Comments:
Most likely the issue lies in Fast Tokenizers here:

https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/tokenization_utils_fast.py#L319-L388

I don't actually see anyplace where the original strategy is restored.

Because this snippet

```python
from transformers import AutoTokenizer

t = AutoTokenizer.from_pretrained('bert-base-uncased')
p1 = t._tokenizer.padding
text = "This is an example text"
ttext = t(text, max_length=256, padding="max_length", truncation=True)
p2 = t._tokenizer.padding
```

p1 and p2 are different. i.e the value of padding changes.


Hey! This is actually expected. When you save a tokenizer, the `init_kwargs` that were last used are saved along. 
If you initialize the model with `from_slow = True`, then this will be saved. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.