noamaanMulla-03/transformers-issues huggingface dataset:
Inconsistency in encoding user special tokens between LlamaTokenizerFast and LlamaTokenizer
Author: hzphzp
Pull request: no

### System Info

tokenizers==0.13.3
transformers==4.32.1

### Who can help?

@ArthurZucker  When encoding user special tokens (e.g., \<img\>), LlamaTokenizerFast adds extra spaces before and after the token, while LlamaTokenizer does not. This inconsistent behavior leads to different results when using the two tokenizers.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

To reproduce the issue, I used the following code:
```python
from transformers import LlamaTokenizerFast  
from transformers import LlamaTokenizer  
  
fast_tokenizer = LlamaTokenizerFast.from_pretrained("lmsys/vicuna-13b-v1.3", use_fast=True)  
tokenizer = LlamaTokenizer.from_pretrained("lmsys/vicuna-13b-v1.3", use_fast=False)  
  
tokenizer.add_special_tokens({"additional_special_tokens": ['<img0>', '</img0>', '<img_content>']})  
fast_tokenizer.add_special_tokens({"additional_special_tokens": ['<img0>', '</img0>', '<img_content>']})  
  
prompt = "<img0> <img_content> </img0>"  
  
print(tokenizer.tokenize(prompt))  
print(fast_tokenizer.tokenize(prompt))  
print(tokenizer(prompt))  
print(fast_tokenizer(prompt))  
```

The results are as follows:
```
['<img0>', '<img_content>', '</img0>']  
['<img0>', '▁▁', '<img_content>', '▁▁', '</img0>']  
{'input_ids': [1, 32000, 32002, 32001], 'attention_mask': [1, 1, 1, 1]}  
{'input_ids': [1, 32000, 259, 32002, 259, 32001], 'attention_mask': [1, 1, 1, 1, 1, 1]}  
```

### Expected behavior

As you can see, LlamaTokenizerFast adds unnecessary spaces ('▁▁') before and after the <img_content> token, while LlamaTokenizer does not. This results in different input_ids and attention_mask values when using the two tokenizers. But we expect that they output the same.

Waiting for kindly response ... ...

Comments:
Hey! This is expected, a fix is on its way here: #23909. For now you should make sure the added tokens have `rstrip = True` and `lstrip=True` 😉 

problem solved, thank you!
Resolving Attribute error when using the FSDP ram efficient feature
Author: pacman100
Pull request: yes

# What does this PR do?
1. Currently, for few models getting Attribute errors such as `AttributeError: 'OPTForCausalLM' object has no attribute 'decoder'` when using the FSDP ram efficient feature. This PR fixes it. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix register
Author: zspo
Pull request: yes

# What does this PR do?

Fix register in some auto class.

Fixes  #25453 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @sgugger 



Comments:
Sure! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25779). All of your documentation changes will be reflected on that endpoint.
Local variable 'tokens' referenced before assignment error in tokenization_llama.py
Author: avnishn
Pull request: no

### System Info

- `transformers` version: 4.33.0.dev0
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?:N/A

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers.models.llama.tokenization_llama import LlamaTokenizer

tokenizer = LlamaTokenizer()

tokenizer.tokenize("")
```

which gives the error:
```
UnboundLocalError: local variable 'tokens' referenced before assignment2346    if tokens[0] == SPIECE_UNDERLINE and tokens[1] in self.all_special_tokens
```

### Expected behavior

The tokenizer should return an empty list if an empty string is passed, or possibly error with a helpful error message, but I shouldn't get a variable referenced before declaration error.

Comments:
+1

Btw `LlamaTokenizerFast` seems to be fine with an empty string

```py
tokenizer = LlamaTokenizerFast.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
tokenizer.tokenize("") # returns []
```

but `LlamaTokenizer` returns this error:
```
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Cell In[25], line 2
      1 tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
----> 2 tokenizer.tokenize("")

File ~\Documents\llama2\lib\site-packages\transformers\models\llama\tokenization_llama.py:214, in LlamaTokenizer.tokenize(self, text, **kwargs)
    211 if len(text) > 0:
    212     tokens = super().tokenize(SPIECE_UNDERLINE + text.replace(SPIECE_UNDERLINE, " "), **kwargs)
--> 214 if tokens[0] == SPIECE_UNDERLINE and tokens[1] in self.all_special_tokens:
    215     tokens = tokens[1:]
    216 return tokens

UnboundLocalError: local variable 'tokens' referenced before assignment
```

Hey! Thanks I can indeed reproduce this, and this should just return `[]`.
Loading XGLM triggers warning "Some weights of .. were not initialized from" for a module with no params
Author: MattYoon
Pull request: no

### System Info

- `transformers` version: 4.32.0
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada 
~~But probably a general issue in the `PretrainedModel` class.~~

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('facebook/xglm-564M')
# Some weights of XGLMForCausalLM were not initialized from the model checkpoint 
# at facebook/xglm-564M and are newly initialized: ['model.embed_positions.weights']
# You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

### Expected behavior

The warning should not be triggered.

The positional embedding for XGLM uses `XGLMSinusoidalPositionalEmbedding` which does not have actual trainable parameters. My guess is that since there is no parameters, the key is not actually stored in the checkpoint and triggers the warning. This issue might exist in other models that similarly have `nn.Module` with no trainable parameters.

Comments:
Thanks for raising the issue @MattYoon 
@ArthurZucker can confirm, I think the fix would be to make the weight of that module non-persistent (`persistent=False`) so that they won't get saved in the state_dict: https://github.com/huggingface/transformers/blob/main/src/transformers/models/xglm/modeling_xglm.py#L179 
Indeed there is not need to save them in the state dict (or to consider them when loading the weights) as they are created on the fly. If that fixes the issue, would you be happy to open a PR with the fix?
🌐 [i18n-KO] Translated peft.md to Korean
Author: nuatmochoi
Pull request: yes

# What does this PR do?

Translated the `peft.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

Team OSSCA, may you please review this PR? @bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @Sunmin0520, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

May you please review this PR? @sgugger, @ArthurZucker, @eunseojo

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25706). All of your documentation changes will be reflected on that endpoint.

LGTM! 👍
BERT: TensorFlow Model Garden Conversion scripts
Author: stefan-it
Pull request: no
Labels: name: WIP

### Feature request

Hi,

after working some time with the [TensorFlow Model Garden Repository](https://github.com/tensorflow/models) and training BERT models, I found out the following things that could be changed in Transformers library:

I added the Token Dropping BERT Conversion script a while ago, see #17142. Now I found out, that latest BERT models pretrained with Model Garden Repository repository can also be converted with this script.

For this reason I would propose to rename the script `convert_bert_token_dropping_original_tf2_checkpoint_to_pytorch.py` just to `convert_bert_original_tf2_checkpoint_to_pytorch.py`.

However, this script also exists, but it is no longer working, as this was deprecated in Model Garden Repository a while ago, I added this notice in #16171.

I see now two possibilities to proceed with the different conversion scripts:

* Rename the current `convert_bert_original_tf2_checkpoint_to_pytorch.py` to something like `convert_deprecated_bert_original_tf2_checkpoint_to_pytorch.py` so that this name is free for the "new" conversion script that supports Token Dropping BERT und latest BERT models from Model Garden Repository.
* Delete the old script completely

### Motivation

More recent BERT and Token Dropping BERT models can be pretrained with TensorFlow Model Garden repository.

There should be one script that does these conversions, the old one that is only working with deprecated models from Model Garden repo should be renamed or deleted.

### Your contribution

I can take care of renaming/deletion and extending the conversion script to have better documentation.

Comments:
cc @Rocketknight1 

This isn't really my area either! A lot of this code goes back to the earliest code in `transformers` when it was a port of TF code for BERT to PyTorch. Pinging @LysandreJik - do you know if the code is intended to support ports from recent versions of the TF Model Garden?

As long as everything is correctly documented, I'm all for having up to date scripts that work with the most recent BERT releases.

In that case @stefan-it I think it's okay to delete the old script entirely and replace it with a modern one, since it's no longer usable and people who need it for historical purposes can always find it in past release branches.

Thanks for your feedback! I will prepare PR for this soon.

Please unstale :robot:
Migarate old cache to transformers v4.22.0
Author: yzGao22
Pull request: no

### System Info

- `transformers` version: 4.22.0
- Platform: Linux-5.4.0-1089-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.10
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@LysandreJik

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When update transformers from v4.12.3 to v4.22.0,  got following message:

```
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
Moving 55 files to the new cache system
0%
0/55 [00:00<?, ?it/s]
There was a problem when trying to move your cache:

  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/utils/hub.py", line 1127, in <module>
    move_cache()

  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/utils/hub.py", line 1071, in move_cache
    hub_metadata[url] = get_hub_metadata(url, token=token)

  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/utils/hub.py", line 996, in get_hub_metadata
    huggingface_hub.file_download._raise_for_status(r)


Please file an issue at https://github.com/huggingface/transformers/issues/new/choose and copy paste this whole message and we will do our best to help.
```

### Expected behavior

The migrating process can be done successfully.

Comments:
I am also facing this issue with 4.24

Looks like there was a connection problem when the util tried to migrate the cache. If you don't care about the cached models, you can just ignore this, everything will work fine.
To try again to move an old cache to the new format, you can execute
```
from transformers.utils.hub import move_cache

move_cache()
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Looks like there was a connection problem when the util tried to migrate the cache. If you don't care about the cached models, you can just ignore this, everything will work fine. To try again to move an old cache to the new format, you can execute
> 
> ```
> from transformers.util.hub import mvoe_cache
> 
> move_cache()
> ```

In case anyone else finds this, there are two typos in the supplied command.  The correct import is:

`from transformers.utils.hub import move_cache`
🌐 [i18n-KO] `model_memory_anatomy.md` to Korean
Author: mjk0618
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다! -->
# What does this PR do?

Translated the `model_memory_anatomy.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee
<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
May you please review this PR? @sgugger, @ArthurZucker, @eunseojo

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25755). All of your documentation changes will be reflected on that endpoint.

May you please review this PR? @sgugger, @ArthurZucker, @eunseojo
Error with checking args.eval_accumulation_steps to gather tensors
Author: chaumng
Pull request: yes

The error is in ```trainer.py```.

While the deprecated (legacy) code has the correct check (line 3772):  
```python
if args.eval_accumulation_steps is not None and (step + 1) % args.eval_accumulation_steps == 0:
```

The current code does not (line 3196):
```python
if args.eval_accumulation_steps is not None and self.accelerator.sync_gradients:
```

We need to check ```(step + 1) % args.eval_accumulation_steps == 0```, because otherwise, if we set args.eval_accumulation_steps = 10, the code may still gather tensors for each step.

Hence, the line 3196 should be modified to: 
```python
if args.eval_accumulation_steps is not None and (step + 1) % args.eval_accumulation_steps == 0 and self.accelerator.sync_gradients:
```

Comments:
cc @muellerzr 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25819). All of your documentation changes will be reflected on that endpoint.

@muellerzr Thank you. I have run the make commands. All tests passed.
Tests: detect lines removed from "utils/not_doctested.txt" and doctest ALL generation files
Author: gante
Pull request: yes

# What does this PR do?

As the title indicates, this PR:
1. Detects lines removed from `utils/not_doctested.txt`, and adds the removed files to the list of files to doctest
2. Doctest ALL generation files 🫡 

This PR started off with 2. -- however, as you can see in the CI in the first commit, no doctests were triggered. As such, since the doctest fixes were minimal, I've decided to add 1. as well :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
replace roberta embedding with bge_base
Author: shameem198
Pull request: no
Labels: name: New model

### Model description

I'm experimenting Funsd like dataset using layoutLmv3, I'm trying to replace roberta embedding with new bge_large, Is it possible to replace the embedding? will it improve the accuracy?

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!
Add `OwlViTForObjectDetection` to `MODEL_FOR_OBJECT_DETECTION`
Author: ydshieh
Pull request: yes

# What does this PR do?

~~This seems a miss, but need @NielsRogge to make sure.~~

well, he told me

> it's not a bug, it's intended. OWL-ViT can be loaded using the AutoModelForZeroShotObjectDetectionModel class (few that's a long name)

Comments:
well

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24992). All of your documentation changes will be reflected on that endpoint.
Fix `token` in auto classes
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `token` in auto classes.

Fix #25008

We (me) should start to:
  - add (some) explicit arguments to `from_pretrained` for auto classes
  - (probably) clean-up the usage of `use_auth_token` even internally

But let's do this in separate PR(s), I promise.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25059). All of your documentation changes will be reflected on that endpoint.

Good catch. I guess I have to deal with the internal usages of those `kwargs`

Yes, ideally we want to only use `token` everywhere now :-) Happy to help if you need some!
🌐 [i18n-KO] Translated `add_new_pipeline.md` to Korean
Author: heuristicwave
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `add_new_pipeline.md` to Korean" 으로 부탁드립니다! -->
# What does this PR do?

Translated the `add_new_pipeline.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

May you please review this PR? @nuatmochoi, @bolizabeth, @hyunhp, @mjk0618, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

May you please review this PR? @sgugger, @ArthurZucker, @eunseojo

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25498). All of your documentation changes will be reflected on that endpoint.

May you please review this PR @stevhliu ? cc: @sgugger , @ArthurZucker, @eunseojo
Thank you so much for your help!