noamaanMulla-03/transformers-issues huggingface dataset:
Llama2 vocabulary duplicates
Author: don-tpanic
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.16.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: GTX 3090ti
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```import transformers

llm = 'meta-llama/Llama-2-7b-chat-hf'

tokenizer = transformers.AutoTokenizer.from_pretrained(
        llm,
        use_auth_token=True,
    )

print('id 29909:', tokenizer.decode([29909]))
print('id 319:', tokenizer.decode([319]))
```
get output 
```
id 29909: A
id 319: A
```

In fact, it appears the vocab has many duplicates:
```
counter = {}
for id in range(32000):
    counter[tokenizer.decode([id])] = counter.get(tokenizer.decode([id]), 0) + 1
print(len(counter))
```
I get 
```
26519
```

### Expected behavior

I was expecting all ids (32k of them) will map to a unique subword which is not the case in my little example. Did I miss something here? 

Thanks in advance!

Comments:
You should try using:
```python 
In [4]: tokenizer.convert_ids_to_tokens(29909)
Out[4]: 'A'

In [5]: tokenizer.convert_ids_to_tokens(319)
Out[5]: '▁A'
```
The decode function remove the extra `'▁'`

> You should try using:
> 
> ```python
> In [4]: tokenizer.convert_ids_to_tokens(29909)
> Out[4]: 'A'
> 
> In [5]: tokenizer.convert_ids_to_tokens(319)
> Out[5]: '▁A'
> ```
> 
> The decode function remove the extra `'▁'`

Thanks for your prompt reply @ArthurZucker!! Can I ask a follow-up question on this?

I also experimented the following but get unexpected results - 
```
token_id_A1 = tokenizer.convert_tokens_to_ids('A')
token_id_A2 = tokenizer.convert_tokens_to_ids('_A')
print('token_id_A1: ', token_id_A1)
print('token_id_A2: ', token_id_A2)
```
While `A` is mapped to 29909, `_A` is mapped to `0` which is unknown token, but shouldn't it be `319`?

I also tried 
```
token_id_A1 = tokenizer("A", return_tensors='pt').input_ids
```
from which I get 
```
tensor([[  1, 319]])
```
While `1` is mapped to `
`, however `319` as your example shows should be mapped to `_A` not `A`

Thanks again I hope my question makes sense!

The reason is because `'_A'` and `'▁A'` are not the same tokens. `_` is the under score (first case) while `'▁'` is the special SPIECE_UNDERLINE token. 

> The reason is because `'_A'` and `'▁A'` are not the same tokens. `_` is the under score (first case) while `'▁'` is the special SPIECE_UNDERLINE token.

Thanks for your clarification. Very helpful! Closing the issue for now.
Sets the stalebot to 10 AM CEST
Author: LysandreJik
Pull request: yes

This sets the stale bot trigger time at 10 AM CEST rather than 5 PM CEST as all core maintainers on watch duty are now in the European timezone

cc @amyeroberts @ArthurZucker @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Transformers documentation translation to Macedonian
Author: NinoRisteski
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @ArthurZucker, @sgugger for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) 
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
Hi @sgugger, is it ok to work on this?
Avoid importing all models when instantiating a pipeline
Author: sgugger
Pull request: yes

# What does this PR do?

The check on the model being in the right mapping for the pipeline requires the import of all models associated with that pipeline. This is because it uses the `MODEL_XXX_MAPPING` instead of the `MODEL_XXX_MAPPING_NAMES`. This PR suggests to switch to that, since the check is done on the model name anyway, to avoid weird warnings unrelevant to the user (for instance #24903 gives one example, I also end up having some logs of CUDA kernel being built on my side).

The only feature of the `MODEL_XXX_MAPPING` we need to keep compared to `MODEL_XXX_MAPPING_NAMES` is the extra content (coming from model on the Hub), so this PR adds a reference from the mapping names to the mapping.

Fixes #24903

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It works too, but I think with the design selected for the PEFT integration in `from_pretrained` (which will supercede #24827), we won't need to remove them. I think it's a useful warning to have when users try to instantiate a pipeline with a model not suitable for it.
self.pad_token detect should be is None rather than "not self.pad_token"
Author: aohan237
Pull request: no

https://github.com/huggingface/transformers/blob/2f8acfea1ca11fe3479fb379ccbded516d0cff57/src/transformers/tokenization_utils_base.py#L2506C71-L2506C71

Comments:
Thanks! I agree.

@ArthurZucker If you agree, I can open a PR to update this tiny place.

Sure!
⚠️ [CLAP] Fix dtype of logit scales in init
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

The dtype of the CLAP logit scale parameters was always float64 by default (even if the rest of the model was initialised in float32). This PR fixes the logit scales, such that they respect the default dtype of the model.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Note that in the original repo, the model is always cast to float16 for all training / inference. Thus, they likely never used the model in it's default dtype, and always relied on explicitly casting to float16
Generate: general test for decoder-only generation from `inputs_embeds` 
Author: gante
Pull request: yes

# What does this PR do?

As discussed in #25664, adds a test to detect whether generating from `inputs_embeds` has exactly the same output as from the corresponding `input_ids`. This ensures contributed implementations are correct 🤗 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh #25664 motivates this PR because it was a reaction to #25659, which had an uncaught bug. This uncaught bug would have been detected by the test added in this PR :)

Regarding the other PR comments: going to elaborate in the test comments (we disable EOS to always have a test length of 20 tokens, and disable PAD because generate tries to infer the attention mask from `input_ids` when it exists [and it obviously fails from `inputs_embeds`])

(not important question)

I know it's from #25659, but #25664 only avoids putting `input_ids` into the `kwargs` (using `update`). I am not seeing why the test added in this PR can fail if it is run against  #25659 😅 I must miss something.



@ydshieh before the fix, it would crash when `inputs_embeds` and `inputs_ids` are passed to generate (because the forward pass gets the two, and it triggers an exception), but it should be allowed :)

This is a case that the new test checks 

I see, thank you for explaining! You are really king of generate (and Arthur knows everything now!)
Added max_length warning to pretrained tokenizer initialization
Author: grantslewis
Pull request: yes

# What does this PR do?

A truncation warning, "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.", while not incorrect (max_length is a parameter for running the tokenize function), could also be interpreted as suggesting 'max_length' is a parameter used across the tokenizer class. However, 'max_length' silently does nothing when it is passed into AutoTokenizer.from_pretrained(), for example, and the previously mentioned warning will still occur. The proposed addition provides a simple warning when the 'model_max_length' is None and 'max_length' is passed in as an argument. The warning suggests the correct parameter name instead. This is meant to help provide a warning to direct developers to the correct variable name, helping prevent any unnecessary troubleshooting. This change should not require any additional tests or dependencies.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

CC @ArthurZucker

Comments:
[DOCS] Added docstring example for EpsilonLogitsWarper #24783
Author: sanjeevk-os
Pull request: yes

# What does this PR do?

See #24783 
Added docstring example for EpsilonLogitsWarper

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


## Who can review?

@gante 


Comments:
Hi @gante, addressed your comments. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25378). All of your documentation changes will be reflected on that endpoint.

> The example has 40 lines at the moment, which is too long. Our docs should be concise, otherwise users won't bother reading them :)
> 
> Let's get a case where we can showcase the processor with 2 `generate` calls (one with and another without `epsilon_cutoff`). Note that `set_seed` needs to be called before each `generate` call, otherwise we can't show that `epsilon_cutoff` had an impact (i.e. otherwise the difference can be attributed to sampling, and not to `epsilon_cutoff`)

Hi @gante made the suggested changes.

@sanjeevk-os Thank you for your contribution 💛
correct resume training steps number in progress bar
Author: pphuc25
Pull request: yes

Hi,
In this code of resume training with step, I see that completed_steps will always start at epoch 0, and it does not seem to be corrected (progress_bar needs to be at the step in the file save). So I changed the location of this code to make the progress_bar update with the latest step.

I would like to cc @sgugger to review it.

Comments:
fix ram efficient fsdp init
Author: pacman100
Pull request: yes

# What does this PR do?
1. Currently, when using Trainer if the model is loaded before creating `TrainingArguments` object, torch distributed process group won't be initialized and as such when FSDP is enabled via accelerate config, it will end up initializing the model with random weights on all ranks as the `is_fsdp_enabled_and_dist_rank_0` function will always return `False`. This results in NaN losses. Quite a journey to uncover this bug. This PR fixes it.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix failing `test_batch_generation`  for bloom
Author: ydshieh
Pull request: yes

# What does this PR do?

#25571 changed this test. There was a tiny issue. See my comment along the change in this PR.

We should probably to change this variable names in multiple places for house keeping.

Also cc @gante  😄 

Comments:
> 😅 Nice catch !

I was caught by the failed CI 😢 

_The documentation is not available anymore as the PR was closed or merged._
docs: Resolve typos in warning text
Author: tomaarsen
Pull request: yes

# What does this PR do?

Resolves a warning & a double space in a new warning text.

## Before submitting
- [x] This PR fixes a typo or improves the docs

## Who can review?

@ArthurZucker

- Tom Aarsen

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25711). All of your documentation changes will be reflected on that endpoint.
LLama2 tokenizer unexpected behaviors with special tokens
Author: ShomyLiu
Pull request: no

### System Info

- `transformers` version: 4.33.0.dev0
- Platform: Linux-3.10.0-1160.83.1.el7.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0a0+b5021ba (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <yes>
- Using distributed or parallel set-up in script?: <yes>

### Who can help?
@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

With the latest version of transformers:
```
import transformers
tok1 = transformers.AutoTokenizer.from_pretrained("llama2-7b", use_fast=False)
tok2 = transformers.AutoTokenizer.from_pretrained("llama2-7b", use_fast=False, legacy=True)

print(tok1.tokenize("
Human is Here"))
# output: ['
', 'H', 'uman', '▁is', '▁Here']

print(tok2.tokenize("
Human is Here"))
# output  ['
', '▁Human', '▁is', '▁Here']

```
It seems that it is wrong with the default setting (legacy = False).

### Expected behavior

The expected behavior is:
 ```
['
', '▁Human', '▁is', '▁Here']
```

Comments:
You expectation is wrong, the output of the slow tokenizer is correct. It's a known bug, as the `fast` fix for special tokens is not there yet. 

Thanks for the explanation. It is indeed a large change between different versions.
In transformers==4.31.0, the behavior is also different as follows:
```
In [8]: tok.tokenize("
Human")
Out[8]: ['
', 'Human']

In [9]: tok("
Human")
Out[9]: {'input_ids': [1, 2, 0], 'attention_mask': [1, 1, 1]}
```
Although I have read the PR documents before, it is also confusing about which one is right.
Is it a good choice that uses the legacy=True for stable consideration?



Yes `legacy = True` is a very good choice. The `transformers==4.31.0` had bugs, which were fixed in the latest version (which is why you have different results). 
```python 
>>> tokenizer.sp_model.encode("Human", out_type=str)
```
is the correct output (without a prefix space being added.
Now previously you would have 
```python 
>>> tokenizer.sp_model.encode("Human", out_type=str)
['▁Human']
```
and the `▁` was stripped 

I am sorry for the confusion and for the issues, this was a very nasty bug 

Got it. Thanks a lot for your efforts on this confusing bug, and have seen that there are quite several commits about this issue.
I will keep `legacy = True` now and look forward to your final fix. Thanks again.


Thanks for your valuable feedback! Community issues help us know most of the edge cases that we might have missed

🤗
[`PEFT`] Fix peft version
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the peft version check, in fact we should check if the current version is strictly greater than the required minimum version, not the opposite. That was leading to failing tests in the nightly CI.

cc @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25710). All of your documentation changes will be reflected on that endpoint.
Remove the input tokens
Author: satnair
Pull request: no

Hi,

Scenario: I have got 5 files , which has code in it. Now I am trying to evaluate the files and get some recommendations via starcoder model. 

Challenge: I am able to iterate thru all files and get recommendations independently. But when running in a single flow in a loop, after the first file is encoded and decoded, for the second file, the input_ids of the previous file remains. How to remove the input_ids tokens of the previous file. 

for each file
input_ids: torch.Tensor = self.tokenizer.encode(query, max_length=7000, return_tensors='pt', truncation=True).to(self.device)
        print(len(input_ids[0]))

For example: 
1st file: Len of input IDs is , 1111
2nd file[2nd iteration]: Len of input IDs is, 3018 [but it should 1907]


Please help with a solution for this.

Comments:
Hey, this kind of question should be asked on [the forum](https://discuss.huggingface.co/) 🤗 

Sure. Thanks.
Fix number of minimal calls to the Hub with peft integration
Author: sgugger
Pull request: yes

# What does this PR do?

This PR makes sure we don't add two calls to every model instantiation when PEFT is installed by moving the calls to `find_adapter_config_file` after the config is created so we can use the commit hash.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25715). All of your documentation changes will be reflected on that endpoint.
[`AutoGPTQ`] Add correct installation of GPTQ library + fix slow tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

Per the instructions of installing auto-gptq library we need to slightly update the Dockerfile otherwise it will install the `0.3.2` version which is not compatible with the integration

Will also update some expected values to make the slow tests pass

cc @ydshieh @SunMarc 

Can confirm the Docker image is built successfully


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can confirm the slow tests now pass in the docker image

> So basically we want to install the same version but with cu118 support, right?

It is slightly trickier than that, auto-gptq listed in pypi does not contain the latest supported versions (>= `0.4.1`), one needs to install that directly through the command that I have shared. Also that way you get the pre-built wheels instead of building auto-gptq at each install (which is the case right now if you do `pip install auto-gptq`)
Generate: nudge towards `do_sample=False` when `temperature=0.0`
Author: gante
Pull request: yes

# What does this PR do?

Related issue: https://github.com/facebookresearch/llama/issues/687

Improves the error message when `temperature=0.0`, which asymptotically corresponds to greedy decoding... except that it results in numerical problems :D 

___________________________
test run:
```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")

inputs = tokenizer(["The quick brown"], return_tensors="pt")
gen_out = model.generate(**inputs, do_sample=True, temperature=0.0)
```

yields
```
ValueError: `temperature` (=0.0) has to be a strictly positive float, otherwise your next token scores will be invalid. If you're looking for greedy decoding strategies, set `do_sample=False`.
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> We could also just set do_sample = False  in case temperature = 0. Will let you decide !

I agree we should do that! But I'm going to leave that for the generate refactor, as it implies significant code changes to do it right :)