noamaanMulla-03/transformers-issues huggingface dataset:
Different handling of added tokens between fast and slow LlamaTokenizer
Author: cg123
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
tok_slow = transformers.AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", use_fast=False)
tok_slow.add_tokens(["
", "
"])

>>> tok_slow("
test

\n
")
{'input_ids': [1, 32000, 1369, 32001, 32000, 28705, 13, 32001], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}

tok_fast = transformers.AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", use_fast=True)
tok_fast.add_tokens(["
", "
"])
>>> tok_fast("
test

\n
")
{'input_ids': [1, 32000, 1613, 28789, 28766, 21133, 28729, 28766, 3409, 28766, 14908, 28766, 28767, 13, 28789, 28766, 21133, 28729, 28766, 28767], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

### Expected behavior

I would expect the tokenizers to handle the added tokens in the same way regardless of use_fast. The behavior of the slow tokenizer is what I was expecting.

Comments:
guess it's the same issue as this one: https://github.com/huggingface/transformers/issues/27132

Yep looks pretty similar. This nevered worked for any tokenizers before 😉 

so if I understand the answer in the other thread correctly, using `normalized=False` would yield consistent results for fast and slow tokenizer?

like this for @cg123's case:  
```
tokenizer.add_tokens(AddedToken("
", normalized=False))
tokenizer.add_tokens(AddedToken("
", normalized=False))
```


Yes, if you are using `transformers>=4.34` and `tokenizers>=0.14.0` and the flag `legacy` is set to `False` in the slow tokenizer 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generate: `assisted_decoding` now accepts arbitrary candidate generators
Author: gante
Pull request: yes

# What does this PR do?

A common trend is starting to pop up: people are experimenting with new strategies to generate candidate sequences, to then run an assisted-generation-like strategy. A key example is the new technique in https://github.com/huggingface/transformers/issues/27722, which is equal to `assisted_decoding` except for the candidate generation part. This technique in particular achieves nice speedups in some settings, and doesn't need an assistant model -- a model-free speedup!

To facilitate experimentation and the addition of new candidate generation techniques, this PR abstracts the candidate generation part in `assisted_decoding` to a new class with a stable API. This was inspired in classes like `LogitsProcessor` or `StoppingCriteria` -- components of `generate` that can easily be replaced. All these changes are backwards compatible! 🤗 

Suggested review order:
1. `utils.py`, to see the shape of `assisted_decoding` under the abstracted API
2. `candidate.py`, to see the structure of the new base class (and the specific case of the original assisted generation)

_____________________________________________

The following tests are passing:
1. `RUN_SLOW=1 py.test tests/models/whisper/ -k speculative`
4. `py.test tests/ -k test_assisted` (which catches mixin and integration tests associated with assisted generation)

Happy to add more tests if needed :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27750). All of your documentation changes will be reflected on that endpoint.

One more place needs to change I think - `generation_mode` is currently set using `_get_generation_mode` , where this is the logic:

```
if assistant_model is not None:
            if generation_mode in ("greedy_search", "sample"):
                generation_mode = GenerationMode.ASSISTED_GENERATION
            else:
                raise ValueError(
                    "You've set `assistant_model`, which triggers assisted generate. Currently, assisted generate "
                    "is only supported with Greedy Search and Sample."
                )
```

how do u suggest this should change to support prompt lookup decoding? @gante 

@apoorvumang I'd add an `or` after `if assistant_model is not None`
Hot-fix-mixstral-loss
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes 
```python 
load_balancing_loss_func
    gate_logits = torch.cat(gate_logits, dim=0)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:7! (when checking argument for argument tensors in method wrapper_cat)
    gate_logits = torch.cat(gate_logits, dim=0)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:6! (when checking argument for argument tensors in method wrapper_cat)
```
which appears when computing the loss in parallel settings (accelerate) .
The actual tensors are pretty small ( batch x seq_length , 2) so putting them all on cpu should be alright. There is no perfect solution for now.

Either this or we use some gather operation

Comments:
Actually will compute on GPU!
Link is invalid in  “examples/flax/image-captioning/README.md”
Author: wplf
Pull request: no

**This repository is focused on the Hub experience and documentation. If you're facing an issue with a specific library, please open an issue in the corresponding GitHub repo. If you're facing an issue with a specific model or dataset, please open an issue in the corresponding HF repo.**


**Bug description.**
A clear and concise description of what the problem is. Ex. Clicking this button is not working when [...]

The superlink behind 【FlaxVisionEncoderDecoderModel】 is not working
![image](https://github.com/huggingface/hub-docs/assets/95006218/4459b545-7c3c-4b6f-9b53-7717ba4fbb51)



**Describe the expected behaviour**
A clear and concise description of what you want to happen.

**Additional context**
Add any other relevant context or screenshots here. Please share details such as browser when appropriate.


Comments:
Fix link in README.md of Image Captioning
Author: saswatmeher
Pull request: yes

Update the link for vision encoder decoder doc used by FlaxVisionEncoderDecoderModel link inside README.md of Image Captioning.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #27968 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@stevhliu and @MKhalusova

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Better key error for AutoConfig
Author: Rocketknight1
Pull request: yes

When users try to load a model with AutoModel/AutoConfig but the model type isn't recognized, they get a confusing error about missing keys. However, these errors are usually caused by their version of `Transformers` being out of date. I've seen several users asking for help with this issue trying to load `mixtral`, so I wrote a better error message for next time!

Comments:
Fix SDPA correctness following torch==2.1.2 regression
Author: fxmarty
Pull request: yes

As explained in https://github.com/pytorch/pytorch/issues/112577, `torch==2.1.2` reintroduces a bug (that was first introduced in 2.1.0 and fixed in 2.1.1) in SDPA where the operator produces wrong outputs when using a custom `attn_mask`, cuda device and memory-efficient attention backend.

This PR makes it so that we don't silently fall into this bug.

Running `RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/gpt_bigcode/ -s -vvvvv`, we have:

### With `torch==2.1.1` without this patch

all tests pass

### With `torch==2.1.2` without this patch (regression)
```
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_beam_sample_generate - RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_beam_search_generate - AssertionError: Lists differ: [[15, 95, 23, 94, 98, 62], [82, 51, 84, 98, 1, 0]] != [[15, 95, 23, 94, 98, 62], [82, 51, 84, 98, 66, 21]]
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_constrained_beam_search_generate - AssertionError: Lists differ: [[29,[207 chars] 48, 73, 79, 64, 93, 83, 40], [74, 76, 22, 92,[58 chars] 40]] != [[29,[207 chars] 48, 14, 82, 4, 46, 83, 4...
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_flash_attn_2_generate_padding_right - AssertionError: False is not true
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_generate_continue_from_past_key_values - AssertionError: False is not true
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_group_beam_search_generate - AssertionError: Lists differ: [[85,[33 chars] 31, 68, 25], [93, 70, 87, 4, 69, 8], [93, 70, 87, 31, 68, 91]] != [[85,[33 chars] 31, 68, 7], [93, 70, 87,...
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelLanguageGenerationTest::test_generate_batched - AssertionError: Lists differ: ['def[78 chars]say_hello():\n    // 1. Create a new array with the values of'] != ['def[78 chars]say_hello():\n    print("...
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelLanguageGenerationTest::test_generate_simple - AssertionError: 'def print_hello_world():\n    print("Hello World")\n\ndef print_hello_' != 'def print_hello_world():\n    print("Hello World!")\n\n\nde...
```

### With `torch==2.1.2` & `torch==2.1.1` with this patch

All tests pass.

For the other archs supporting SDPA (llama, whisper, falcon, idefics, bart), the tests are running fine & manual tests go fine as well.

Comments:
Don't close ClearML Task when training is complete
Author: johnml1135
Pull request: yes

This is to fix https://github.com/huggingface/transformers/issues/26762.

Comments:
cc @skinan for first review - as integrations are maintained by the contributors who added them 

@johnml1135 Am I correct in understanding this PR can be closed as the issue was resolved in #26614 ? 

Yes - it should be resolved now.

@johnml1135 Thanks for confirming!
fix bug in dvclive callback
Author: dberenbaum
Pull request: yes

# What does this PR do?

Fixes a typo in the dvclive callback that prevents it from being set as initialized.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts or @muellerzr Would one of you mind taking a look? Apologies for not catching this. Our internal tests missed this scenario where initialization depends on the `setup()` method.


Comments:
fix typo in dvclive callback
Author: dberenbaum
Pull request: yes

# What does this PR do?

Fixes a typo in the dvclive callback that prevents it from being set as initialized.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts or @muellerzr Would one of you mind taking a look? Apologies for not catching this. Our internal tests missed this scenario where initialization depends on the `setup()` method.


Comments:
[`Tokenizer Serialization`] Fix the broken serialisation 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Should fix some serialization issues, mostly save_pretrained with all the init kwargs, and from_pretrained with dicts.
fixes #26732

With main:
```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/Llemma_7b", use_fast=False)
File ~/Work/transformers/src/transformers/tokenization_utils_base.py:2253, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, *init_inputs, **kwargs)
   2251     if added_tokens_map != {} and init_kwargs[key] is not None:
   2252         if key != "additional_special_tokens":
-> 2253             init_kwargs[key] = added_tokens_map.get(init_kwargs[key], init_kwargs[key])
   2255 init_kwargs["added_tokens_decoder"] = added_tokens_decoder
   2256 # convert {'__type': 'AddedToken', 'content': '<ent>', 'lstrip': False, 'normalized': True, ...} to AddedTokens

TypeError: unhashable type: 'dict'
```

This is because the tokenizer had special tokens saved as dicts, and the call to `convert_added_tokens`. is made after this.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27099). All of your documentation changes will be reflected on that endpoint.

Pegasus is the only slow failure I witnessed so checking this now before merging! 

Ok, the issue is that when we force the added tokens encoder in the slow tokenizer, the fast of course can't do this. So the eos token gets replaced at index 0 in slow but not in fast. 
Will update to force the default vocab to the default tokens.
Adds VIP-llava to transformers
Author: younesbelkada
Pull request: yes

# What does this PR do?

VIP-llava is a new Llava variant. It seems the only difference between Llava and VIP-Llava is that VIP-llava uses a projector layernorm before passing the hidden states into the MM projector. It also concatenates many hidden states from the image encoder before passing it to the multi-modal projector.

```python
from transformers import pipeline
from PIL import Image    
import requests

model_id = "ybelkada/vip-llava-7b"
pipe = pipeline("image-to-text", model=model_id, model_kwargs={"load_in_4bit": True, "use_flash_attention_2": True})
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png"

image = Image.open(requests.get(url, stream=True).raw)
prompt = "USER: <image>\nCan you please describe this image?\nASSISTANT:"

outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 100})
print(outputs[0]["generated_text"])
>>> USER: <image> 
Can you please describe this image?
ASSISTANT: The image features a brown and white cat sitting on a green surface, with a red ball in its paw. The cat appears to be playing with the ball, possibly a sports ball, as it is positioned in a relaxed manner. The cat's eyes are wide open, indicating that it is focused on the ball and possibly in the middle of a playful moment.
```

![image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png)

> The image features a brown and white cat sitting on a green surface, with a red ball in its paw. The cat appears to be playing with the ball, possibly a sports ball, as it is positioned in a relaxed manner. The cat's eyes are wide open, indicating that it is focused on the ball and possibly in the middle of a playful moment.

Also compatible with Flash Attention 2.

https://github.com/mu-cai/ViP-LLaVA

cc @ArthurZucker @NielsRogge @mu-cai @haotian-liu 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27932). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
(LLama-2) TensorParallelPreTrainedModel does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention
Author: VasilGeorgiev39
Pull request: no

```python
import transformers
import tensor_parallel as tp

tokenizer = transformers.AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-chat-hf")
model = transformers.AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-13b-chat-hf")

modelp = tp.tensor_parallel(model) #error
```

As is the example from https://github.com/BlackSamorez/tensor_parallel

Comments:
Hi @VasilGeorgiev39, thanks for raising an issue! 

So that we can best help you, please make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and provide information about the running environment and full details of the error encountered, including traceback. 

Note than widespread support for SDPA was added in the [recent v4.26 release](https://github.com/huggingface/transformers/releases/tag/v4.36.0). I would first make sure to install the latest version of transformers and see if this resolves the issue. 

`tp.tensor_parallel` is a third party to the transformers repo. Providing the full traceback and error information will enable us to figure out which library the error is coming from. 

Hi @amyeroberts, apologies for not following the format. I was just following the link that was displayed when [throwing the exception](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1466) I have now opened a more detailed [issue](https://github.com/huggingface/transformers/issues/28003) using the template so I will close this one.
[`CI slow`] Fix expected values
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fix slow test. The init was probably done twice because:

```pyton 
Some weights of ViTMSNForImageClassification were not initialized from the model checkpoint at facebook/vit-msn-small and are newly initialized: ['classifier.bias', 'classifier.weight']
```
so the weights should no be initialized by the form_pretrained but by the `_init_weights`

Comments:
Thanks a lot!
[i18n-jp] Translating docs tojp
Author: rajveer43
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the jp-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `jp` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `jp/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section



Comments: