noamaanMulla-03/transformers-issues huggingface dataset:
Inconsistent results between LlamaTokenizer and LlamaTokenizerFast
Author: zzzzzzrc
Pull request: no

### System Info

transformers==4.34.1
tokenizers==0.14.1


### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
```
import transformers

fast = transformers.AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', use_fast=True)
slow = transformers.AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', use_fast=False)

>>> fast.encode("abs
USER")
[1, 6425, 2, 3148, 1001]
>>> slow.encode("abs
USER")
[1, 6425, 2, 11889]
>>> fast.decode([1, 6425, 2, 3148, 1001])
'
 abs
 USER'
>>> slow.decode([1, 6425, 2, 11889])
'
 abs
USER'
```
fast tokenizer would add a whitespace between eos and USER

### Expected behavior

fast and slow tokenizer get the same results

Comments:
Hey! This is a duplicate of #26318, #26455, #25881, and will be fixed in #26678 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[LLaVa] Some improvements
Author: NielsRogge
Pull request: yes

# What does this PR do?

Some minor improvements when going over the LLaVa code.

Comments:
They are failing on my setup:
```
FAILED tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_small_model_integration_test_batch - AssertionError: Lists differ: ['USER:  \nWhat are the things I should be [267 chars]ock'] != ['\nUSER: What are the things I should be c[306 chars] R.']
FAILED tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_small_model_integration_test_llama - AssertionError: 'USER[116 chars]hich appears to be a dock or pier extending ov[562 chars]ier.' != 'USER[116 chars]hich is a pier or dock extending over a body o[572 chars]ies.'
FAILED tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_small_model_integration_test_llama_batched - AssertionError: Lists differ: ['USE[114 chars]ANT: When visiting this serene location, one s[177 chars]ed.'] != ['USE[114 chars]ANT: the water is calm and clear\n\nThe image [154 chars]ed.']
```
Could possibly also be the case on main, will check

Yeah it still fails for me both on main and my branch even after #27909. So @ArthurZucker could you perhaps run the slow tests from my branch before merging? But I'm pretty sure it doesn't affect integration tests.
Fix M4T v2 integration tests
Author: ylacombe
Pull request: yes

# What does this PR do?

Some M4T-v2 integration tests are [causing GPUs OOM](https://github.com/huggingface/transformers/actions/runs/7054968060/job/19204890066). It happens when we load two models together. I thus shifted some integration tests to half precision which should solve the issue.

cc @ydshieh @ArthurZucker 


Comments:
OC, I've already verified that the tests pass (on a 24GB TITAN RTX)! I've also monitored by hand GPU utilization (around 10GB) so I believe it should pass on a 16GB GPU.

FYI, the tests that I've changed verify that every task-specific model has a 1-to-1 correspondence with the task-agnostic model
Fix SDPA dispatch & make SDPA CI compatible with torch<2.1.1
Author: fxmarty
Pull request: yes

As per title.

On torch==2.0.1, these do pass
```
RUN_SLOW=1 pytest tests/models/bart -s -vvvvv -k "torchscript"
RUN_SLOW=1 pytest tests/models/llama -s -vvvvv -k "torchscript"
RUN_SLOW=1 pytest tests/models/whisper -s -vvvvv -k "torchscript"
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/bert -s -vvvvv
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/llama -s -vvvvv
```

On torch==2.1.1, these do pass (https://github.com/huggingface/transformers/pull/26572#issuecomment-1847774858)
```
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/ -s -vvvvv -k "flash or sdpa"
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/whisper -s -vvvvv -k "llama"
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/llama -s -vvvvv
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/bart -s -vvvvv
RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/bert -s -vvvvv
```

There was a bug where even though we manually request `attn_implementation="eager"`, we would still go into the SDPA controlflow and hard check that the requirements are fine. Which is not what we want.

Comments:
[`from_pretrained`] Make from_pretrained fast again
Author: ArthurZucker
Pull request: yes

# what does this PR do 
Skips all layer initialization when loading from pretrained without accelerate. 

From ~20seconds to 5 seconds for a 7B model like llama.

The Weights are effectively initialized in « init_weights_ » of the pretrained method. All internal calls are skipped

- [x] Check that if a linear layer is missing it will be initialized! (Loading `AutoModelForCausalLM` from `AutoModel`

fixes #26258 and fixes #18505


` model = XXXX.from_pretrained(model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True")`  might fail

Comments:
- explicit overwrite breaks fx and is not that faster 
- Non initialized is not always zeros (failing tests) just make sure it’s not initialized 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27709). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
use logger.warning_once to avoid massive outputs
Author: ranchlai
Pull request: yes


# What does this PR do?
This is a quick fix to to avoid massive outputs when training/finetuning longformer (for text classifcation) by using ```logger.warning_once``` rather than ```logger.info```



Comments:
Sure! Happy to do that.  If I understand correctly, this will only apply to `logger.info/warn` in the `forward()` function  and the functions called by  `forward()`. 

@ranchlai Awesome - thanks! Yep, that's right. Feel free to ping with any questions if you're not sure about any of them.
[`Add Mixtral`] Adds support for the Mixtral MoE
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Adds the latest MoE model from mistral AI

Comments:
Are there any other aux losses apart from the LM loss?

The auxiliary loss can be computed with `output_router_logits = True` automatically, other losses like `z_loss` can be imported from Switch transformers modeling code! Custom losses should be able to use the router_logits returned by the model
Docs for AutoBackbone & Backbone
Author: merveenoyan
Pull request: yes

This is the docs for `AutoBackbone` class and backbone. I'm also drafting a notebook but I will be off this week so it will be delayed. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27456). All of your documentation changes will be reflected on that endpoint.

@amyeroberts this didn't throw an error during inference:
```python
model = AutoBackbone.from_pretrained(
"microsoft/swin-tiny-patch4-window7-224", out_indices=(0,1,2), out_features=["stage4"]
)
```

I addressed rest of your comments. 🤗 

> @amyeroberts this didn't throw an error during inference:

Thanks for flagging - it should! I'll look into it 👀 

@MKhalusova I added illustrations and defined backbone/neck/head.

> @amyeroberts this didn't throw an error during inference:
> 
> ```python
> model = AutoBackbone.from_pretrained(
> "microsoft/swin-tiny-patch4-window7-224", out_indices=(0,1,2), out_features=["stage4"]
> )
> ```
> 
> I addressed rest of your comments. 🤗

OK, so this is a bit tricky to resolve. The reason this is happening is: 
* The checkpoint `"microsoft/swin-tiny-patch4-window7-224"` was created before the swin backbone had been added. It therefore doesn't contain `out_features` or `out_indices` in the config. 
* When loading the model in `from_pretrained` the config is first created [from the json here](https://github.com/huggingface/transformers/blob/bd50402b56980ff17e957342ef69bd9b0dd45a7b/src/transformers/models/auto/configuration_auto.py#L1064) and then the kwargs (in this case `out_indices` and `out_features`) are [passed in here](https://github.com/huggingface/transformers/blob/bd50402b56980ff17e957342ef69bd9b0dd45a7b/src/transformers/models/auto/configuration_auto.py#L1081).
* In the config's `from_dict` method, the kwargs are applied [one at a time](https://github.com/huggingface/transformers/blob/bd50402b56980ff17e957342ef69bd9b0dd45a7b/src/transformers/configuration_utils.py#L766). This means that they're not verified against each other (but are still verified if they're consistent with `stage_names`).

This is part of the reason I'm wanting to deprecate `out_features` because handling this is annoying. I'll see if I can do something reasonable in the `BackboneConfigMixin` which isn't too convoluted to resolve. 

@amyeroberts since Maria has approved, can you merge? 

@MKhalusova Can you merge this if you can? Unfortunately, this wasn't included in recent release as I hope it would :/ 

Hi,

Thanks a lot for this PR! 🙏 was wondering whether we could remove the out_features from the docs, since after offline discussion with @amyeroberts, we proposed to only favor `out_indices` from now on since it's easier to maintain. Could you open a follow-up PR regarding that?
Fix test for auto_find_batch_size on multi-GPU
Author: muellerzr
Pull request: yes

# What does this PR do?

The new test added in https://github.com/huggingface/transformers/pull/27568 doesn't account for multi-GPU, when the `bs` is multiplied by `n_gpu` for the effective train batch size. This PR modifies the test slightly as a result to work on any number of GPUs (and CPU)

Fixes # (issue)

Failing nightly test

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 


Comments:
Update import message
Author: NielsRogge
Pull request: yes

# What does this PR do?

Currently, if you do have Accelerate installed, but you don't have the `min_version` specified [here](https://github.com/huggingface/transformers/blob/56be5e80e6cd5264012eb9ea84bd589233a503d9/src/transformers/utils/import_utils.py#L671), you will get a message saying Accelerate is not installed.

So I've improved the error message.

cc @muellerzr 

Comments:
Fix parameter count in readme for mixtral 45b
Author: CyberTimon
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR fixes the parameter count number in the readme. In the [mistral blog post](https://mistral.ai/news/mixtral-of-experts/) they state that it's a 45b model and not a 84b / 85b.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27945). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: return `past_key_values`
Author: gante
Pull request: yes

# What does this PR do?

Enables returning `past_key_values` from `generate`, if `return_dict_in_generate=True` (otherwise only the generated `input_ids` are returned) and `use_cache=True` (otherwise there is no cache to return ;) ).

In more abstract terms, this enables features like:
1. continuing a given generation without having the more expensive prefill step -- like in multi-turn conversations
2. exploring the KV values without having to place a breakpoint in `generate` 👀 🐛 

The added code for the feature is minimal, so most of the PR is docs and tests 🤗  

Fixes #24841 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is a killer feature :+1: 

@gante Hi! Thanks for PR!
Did you test feeding output past_key_values into .generate() method? Like take first 250 tokens input, run .generate(), get output past_key_values, take another 50 tokens input and run .generate() with previous 250 past_key_values? With beam search it seems to be kinda tricky. I'm trying to resolve multiple dimension mismatch problems.

(merging and leaving the conversion to a skip as a TODO)

I dont see a version number when will this be out?

Next release :) (v4.36)

@nevakrien 4.36v is now out :)
fix no sequence length models error
Author: AdamLouly
Pull request: yes

# What does this PR do?
Models with no sequence length limit will fail to create a dataset because the block_size will be -1.
and it will error this:
`ValueError: num_samples should be a positive integer value, but got num_samples=0`

this PR handles the case where max_position_embeddings  is -1, we set block size to a default value.

This will fix this issue:
https://github.com/huggingface/transformers/issues/27521


Comments:
@amyeroberts  can you check this when you have time?
Thank you :) 

Hi @AdamLouly - thanks for opening this PR and apologies for the delay! 

I'm not sure that this is where we want to apply the fix: not having a fixed sequence length is a feature of some models. I think it would make more sense to change the setting of `block_size` so that it isn't -1 if `max_pos_embeddings` is -1.

@amyeroberts I changed it, let me know if this works.
[`Mixtral`] Change mistral op order
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR slightly refactors the forward pass logic of `MixtralBLockSparseTop2MLP` to not have `routing_weights` as a required arg in the forward pass as AWQ does not handle multiple args in the forward pass of the modules (assumes all modules have `hidden_states` as input)

cc @ArthurZucker 


Comments:
Update bounding box format everywhere
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes an issue pointed out by https://discuss.huggingface.co/t/owl-vit-postprocess-api-bbox-conversion/65309/2, namely we state that we postprocess to the COCO API, but effectively we're using the Pascal VOC format.

A [very nice blog post](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#:~:text=albumentations%20is%20similar%20to%20pascal_voc,the%20height%20of%20the%20image) explaining all bounding box formats goes into more detail.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27944). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Typos with Knowledge Distillation for Computer Vision documentation 
Author: asusevski
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@stevhliu and @MKhalusova and @merve

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Issue 1**: ```NameError: name 'teacher_extractor' is not defined```
```
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator()
trainer = ImageDistilTrainer(
    student_model=student_model,
    teacher_model=teacher_model,
    training_args=training_args,
    train_dataset=processed_datasets["train"],
    eval_dataset=processed_datasets["validation"],
    data_collator=data_collator,
    tokenizer=teacher_extractor,
    compute_metrics=compute_metrics,
    temperature=5,
    lambda_param=0.5
)
```
**Issue 2**: Trainer doesn't initialize
```
class ImageDistilTrainer(Trainer):
    def __init__(self, *args, teacher_model=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.teacher = teacher_model
        self.student = student_model
        self.loss_function = nn.KLDivLoss(reduction="batchmean")
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        self.teacher.to(device)
        self.teacher.eval()
        self.temperature = temperature
        self.lambda_param = lambda_param
```

### Expected behavior

**Issue 1**: ```teacher_extractor``` should be ```teacher_processor```
**Issue 2**:  ```ImageDistilTrainer``` should be:
```
class ImageDistilTrainer(Trainer):
    def __init__(self ,teacher_model=None, student_model=None, temperature=None, lambda_param=None,  *args, **kwargs):
        super().__init__(model=student_model, *args, **kwargs)
        self.teacher = teacher_model
        self.student = student_model
        self.loss_function = nn.KLDivLoss(reduction="batchmean")
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        self.teacher.to(device)
        self.teacher.eval()
        self.temperature = temperature
        self.lambda_param = lambda_param
```
Will raise PR for both fixes!

Comments:
Support PeftModel signature inspect
Author: dancingpipi
Pull request: yes

# What does this PR do?

If we set `remove_unused_columns` to 'True' while training a `LoRA` model, the all dataset columns will be removed.
This is because the `_set_signature_columns_if_needed` function directly checks the signature of `self.model`. If `self.model` is a `PeftModel`, the signature will become `['args', 'kwargs']` , causing the valid columns in the dataset to be deleted.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@muellerz @pacman100 


Comments:
cc @amyeroberts for a second look.
The "source" button in docs points to 404
Author: R-N
Pull request: no

### System Info

Windows 11 64 bit 21h2
Google chrome latest

### Who can help?

@stevhliu 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Open [latest Trainer docs](https://huggingface.co/docs/transformers/main_classes/trainer)
2. Scroll to training_step
3. Click "source"

For me, it opens [this link](https://github.com/huggingface/transformers/blob/v4.36.0/src/transformers/trainer.py#L2697), which shows 404 for me.

### Expected behavior

It opens the source code, for example, source code of training_step.

Comments:
Hi @R-N, 

Thanks for raising this issue! The link should now work as v4.36 has been released. 

> Hi @R-N,
> 
> Thanks for raising this issue! The link should now work as v4.36 has been released.

Confirmed, it works now. Thank you