noamaanMulla-03/transformers-issues huggingface dataset:
Flax Refactor v2
Author: cgarciae
Pull request: yes

# What does this PR do?

Alternative to #22627. Instead of making `FlaxPretrainedModel` a Flax `Module`, this PR aims to make all inner `.module`s usable in a standalone way by moving any pre/posprocessing done by its `*Model` container into the Module itself.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22866). All of your documentation changes will be reflected on that endpoint.

Hey @cgarciae - we're nearly finished with this PR far as I can tell. Do you have the bandwidth to see this through to completion?  Happy to help with the last stages of integration here!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`pipeline`] revisit device check for pipeline
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/23336#issuecomment-1657792271 

Currently `.to` is called to the model in pipeline even if the model is loaded with accelerate - which is a bad practice and can lead to unexpected behaviour if the model is loaded across multiple GPUs or offloaded to CPU/disk. 

This PR simply revisits the check for device assignment

Simple snippet to reproduce the issue:

```python 
from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer, pipeline
import torch

model_path="facebook/opt-350m"

config = AutoConfig.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, load_in_8bit=True, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_path)
params = {
        "max_length":1024,
        "pad_token_id": 0, 
        "device_map":"auto", 
        "load_in_8bit": True,
        # "torch_dtype":"auto"
        }
pipe = pipeline(
    task="text-generation",
    model=model,
    tokenizer=tokenizer,
    device=0,
    model_kwargs=params,
)
```

cc @sgugger @Narsil 

Comments:
After thinking about it, maybe this shouldn't be the right fix, it is a bad intent from users to add a `device_map` + `device` argument.
Let me know what do you think

_The documentation is not available anymore as the PR was closed or merged._

Yeah let's raise an error!
Update tiny model info. and pipeline testing
Author: ydshieh
Pull request: yes

# What does this PR do?

Just a regular update.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25213). All of your documentation changes will be reflected on that endpoint.
Fix docker image build failure
Author: ydshieh
Pull request: yes

# What does this PR do?

We again get not enough disk size error on docker image build CI. I should try to learn some ways to reduce the size and avoid this error, but this PR fixes this situation in a quick way: install torch/tensorflow before running `pip install .[dev]`, so they are only install once, and we have fewer docker layers produced.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
T5 Tokenizer Legacy behaviour warning
Author: pointonjoel
Pull request: no

### System Info

I am running on Google Collab (Python 3.10.6) using tokenizers-0.13.3, transformers-4.31.0,  huggingface-hub-0.16.4 ,safetensors-0.3.1

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm getting a legacy behaviour warning come up when simply loading a T5 tokenizer - it appears even before using the tokenizer. Is there an updated way to load the tokenizer? The warning appears when running the following lines of code:

_from transformers import AutoTokenizer
tokeniser = AutoTokenizer.from_pretrained("google/mt5-small")_

The warning is:
_You are using the legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
/usr/local/lib/python3.10/dist-packages/transformers/convert_slow_tokenizer.py:470: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
warnings.warn(_


### Expected behavior

No warning

Comments:
Hey! Will just answer for potentially confused users: 
- the warning is triggered if `legacy=True` which is the default for backward compatibility
- to use the latest behaviour, use `tokeniser = AutoTokenizer.from_pretrained("google/mt5-small", legacy=False)`


It might be nice to add the fix to the error message -- it's a bit hard to find :)
inject automatic end of utterance tokens
Author: stas00
Pull request: yes

This adds a new feature:

For select models add `<end_of_utterance>` token at the end of each utterance.

The user can now easily break up their prompt and not need to worry about messing with tokens.

So for this prompt:

```
    [
        "User:",
        image,
        "Describe this image.",
        "Assistant: An image of two kittens in grass.",
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image.",
        "Assistant:",
    ],
```

this new code with add_end_of_utterance_token=True will generate:

`full_text='
User:<fake_token_around_image><image><fake_token_around_image>Describe this image.<end_of_utterance>Assistant: An image of two kittens in grass.<end_of_utterance>User:<fake_token_around_image><image><fake_token_around_image>Describe this image.<end_of_utterance>Assistant:'
`


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25218). All of your documentation changes will be reflected on that endpoint.
LLAMA 2 HF tokenizer len is 32001
Author: ari9dam
Pull request: no

### System Info

Installed from source (4.32.0.dev0)


```
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-13b-hf")
print(len(tokenizer)) #32001
print(model.config.vocab_size) #32000
```



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-13b-hf")
print(len(tokenizer)) #32001
print(model.config.vocab_size) #32000
```

### Expected behavior

model vocab size and tokenizer len should be 32000. It seems the padding token of the tokenizer is set to '\<unk\>'. Which is not normally the case. It's normally not set.

Comments:
The actual config says that `pad_token_id=0` - so I assume this is correct?

What is interesting is that id `32000` maps to a token `'<pad>'` while the original vocab does not contain this token:
https://huggingface.co/meta-llama/Llama-2-7b-hf/raw/main/tokenizer.json

It seems this is being added somewhere in HF code?

cc @ArthurZucker 

Hey! Yes this is not entirely expected, we update the slow tokenizer, but the fast version did not get the update. I'll open PRs to fix this! 

@ArthurZucker jfyi I see the same behavior for slow and fast on 4.31.0

edit: correction, indeed it is not set in the tokenizer

Is there any plan to fix it?

It is fixed on all model! 

@ArthurZucker  Does the fix really resolve this? I pip installed `transformers` from Github, and there is still a mismatch.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

# Using fast tokenizer by default
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
print(len(tokenizer)) # 32001
print(model.config.vocab_size) # 32000
print(tokenizer.get_added_vocab()) # {'<pad>': 32000}
```

Transformers version: `4.32.0.dev0`

Will check asap! Might be the fast tokenizer that did not get updated.

Sorry but no, I cannot reproduce your issue. I do not know if you maybe did not update cached files but I have this: 
```python 
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> len(tokenizer)
32000
>>> print(tokenizer.get_added_vocab())
{}
```

Thanks for confirming. You're right, it's because of the locally cached files. Downloading them from huggingface and re-running the code now works fine.
make build_mpt_alibi_tensor a method of MptModel so that deepspeed co…
Author: sywangyi
Pull request: yes

…uld override it to make autoTP work

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
enable autoTP for mpt in huggingface model.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada



Comments:
@sgugger @ArthurZucker please review. thanks

should work with https://github.com/microsoft/DeepSpeed/pull/4062

_The documentation is not available anymore as the PR was closed or merged._
[Bis] Adding new tokens while preserving tokenization of adjacent tokens
Author: Madjakul
Pull request: no

### System Info

* `transformers` version: 4.31
* Platform: Linux [...] 5.19.0-50-generic 50-Ubuntu  x86_64 GNU/Linux
* Python version: 3.10.12
* Huggingface_hub version: 0.16.4
* PyTorch version (GPU?): 2.0.1+cu118 (True)
* Using GPU in script?: No
* Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This issue is related to [this HuggingFace post on the official forum](https://discuss.huggingface.co/t/adding-new-tokens-while-preserving-tokenization-of-adjacent-tokens/12604), hence the similar title, and to my knowledge, no answer was given as to whether this is the normal tokenizer behavior. I ran into the same problem as the original poster while trying to tokenize a sentence after adding new tokens: the adjacent tokens of the newly added ones aren't computed with their preceded escape symbol.

```py
>>> import transformers
>>> tok = transformers.RobertaTokenizer.from_pretrained("roberta-base")
>>> lotr_sent = 'Aragorn told Frodo to mind Lothlorien'
>>> tok.convert_ids_to_tokens(tok(lotr_sent)['input_ids'])
['
', 'Ar', 'ag', 'orn', 'Ġtold', 'ĠFro', 'do', 'Ġto', 'Ġmind', 'ĠL', 'oth', 'lor', 'ien', '
']
>>> tok.add_tokens(['Aragorn', 'Frodo', 'Lothlorien'])
3
>>> tok.convert_ids_to_tokens(tok(lotr_sent)['input_ids'])
['
', 'Aragorn', 'told', 'Frodo', 'to', 'Ġmind', 'Lothlorien', '
']
```

### Expected behavior

The tokens `told`, `Frodo`, `to` and `Lothlorien` should be preceded with a `Ġ` character if I am not mistaken ; e.g.:
```py
>>> import transformers
>>> tok = transformers.RobertaTokenizer.from_pretrained("roberta-base")
>>> lotr_sent = 'Aragorn told Frodo to mind Lothlorien'
>>> tok.convert_ids_to_tokens(tok(lotr_sent)['input_ids'])
['
', 'Ar', 'ag', 'orn', 'Ġtold', 'ĠFro', 'do', 'Ġto', 'Ġmind', 'ĠL', 'oth', 'lor', 'ien', '
']
>>> tok.add_tokens(['Aragorn', 'Frodo', 'Lothlorien'])
3
>>> tok.convert_ids_to_tokens(tok(lotr_sent)['input_ids'])
['
', 'Aragorn', 'Ġtold', 'ĠFrodo', 'Ġto', 'Ġmind', 'ĠLothlorien', '
']
```

Comments:
Hey! This has already been answered, and is a duplicate of #14770. Will be fixed by #23909.
Pix2Struct -- mismatched output of cross attention weights
Author: leitro
Pull request: no

### System Info

Hi huggingface team! 

The output of cross attention weights is mismatched as shown in [https://github.com/huggingface/transformers/blob/05cda5df3405e6a2ee4ecf8f7e1b2300ebda472e/src/transformers/models/pix2struct/modeling_pix2struct.py#L1551C22-L1551C22](https://github.com/huggingface/transformers/blob/05cda5df3405e6a2ee4ecf8f7e1b2300ebda472e/src/transformers/models/pix2struct/modeling_pix2struct.py#L1551C22-L1551C22). 

In the code: `all_cross_attentions = all_cross_attentions + (layer_outputs[3],)`

where `layer_outputs[3]` is still the self attention weights, the REAL cross attention weights should be `layer_outputs[5]`.  

Please correct me if I made some mistakes. Looking forward to the updated version. Thank you! @amyeroberts @ArthurZucker @younesbelkada 



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hightlight of the training code:
```
model = Pix2StructForConditionalGeneration.from_pretrained('google/pix2struct-docvqa-base')
outputs = model.forward(**inputs, labels=labels, output_attentions=True)
```
Turn on the attention output button by `output_attentions=True`, and then get the cross attention weights by `outputs.cross_attentions` where the bug exists.


### Expected behavior

Change the index from `3` to `5` for selecting the correct cross attention weights, then everything's done hopefully. 

Comments:
Nice catch @leitro ! 
I can confirm this is correct, just made https://github.com/huggingface/transformers/pull/25200 to fix the issue on the main branch.

Cheers!!
[`Pix2Struct`] Fix pix2struct cross attention
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/25175

As pointed out by @leitro on the issue, I can confirm the cross-attention should be in `layer_outputs[5]`. Also fixes the attention output index that should be in index `3` as the index `2` is the `position_bias` (they have the same shape so we didn't noticed the silent bug on the CI tests.)

to repro:

```python
import requests
import torch
from PIL import Image
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor

url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)

model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-textcaps-base")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-textcaps-base")

input_ids = torch.LongTensor([[0, 2, 3, 4]])

# image only
inputs = processor(images=image, return_tensors="pt")
outputs = model.forward(**inputs, decoder_input_ids=input_ids, output_attentions=True)

print(outputs.cross_attentions[0].shape)
>>> should be torch.Size([1, 12, 4, 2048])
```

cc @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Fixed a slow test of torchscript that was failing, however the test:

```bash
tests/models/pix2struct/test_modeling_pix2struct.py::Pix2StructIntegrationTest::test_batched_inference_image_captioning_conditioned
```

is failing but can confirm is also failing on main, I think it is unrelated to this PR (env issues on my VM probably) so I am merging
[`Docs`/`quantization`] Clearer explanation on how things works under the hood. + remove outdated info
Author: younesbelkada
Pull request: yes

# What does this PR do?

As discussed internally with @amyeroberts , this PR makes things clearer to users on how things work under the hood for quantized models. Before this PR it was not clear to users how the other modules (non `torch.nn.Linear`) were treated under the hood when quantizing a model. 

cc @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`MPT`] Add  `require_bitsandbytes` on MPT integration tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title and as discussed offline with @ydshieh 
adding `require_bitsandbytes` is needed to avoid issues with past torch CI that doesn't have bnb installed on their Docker images.

cc @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Jus out of curiosity does the `tooslow` decorator leads to tests still being run? 
```bash
    Slow tests are skipped while they're in the process of being fixed. No test should stay tagged as "tooslow" as
    these will not be tested by the CI.
```
Currently running MPT-7B is the only way to check if we are in sync with trust remote code weights, as it is the smallest model available

We also have `Salesforce/instructblip-vicuna-7b` but I haven't checked how long it takes to run on CI. So far it doesn't seems too problematic (those 7b models)

@ydshieh we also load that model using bnb: https://github.com/huggingface/transformers/blob/main/tests/models/instructblip/test_modeling_instructblip.py#L526 perhaps I can also add `require_bitsandbytes` there too

Yes, please. I missed that in the Past CI report. Thanks a lot!

No the `tooslow` tests are only run manually, not a runner.

Ok so I would say maybe we should keep testing mpt-7b so that we're aware of any potential issue through the daily CI
[`Detr`] Fix detr BatchNorm replacement issue
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the current failing CI on #25077 / related failing jobs: https://app.circleci.com/pipelines/github/huggingface/transformers/69452/workflows/999f3686-2d9a-4324-bed6-1c858f4d8246/jobs/871127 

In #25077 I decided to [add a property method `current_adapter`](https://github.com/younesbelkada/transformers/blob/peft-integration-attempt-2/src/transformers/adapters/peft_mixin.py#L156) to easily switch between adapters. This leads to failing CI because `PreTrainedModel` will inherit from `AdapterMixin` (that will contain that attribute) and `replace_batch_norm` loops over `dir(model)` and calls `getattr(model, attr_str)`, therefore checks for all available attributes including `current_adapter`. 

I can also change the property method to an instance method to avoid this issue, but I find it cleaner to do the module replacement in a pure PyTorch manner rather than using `dir(model)` which can cause weird behaviours in the future .

Can confirm slow DETR / DETA integration tests pass with this change

cc @sgugger @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25230). All of your documentation changes will be reflected on that endpoint.
Move rescale dtype recasting to match torchvision ToTensor
Author: amyeroberts
Pull request: yes

# What does this PR do?

The dtype casting of the input image when rescaling was moved in #25174 so that precision was kept when rescaling if desired. However, this broke equivalence tests with torchvision's `ToTensor` transform c.f. [this comment](https://github.com/huggingface/transformers/pull/24796#issuecomment-1657275333).


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you very much, Amy!