noamaanMulla-03/transformers-issues huggingface dataset:
Weird Tokenization when Training New Tokenizer from GPT2 Tokenizer using train_new_from_iterator
Author: minmie
Pull request: no

### System Info

- `transformers` version: 4.33.0     
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
from datasets import load_dataset

path = r'E:\pythonWork\nlp\ner\soucre_data\cluener.jsonl'  # a chinese text dataset
raw_data = load_dataset("json", data_files=path, split='train')

training_corpus = (
    raw_data[i : i + 1000]["text"]
    for i in range(0, len(raw_data), 1000)
)

old_tokenizer = AutoTokenizer.from_pretrained("E:\pythonWork\models\gpt2")
tokenizer = old_tokenizer.train_new_from_iterator(training_corpus, 10000)

example = '就是去美国大使馆的官方网站，它有中文版，去把每一条仔细研究透了，把每一个表格和材料都准备好了'  # chinese text
old_tokens = old_tokenizer.tokenize(example)
print('old_tokens:',old_tokens)

new_tokens = tokenizer.tokenize(example)
print('new_tokens',new_tokens)
```



cluener.jsonl as follow:
![image](https://github.com/huggingface/transformers/assets/40080081/da268899-8454-47cc-af19-50e3804ff51f)


### Expected behavior

I  trained my own tokenizer using chinese text base on this demo (https://huggingface.co/learn/nlp-course/chapter6/2) .

And the outputs of the two tokenizer are as follow:

```
old_tokens: ['å°', '±', 'æĺ¯', 'åİ', '»', 'ç', '¾', 'İ', 'åĽ', '½', 'å¤§', 'ä½¿', 'é', '¦', 'Ĩ', 'çļĦ', 'å®', 'ĺ', 'æĸ¹', 'ç', '½', 'ĳ', 'ç«', 'Ļ', 'ï', '¼', 'Į', 'å®', 'ĥ', 'æľ', 'ī', 'ä¸Ń', 'æĸ', 'ĩ', 'çīĪ', 'ï', '¼', 'Į', 'åİ', '»', 'æ', 'Ĭ', 'Ĭ', 'æ', '¯', 'ı', 'ä¸Ģ', 'æĿ', '¡', 'ä»', 'Ķ', 'ç', '»', 'Ĩ', 'ç', 'ł', 'Ķ', 'ç', '©', '¶', 'éĢ', 'ı', 'äº', 'Ĩ', 'ï', '¼', 'Į', 'æ', 'Ĭ', 'Ĭ', 'æ', '¯', 'ı', 'ä¸Ģ', 'ä¸', 'ª', 'è¡', '¨', 'æł', '¼', 'å', 'Ĵ', 'Į', 'æĿ', 'Ĳ', 'æĸ', 'Ļ', 'éĥ', '½', 'åĩ', 'Ĩ', 'å¤', 'ĩ', 'å¥', '½', 'äº', 'Ĩ']
new_tokens ['å°±æĺ¯', 'åİ»', 'ç¾İåĽ½', 'å¤§ä½¿é¦Ĩ', 'çļĦ', 'å®ĺæĸ¹ç½ĳç«Ļ', 'ï¼Į', 'å®ĥ', 'æľī', 'ä¸ŃæĸĩçīĪ', 'ï¼Į', 'åİ»', 'æĬĬ', 'æ¯ı', 'ä¸ĢæĿ¡', 'ä»Ķç»Ĩ', 'çłĶç©¶', 'éĢı', 'äºĨ', 'ï¼Į', 'æĬĬ', 'æ¯ıä¸Ģä¸ª', 'è¡¨', 'æł¼', 'åĴĮ', 'æĿĲæĸĻ', 'éĥ½', 'åĩĨå¤ĩ', 'å¥½', 'äºĨ']
```

emm，both outputs are really weird. is this right?

Comments:
Hey! You should have a look at https://github.com/huggingface/tokenizers/issues/203 it's expected! 🤗 

@ArthurZucker   thanks for your help.
Auto model time series
Author: wgifford
Pull request: yes

# What does this PR do?

Adds auto model capability for PatchTSMixer and PatchTST models. This includes support for: 
- AutoModelForTimeSeriesClassification
- AutoModelForTimeSeriesPrediction
- AutoModelForTimeSeriesRegression

@kashif 

Comments:
Hi @wgifford, thanks for opening this PR! 

AutoModelForXxx APIs are intended for crossloading of models with the same inputs and outputs for a specific task. As there's just two (new) models at the moment, I suggest waiting until we know what the standardised API should be for the time series case. 

Hi @amyeroberts. The intent here was to make it a bit easier to work with a collection of serialized models where we may not know exactly which model class form which it originates. For example, 
```
from transformers import AutoModelForTimeSeriesPrediction
saved_model = "ibm/patchtsmixer-etth1-forecasting"
model=AutoModelForTimeSeriesPrediction.from_pretrained(saved_model)
```
which works when `saved_model` is not known in advance.

One thought I had was that we might enable this for the three other time series models which support the "prediction" task.

@wgifford - I understand. Knowing which model is saved out is really a handling issue on the user's side and one that can be resolved by looking at the model configs. 

Nevertheless, if this is extended to 3 other models then there's more reason for an auto class for the prediction task.   

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Allow add_tokens for ESM
Author: Rocketknight1
Pull request: yes

The tokenizer code for ESM forces all added tokens to be special tokens, presumably because the authors felt that the list of amino acids in proteins was constant and therefore that there wouldn't be a need to actually expand the core vocabulary. However, there are definitely use-cases for expanding the vocabulary - see #28387.

This PR makes `add_tokens()` for ESM tokenizers behave like it does for other tokenizers, and doesn't force the added tokens to be special tokens.

Fixes #28387 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Clarify usage / implementation of padding for SigLIP model processor
Author: skysyk
Pull request: no

### Feature request

Change the implementation of `SiglipProcessor` to use the global default behavior for `padding` of `False` or update the documentation to indicate the usage is different and defaults to `'max_length'` if the `padding` argument is not provided.

### Motivation

In the HF documentation for padding (both the docs as well as the function comments for the processor class), the default behavior (argument) is described to be `False` or `'do_not_pad'`. For the `SiglipProcessor`, `max_length` is the default behavior implemented in [code](https://github.com/huggingface/transformers/blob/98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1/src/transformers/models/siglip/processing_siglip.py#L53) while the [example in the docs](https://huggingface.co/docs/transformers/main/en/model_doc/siglip#using-the-model-yourself) omits the padding argument. This is at odds with the overall documentation as well as behavior / usage examples provided in similar models such as CLIP (where explicitly `padding=True` in the usage example) and could give the wrong impression upon first glance that padding is not used for SigLIP.

### Your contribution

Opening this issue to discuss a clarification / improvement. I can help implement the preferred solution.

Comments:
Good point. cc @ArthurZucker - I added this to make sure people use `max_length` as the pre-trained models use this.

I guess it would be better to explicitly pass `padding="max_length"` so that this is more explicit and users have the choice.

Thank you!
Design of xxxAttention, xxxFlashAttention and xxxSdpaAttention
Author: ccdv-ai
Pull request: no

Hey
Following the addition of `torch.nn.functional.scaled_dot_product_attention` (#26572), there is a lot of deduplicated code between the `xxxAttention`, `xxxFlashAttention2` and `xxxSdpaAttention` classes. The main differences between the classes lie in the attention computation, the rest being the same (Q, K, V computation, cross attention and cache logic etc...).

Wouldn't it be simpler to offload the attention computation in a new shared file making the modeling files cleaner and simplify the use of these optimizations for older models? This would also ease the addition of new variants of attention in the future if there is any.

Comments:
Thank you. It's a tradeoff to have between the "one-model one-file" philosophy (https://huggingface.co/blog/transformers-design-philosophy) and offloading code to other files/classes.

We already kind of deviate from the "one-model one-file" philosophy with the KV cache refactor (https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_attn_mask_utils.py) and attention mask refactor (https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_attn_mask_utils.py), and one can indeed argue that we could do the same for the attention.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Trainer] move dataloader after the model wrapping
Author: kashif
Pull request: yes

# What does this PR do?

Call the dataloader after the model has been prepared

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27970). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fine tuned Mistral inference issue for >4k context length
Author: ghost
Pull request: no

**System Info**
- `transformers` version: 4.36.0-dev (main branch)
- Huggingface_hub version: 0.19.4
- PyTorch version: 2.1.0
- Using GPU in script?: yes

We fine tuned a mistralai/Mistral-7B-Instruct-v0.1using LoRa on some 8k context length data. The inferencing was fine with transformers at 4.34.0, but after updating the version, the inferencing became irrelevant repeition for token length > 4096.

We were able to get around this by disabling fast attention 2, but the overall model performance suffered. It seems to be a problem related to the 4D attention mask implementation in transformers 4.35+. This only happens when the token length exceeds 4k. Any ideas what might be wrong?



Comments:
Hi @oooodoori, thanks for raising this issue! 

So that we can best help, could you provide some more details about the issue? e.g.: 
* Have you tried this on 4.35 as well? How about the recent stable 4.36 release?
* Could you provide an example of a good versus bad output? 
* Is the only difference between getting good and bad outputs the transformers version? 
* Could you share a code snippet we can run to reproduce running the inference? Do you see this behaviour on pretrained weights as well e.g. [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) or only with your own finetuned model? 
* Can you provide more details about how the model was trained e.g. a training script to reproduce including the lora configuration? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Feature request: Adding einops as a Dependency
Author: OhadRubin
Pull request: no

### Feature request

I propose that the einops library be added as a dependency in the HuggingFace Transformers library. einops (Einstein Notation Operations) is a Python library that provides a more expressive language for tensor operations. This would offer highly readable and maintainable code for complex tensor reshaping and rearranging operations.
einops GitHub page: https://github.com/arogozhnikov/einops

### Motivation

The addition of einops as a dependency would greatly facilitate the integration of new models that already use it into the Transformers library.  Adding einops would require less refactoring in adding these models, making the code easier to read, and will decrease the amount of time spent on understanding and maintaining the code.

Given the potential benefits that the inclusion of einops could bring to the Transformers library, and considering its impressive community support highlighted by over 7,000 stars on GitHub, I think that this proposal merits serious consideration. I suggest that we put this feature request up for a community vote. This would allow all contributors to weigh in on the decision. 

### Your contribution

.

Comments:
There is actually a (recent) discussion

> There have been lots of discussions on this in the past, the TLDR is that we aren't supporting it cause it causes issues with things like ONNX

and the approach is more

>  just re-wrote that part in pure pytorch,



For me there is no reason to add a dependency when `einops` would actually be detrimental to all the work done to optimize inference (quantization, ONNX etc.). We can revisit this if the support end up being on par with classic PyTorch operations, but we shouldn't make the code easier to read if it's not as efficient/supported by the ecosystem.

einops has torch.compile support, why should it be a incompatible with onnx?
https://github.com/arogozhnikov/einops/wiki/Using-torch.compile-with-einops

It might be the case some point in the past (I am not the one involved in this topic previously). Not sure the status at this moment. A further investigation into this to make sure the whole ecosystem will run smoothly with models using `einops` is necessary before we add it as a dependency. But the teams have their own priorities, and we would like to see how the community reacts to this feature request first. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Should we write a test to see if einops works with the ecosystem?

@Bachstelze please do!
[`GPTNeoX`] Fix BC issue with 4.36
Author: ArthurZucker
Pull request: yes

# What does this PR do?
We broke some of the GPTNeoX model with the dtype in #25830. Sorry for the inconvenience. 
This was breaking in terms of logits, now this PR will probably make the model slow than with casting to smaller dtype.

Fixes #28360, fixes #28316.

For a sample generation I am getting some slow down. 

Comments:
Pythia (GPTNeoXForCausalLM) Regression (inference time) in transformers 4.35.0
Author: JonasGeiping
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.16.19-76051619-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.0.dev20240104 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### The Problem

The Pythia models, and by extension the `GPTNeoXForCausalLM` implementation don't appear to be working correctly in 4.35. I've attached a simple reproduction snippet below. This code works on 4.34, but produces NaNs on 4.35 during the forward pass. The token ids are not particularly anomalous.

The problem is likely related to the report at https://github.com/huggingface/transformers/issues/28316, but this issue shows that any effects on reward modeling might be 2nd-order effects and that changes between 4.34 and 4.35 are the problem.

@ArthurZucker @younesbelkada

### Reproduction

```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name_or_path = "EleutherAI/pythia-70m"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, torch_dtype=torch.float16)


input_ids = [29, 93, 303, 64, 5478, 49651, 10394, 187, 34, 12939, 875]  # this is '
system\nA chat between' 
# alternative: tokenizer('
system\nA chat between')
input_ids = torch.as_tensor(input_ids)[None]

model.cuda()
input_ids = input_ids.cuda()
model(input_ids)["logits"]  # has NaNs?
```

### Expected behavior

Normal forward pass, without NaNs.

Comments:
Thanks for reporting, this should be a lot easier to debug! 🤗 Having a look (can definitely reproduce).
- quick fix: using `bfloat16` get's rid of the nan's. 
- definitely a regression as I can confirm this was passing before. 
- seems to come from https://github.com/huggingface/transformers/commit/253f9a3f9716d08a81fb305fe71f983122eb608b 
- we don't have failing tests so either because we load in a different dtype or we just didn't test it.
Autocast
Author: jiqing-feng
Pull request: yes

Enable autocast in the pipeline.


Comments:
Hi @jiqing-feng, thanks for opening a PR and contributing to the library! Let us know when it's ready for final review. 

Could you provide some more context on what this addresses and how this should be used - ideally with an example snippet? I'm assuming to enable FA2?

As it's already possible to pass in a pretrained model which has been instantiated with a specific dtype e.g. `torch_dtype=torch.float16` or `torch_dtype="auto", there might be an interplay which should be tested to make sure no unexpected behaviour happens. 

cc @Narsil 

@yao-matrix

Hi @amyeroberts . I am sorry that I don't understand what is FA2 (Flash Attention?).

The motivation is from a bug, and we can reproduce it by the following script:
```python
from transformers import pipeline
from datasets import load_dataset
from datasets import Audio
import torch

minds = load_dataset("PolyAI/minds14", name="de-DE", split="train")
minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
example = minds[0]

asr = pipeline("automatic-speech-recognition", model="maxidl/wav2vec2-large-xlsr-german", torch_dtype=torch.bfloat16)
output = asr(example["audio"]["array"])
```

The ASR pipeline does not support bf16 `torch_dtype`, so I think we could enable `autocast` to fix the problem.

With my changes, we can load the `pipeline` by:
```python
asr = pipeline("automatic-speech-recognition", model="maxidl/wav2vec2-large-xlsr-german", enable_autocast=True, dtype=torch.bfloat16)
``` 
so we can run the script.

With `torch_dtype=torch.bfloat16`, we expected that the input tensors with float data type must be bfloat16, but `torch.autocast` don't have this limitation.

Hi @amyeroberts @Narsil . What do you think about this PR? I could enable autocast or full low-precision on the ASR pipeline.

Hi @jiqing-feng,

Why cannot you wrap `autocast`  directly on the pipeline calls themselves ?

Also note that I've bitten hard by autocast it creates slower that it should inference pretty fast because of all the data movement. In my experience it' s almost always better to add the proper types where needed manually to get good performance.
Fix tensor-parallelism link
Author: steilgedacht
Pull request: yes

# What does this PR do?

Replaces the old link in the llama configuration file to the new section on the website.
Clean PR of #27840

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@stevhliu 



Comments:
Can you run `make style`? 

running `make style` makes no changes, do you have any ideas on what to change?

Make sure you have `ruff==0.1.5` 🤗 and that you rebased on main! 

Okay, I have already `ruff=0.1.5` and rebased it, am I doing anything wrong here? 😅

![image](https://github.com/huggingface/transformers/assets/89748204/f7e55e4a-aa1c-4a56-af64-6b223c31ef00)




Ah no I have a similar issue. just `make fixup` should help I think otherwise just reverting the changes 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TypeError: Llama.create_completion() got an unexpected keyword argument 'min_p'
Author: mclassen
Pull request: no

### System Info

I should have the latest version.
When executing a query against a GGUF model, I get this error:

Traceback (most recent call last):
  File "/workspace/text-generation-webui/modules/callbacks.py", line 57, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "/workspace/text-generation-webui/modules/llamacpp_model.py", line 141, in generate
    completion_chunks = self.model.create_completion(
TypeError: Llama.create_completion() got an unexpected keyword argument 'min_p'

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Create a standard TheBloke runpod template for TextGen WebUI
2. download TheBloke/Phind-CodeLlama-34B-v2-GGUF, doesn't really matter which quantized file specifically
3. any query should trigger the TypeError
4. the output is empty

### Expected behavior

1. There should be no error.
2. The output should show the result of the query.

Comments:
Me to, just getting started will follow 

I've also been getting this error for several days now. Tried resetting the pod, creating new pods with different gpus or different models. Always getting the same error.

Hey! 🤗 seems like the issue is with `llama_cpp` rather than transformers 😅 

The issue is the template installing the wrong requirement.

https://github.com/TheBlokeAI/dockerLLM/issues/12

Still broken. Cannot use GGUF with ooba.

I encountered this error when running textgeneration-web-ui. 

1. Stop the server
2. Run update_linux.sh
3. Start the server

Fixed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
OSError: Unable to load weights from pytorch checkpoint file
Author: isRambler
Pull request: no

OSError: Unable to load weights from pytorch checkpoint file for '/root/autodl-tmp/llama-2-7b/pytorch_model-00001-of-00002.bin' at '/root/autodl-tmp/llama-2-7b/pytorch_model-00001-of-00002.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint

Why do I get an error when I use TensorRT-LLM’s build.py to convert the llam-2-7b model?
<img width="199" alt="19e41154a77724674ccbdc88d21d5ce" src="https://github.com/huggingface/transformers/assets/118053582/49fabd99-1ff3-49f1-ba40-7922bde51062">


Comments:
me too

Hey! I think that a conversion is attempted between pt and tf, but transformers does not support a TF implementation of this model. Could you rather open an issue on the `TensorRT-LLM` library with a full reproducer of what you are trying to do and the full stacktrace 

> Hey! I think that a conversion is attempted between pt and tf, but transformers does not support a TF implementation of this model. Could you rather open an issue on the `TensorRT-LLM` library with a full reproducer of what you are trying to do and the full stacktrace

Okay, I'm going to ask the relevant questions