noamaanMulla-03/transformers-issues huggingface dataset:
Error in pipeline while inferencing Llama2, colab  link below
Author: goblinvalo
Pull request: no

### System Info

here is the colab notebook link 
https://colab.research.google.com/drive/1rjDR7i9MWkTmOhsZEg3oU-AHntQhBFCD?usp=sharing

until yesterday it was working fine got this error today.

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1rjDR7i9MWkTmOhsZEg3oU-AHntQhBFCD?usp=sharing

### Expected behavior

until yesterday it was working fine look like today some update have been made by maintainers got this error all of a sudden.

Comments:
@goblinvalo  you can follow the [model card](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ) for the `TheBloke/Llama-2-7B-Chat-GPTQ` model 
i just used the following code in colab and it worked for me 

```bash
!pip3 install transformers>=4.32.0 optimum>=1.12.0
!pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/  # Use cu117 if on CUDA 11.7
``` 

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name_or_path = "TheBloke/Llama-2-7b-Chat-GPTQ"
# To use a different branch, change revision
# For example: revision="gptq-4bit-64g-actorder_True"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             trust_remote_code=False,
                                             revision="main")

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

prompt = "Tell me about AI" # 👈 your prompt here
prompt_template=f'''
 

You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.


{prompt}


'''

# print("\n\n*** Generate:")
# input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
# output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
# print(tokenizer.decode(output[0]))
print("*** Pipeline:")
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
    top_k=40,
    repetition_penalty=1.1
)

print(pipe(prompt_template)[0]['generated_text'])
```



@not-lain  Thank you so much for helping, it is working fine thanks a lot for the code.
RWKV : support attention_mask
Author: pfeatherstone
Pull request: no

### Feature request

At the moment, `attention_mask` in `RwkvModel` is ignored.
It would be great if this were supported.

### Motivation

I want to be able to train on a padded batch.

### Your contribution

RWKV is a bit alien to me

Comments:
Or is it that you don't need `attention_mask` because RWKV is causal by definition? So you can just ignore padded tokens?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @younesbelkada if you remember about this! 🤗 

RWKV does not use attention so no attention mask 😉
fix resuming from ckpt when using FSDP with FULL_STATE_DICT
Author: pacman100
Pull request: yes

# What does this PR do?
1. Fixes https://github.com/huggingface/transformers/issues/27878 introduced by https://github.com/huggingface/transformers/pull/27652
  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27891). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What's the purpose of Pad to 64 in LLaVA
Author: ShoufaChen
Pull request: no

Hello @younesbelkada,

Thanks for your awesome work that integrates LLaVA to transformers repo.

Would you mind providing more details about the padding tokenizer to 64 here? 

https://github.com/huggingface/transformers/blob/fe44b1f1a974139cd32a8884a63686425283b07c/src/transformers/models/llava/convert_llava_weights_to_hf.py#L71-L73

What's the advantage of 64? I thought 2 is enough? 

Thanks in advance.

Comments:
Hi @ShoufaChen 
Thanks for the issue,
In contrast to the official implementation, we decided to padd the language model's vocabulary size to introduce the new `<image>` token. Instead of adding 2, which is in fact sufficient, we decided to go for 64 for performance reasons as explained in this document: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc shared by @ArthurZucker here: https://huggingface.co/llava-hf/llava-1.5-7b-hf/discussions/5 
Let me know if this makes sense to you

Thanks for your detailed reply

Thank you very much @ShoufaChen !
load tokenizer size is different config vocab_size
Author: lw3259111
Pull request: no

### System Info

transformers==4.35.0
python==3.10
ubuntu==20

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. download model https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base 
2.  from transformers import LlamaTokenizer
3. tokenizer = LlamaTokenizer.from_pretrained("/data/Ziya2-13B-Base/",legacy=False)
4. len(tokenizer)

### Expected behavior

I use the https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base model
config.json
```json

{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "_flash_attn_2_enabled": true,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 5120,
  "initializer_range": 0.02,
  "intermediate_size": 13824,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 4096,
  "model_type": "llama",
  "num_attention_heads": 40,
  "num_hidden_layers": 40,
  "pad_token_id": 0,
  "rms_norm_eps": 1e-06,
  "rotary_emb_base": 10000,
  "rotary_pct": 1,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.30.2",
  "use_cache": true,
  "use_parallel_residual": false,
  "vocab_size": 39424
}
```

when I load the tokenizer,the tokenizer size is 39410
<img width="684" alt="image" src="https://github.com/huggingface/transformers/assets/12690488/55f688d7-194a-4ebf-8c0b-5c0acbff70d1">


Comments:
Hey! I recommend you to open the issue on the repo as we did not upload it and it might just be additional padding used for training that was not added to the tokenizer. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
decoder only beam search decoding indices usage
Author: luyijian3
Pull request: no

### System Info

I am using beam search decoding on opt-1.3b, and I wish to know which beam is selected as the final output along the generation process, I looked at beam_indices, but the size is prompt size+number of token generated, if I ignore the first prompt size indices, the remaining indices are all -1. So what should i do.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

beam_indice=output_with_watermark.beam_indices[:,tokd_input["input_ids"].shape[-1]:][0], which is all -1.

### Expected behavior

na

Comments:
cc @gante 

Hey @Hustcketlyj 👋 

The beam indices ([docs](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation.BeamSearchEncoderDecoderOutput.beam_indices)) are only -1 for unfilled positions ([reference in the code](https://github.com/huggingface/transformers/blob/c5037b459e117b9286c611092f38663f6cb763b0/src/transformers/generation/beam_search.py#L388)).

If that is not the case for you, I'll need a short reproducible script so I can figure out what might be wrong :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP] script to fine tune CLIPSeg
Author: rafaelpadilla
Pull request: yes

# What does this PR do?

This PR aims to include a tutorial on how to fune tune clipseg model.

Fixes #24494

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27344). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add time progress bar to track the group_by_length computation for bigger datasets on Trainer
Author: T-Almeida
Pull request: no
Labels: name: Feature request

### Feature request

When setting the flag `group_by_length=True` on the TrainingArguments, there is no user feedback of the operations running in the background, namely getting the list of lengths for all the samples and running the grouping algorithm. This can be a frustrating problem when dealing with large datasets (Millions of samples) on slow IO devices, since it appears that the Trainer is hanging and does not start!

More precisely, In my current setup, I found out that the following lines take almost 2h to finish. (Due to my slow IO (reading from a NFS from an old machine))

https://github.com/huggingface/transformers/blob/c817c17dbe264329b9f9d227b48ce70edd9e3204/src/transformers/trainer_pt_utils.py#L585

NOTE 1): using `.select_columns(model_input_name)` and then iterating would not be faster? Assuming that the dataset has more feature like "attention_mask" for instance. 

I believe that more feedback could possibly be given to the user, like the time that would take to finish. (Also store the dataset length under .cache).

NOTE 2): After realising this issue, I also noticed the `length_column_name` flag. Maybe raising a warning to let the users know that on larger datasets they should precompute the length. By doing so, the time went from 2h to (15-20)min.

### Motivation

I was training a model on a LM task. My dataset has 22M samples with average length of +/- 512. When I run the model with `group_by_length=True` I thought that something was wrong because the training was not starting (I was actually writing an bug about my problem, because I thought it was an issue with the Trainer). After further inspection, I notice that the main culprit was the computation of the length that is really slow on my current setup. 

### Your contribution

If you feel like this is an issue that is worth to address, I am willing to do PR under your orientation.

Comments:
cc @muellerzr @pacman100
custom 4d attention_mask as transformers .forward() argument
Author: poedator
Pull request: no

### Feature request

somewhere inside transformers models, 2d masks are converted into 4d. I want to be able to pass my own custom 4d mask to  .forward().
Presently it causes error. 
CODE EXAMPLE:
```
model_name = "openlm-research/open_llama_3b"
model = transformers.AutoModelForCausalLM.from_pretrained(model_name, device_map=device)
# preparing KV cache
size0 = 5
max_token = 10000
x0 = torch.randint(max_token, (1, size0), device=device)
y0 = model.forward(x0, )

# forward with mask
size1 = 3
x1 = torch.randint(max_token, (1, size1), device=device)
mask_shape = (1, 1, size0, size1)  # bsz, head_dim=1, query_length, key_value_length
custom_mask = torch.randint(2, mask_shape, device=device)

model.forward(input_ids=x1, attention_mask=custom_mask, past_key_values=y0['past_key_values'])
# expected forward with this custom_mask
```
Error msg: 
```
...
File .../transformers/src/transformers/modeling_attn_mask_utils.py:154, in AttentionMaskConverter._expand_mask(mask, dtype, tgt_len)
    149 @staticmethod
    150 def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
    151     """
    152     Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
    153     """
--> 154     bsz, src_len = mask.size()
    155     tgt_len = tgt_len if tgt_len is not None else src_len
    157     expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)

ValueError: too many values to unpack (expected 2)
```

### Motivation

need custom 4d mask for experiments with causal inference.

### Your contribution

I am ready to get involved, with HF guidance.
tagging @patrickvonplaten who recently authored #27086 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

UPD: this feature gets discussed and implemented in https://github.com/huggingface/transformers/pull/27539
chatglm3-6b-32k does not support Flash Attention 2.0
Author: abc123456cxx
Pull request: no

When I use **AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True, load_in_8bit= True,use_flash_attention=True,device_map="auto")**, it occers:
ValueError: The current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new

Comments:
Hi @abc123456cxx, thanks for raising this. 

Is this a request to add flash attention to the model? If so, the [modeling files live on the hub](https://huggingface.co/THUDM/chatglm3-6b-32k/tree/main). Could you open an issue there to request this feature? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Safetensor boolean is wrong in modeling utils
Author: ylacombe
Pull request: no

# Description

In [`load_pretrained_model`](https://github.com/huggingface/transformers/blob/3258ff93304078b9e27d752e6c19d3813f664855/src/transformers/modeling_utils.py#L3573), the `is_safetensors` boolean doesn't seem to work as expected.

To be more specific, `is_safetensors` is defined at the [beginning of the method](https://github.com/huggingface/transformers/blob/3258ff93304078b9e27d752e6c19d3813f664855/src/transformers/modeling_utils.py#L3591-L3599C13):  
```python
        is_safetensors = False
        ...


        if device_map is not None and "disk" in device_map.values():
            archive_file = (
                resolved_archive_file[0] if isinstance(resolved_archive_file, (list, tuple)) else resolved_archive_file
            )
            is_safetensors = archive_file.endswith(".safetensors")
``` 

As you can see, `is_safetensors` is updated only if `device_map is not None and "disk" in device_map.values()`. The second condition is really specific. Given the name of the variable `is_safetensors`, I would have expected it to be `True` independently from `device_map.values()`.

This boolean is used a few times throughout the method. To name but a few, [here](https://github.com/huggingface/transformers/blob/3258ff93304078b9e27d752e6c19d3813f664855/src/transformers/modeling_utils.py#L3795), [here](https://github.com/huggingface/transformers/blob/3258ff93304078b9e27d752e6c19d3813f664855/src/transformers/modeling_utils.py#L3835C26-L3835C26) and [here](https://github.com/huggingface/transformers/blob/3258ff93304078b9e27d752e6c19d3813f664855/src/transformers/modeling_utils.py#L3883).

Just wanted to raise a warning here, just in case it is not the expected behaviour !

If this is unexpected, I can open a quick PR to correct it, if needed. If not, maybe we should rename the boolean ?

----------------

cc @LysandreJik and @Narsil 






--------------

BTW, this didn't raises any error, so I'm not using the classic bug reporting template

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
bus error when load codellama2 34b
Author: boundles
Pull request: no

### System Info

when I load model on 2*A6000 45G, bus error occurs

`model_path = 'CodeLlama-34b-hf'
model = AutoModelForCausalLM.from_pretrained(model_path, load_in_8bit=True, device_map="auto", trust_remote_code=True)
print('finish load model')
`

looking for your reply, thanks

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run the code

### Expected behavior

load successfully

Comments:
Hi @boundles, thanks for raising an issue. 

So that we can best help you, can you make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and include all important information such as: 
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* A minimal code snippet we can run to reproduce the error
* Full details of the error encountered, including full traceback

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Segmentation fault when initializing training arguments with imported tensorflow
Author: zeionara
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: Linux-6.2.0-33-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.2
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0 (True)
- Tensorflow version (GPU?): 2.14.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@muellerzr @pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import tensorflow as tf

from transformers import TrainingArguments

print('Listing tf gpus...')

print(tf.config.list_physical_devices("GPU"))

print('Initializing training arguments...')

args = TrainingArguments(output_dir="output")

print('Initialization has completed')
```

### Expected behavior

Expected output:

```
Listing tf gpus...
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]
Initializing training arguments...
Initialization has completed
```

Observed output:
```
Listing tf gpus...
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]
Initializing training arguments...
Segmentation fault (core dumped)
```

Comments:
cc @Rocketknight1 as this might be a TF issue 

Hi @zeionara, I ran that code here on TF 2.14 and did not get a segfault. Can you try replacing `TrainingArguments` with `TFTrainingArguments` to see if the issue persists? The `TrainingArguments` class is usually used for Torch examples, and might do some torch initializations, which could cause a segfault if Torch is incorrectly configured and TF has allocated GPU memory already.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.