noamaanMulla-03/transformers-issues huggingface dataset:
Zero classification pipeline issue when using device map
Author: BramVanroy
Pull request: no

### System Info

Google Colab

- `transformers` version: 4.32.1
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.14

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using the `device_map` argument in a zero shot pipeline with BERT, I get an error that the model cannot be loaded.

```python
pipeline("zero-shot-classification", model="LoicDL/bert-base-dutch-cased-finetuned-snli", device_map="auto")
```

Error:

> ValueError: Could not load model LoicDL/bert-base-dutch-cased-finetuned-snli with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForSequenceClassification'>, <class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForSequenceClassification'>, <class 'transformers.models.bert.modeling_bert.BertForSequenceClassification'>, <class 'transformers.models.bert.modeling_tf_bert.TFBertForSequenceClassification'>).

I think the issue is that BertForSequenceClassification does not support device_map:

```python
BertForSequenceClassification.from_pretrained("LoicDL/bert-base-dutch-cased-finetuned-snli", device_map="auto")
```

Error:

> ValueError: BertForSequenceClassification does not support `device_map='auto'`. To implement support, the modelclass needs to implement the `_no_split_modules` attribute.

It would be much clearer if this kind of error message is shown in the pipeline. The current error message is very vague.

### Expected behavior

A better error message, like the one that is triggered when trying to load the model for sequence classification.

Comments:
Hi @BramVanroy We can definitely provide a more detailed (original) traceback. However, the pipeline tries to load with different possible model class (here `4`) and it's not clear what's the best way to display. Put the traceback given by these 4 tries would be quite long I am afraid.

In terms of error traces, it's probably better to have "too much" than "too little". So given a reason for each of the architectures why they failed seems a good thing, not a bad one!

Also the error seems to be the same as #25701, so @ydshieh let's indeed update the error message! 

Sure Sir!
Better error message for pipeline loading
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #25892

The new error looks like:

<img width="1186" alt="Screenshot 2023-09-01 153455" src="https://github.com/huggingface/transformers/assets/2521628/33a8695e-020f-43c9-9763-45d083b85930">




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Feel a bit bloated but it it helps ;)
Example to build composite MusicgenForConditionalGeneration does not work
Author: osanseviero
Pull request: no

### System Info

Last transformers version


### Who can help?

@sanchit-gandhi @Vaibhavs10 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Doc from https://huggingface.co/docs/transformers/main/en/model_doc/musicgen#model-structure

Code snippet

```
from transformers import AutoConfig, AutoModelForTextEncoding, AutoModel, MusicgenForCausalLM, MusicgenForConditionalGeneration

text_encoder = AutoModelForTextEncoding.from_pretrained("t5-base")
audio_encoder = AutoModel.from_pretrained("facebook/encodec_32khz")
decoder_config = AutoConfig.from_pretrained("facebook/musicgen-small").decoder
decoder = MusicgenForCausalLM.from_pretrained("facebook/musicgen-small", **decoder_config)
```

### Expected behavior

.

Comments:
Hi @osanseviero 

Thank you for opening the issue 🤗 

This is probably just a doc issue: the following
```python3
decoder_config = AutoConfig.from_pretrained("facebook/musicgen-small").decoder
decoder = MusicgenForCausalLM.from_pretrained("facebook/musicgen-small", config=decoder_config)
```
works - but with a lot of weights newly initialized, ~~I am not sure what this comes from though.~~

OK, we are loading a `MusicgenForConditionalGeneration` checkpoint into a `MusicgenForCausalLM`. This won't work between these two types as the way they implemented, unfortunately.

 @sanchit-gandhi @Vaibhavs10 Could you take a look too ? Thanks.

Well, it's not va, it's @Vaibhavs10 

Thanks for raising this @osanseviero, Just tested this. This is broken atm.
A couple of points:
1. First of all, this should not be displayed in the docs since we don't support fine-tuning yet. So there is no reason for someone to load the individual models. At least none that are obvious to me.
2. The error is specifically in when we load the decoder with `MusicgenForCausalLM` and the decoder_config.

I recommend we remove the snippet and address this error with the fine-tuning bit.

Thoughts @sanchit-gandhi?

Friendly ping here

Yep think we can just remove what's not required from the docs for now - until we have fine-tuning support these are extremely rare operations.
TrainingArguments frozen: leads to issues in Trainer
Author: BramVanroy
Pull request: no

### System Info

- `transformers` version: 4.32.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)

### Who can help?

@muellerz and @pacman100

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I recently encountered an annoying issue when trying to use the wandb integration for hyperparameter search. When doing so, the trainer will try to reset some of its `args` (TrainingArguments). But it seems that since [recently](https://github.com/huggingface/transformers/commit/ca51499248b986ebf3991848234ef2d8bc81a36a), the choice was made to freeze training arguments.

So this won't work.

```python
from transformers import TrainingArguments
args = TrainingArguments(output_dir="test")
args.output_dir = "test_dir"
args.run_name = "hello"
```

### Expected behavior

As a whole I am not a fan of this commit to freezing a dataclass like that - which is not done in the canonical Python way of using `@dataclass(frozen=True)` so it is hard to spot if you're reading to code. 

If it _must_ stay, then some more changes need to be made throughout the lib. Such as this line

https://github.com/huggingface/transformers/blob/0f08cd205a440d23e6bf924cddd73ff48e09fe35/src/transformers/integrations/integration_utils.py#L758

Should probably be

```python
args = dataclasses.replace(args, run_name=None)
```

(untested)

Currently I do not have the time for a PR but this could be a good first issue for someone who would like to contribute.

Comments:
cc @muellerzr 

Tangentially related: The frozen TrainingArguments also affects third parties facing crashes as they enforce some `TrainingArguments` arguments like `remove_unused_columns`:
https://github.com/tomaarsen/SpanMarkerNER/pull/27
https://github.com/huggingface/trl/pull/676

- Tom Aarsen

It also seems to be causing some issues in `qlora` https://github.com/artidoro/qlora/issues/253 

You are all very right, this would be a good example of "paved with good intentions." From a design perspective when I was talking with another core contributor, we agreed that the `TrainingArguments` weren't really ever meant to be a mutable object. As a result, we made the move to make it fully immutable. However since many libraries relied on this unintentional usage (as we're now finding out), it now doesn't make sense to keep it. I'll be putting in a revert on this here tomorrow, and if you've already migrated some code you have a few versions before the broken version is fully gone.  Really sorry for the trouble 🙏
Revert frozen training arguments
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR reverts a recent change that made the `TrainingArguments` immutable objects. While this may have been our *intended* design decision, as users were already integrated with modifying the training_args directly, we've decided to revert this change.

Users code (if already modified to deal with this issue) does not actually need changing, since it uses dataclass utils under the hood. However they can change any cases of:

```python
training_args = dataclasses.replace(training_args, item_a=value_a)
```

With:
```python
training_args.item_a = value_a
```

Fixes # (issue)

Solves https://github.com/huggingface/transformers/issues/25897

Linking a few known repos/issues/PR's that reference this:

- https://github.com/huggingface/trl/pull/676
- https://github.com/huggingface/optimum-intel/pull/412
- https://github.com/artidoro/qlora/issues/253
- https://github.com/tomaarsen/SpanMarkerNER/pull/27

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker there aren't per-se any known bugs, but I can include a comment on why we're keeping it *unfrozen*
FSDP optimizer not loaded correctly
Author: howard-yen
Pull request: no

### System Info

- `transformers` version: 4.33.0.dev0
- Platform: Linux-4.18.0-477.15.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes, with FSPD 

### Who can help?

@pacman100 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use run_clm.py (https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py) to train a large model using the HuggingFace Trainer, use FSDP and save checkpoints. For example:

```
torchrun --nproc_per_node=4 --master_port=XXXXX experiments/run_clm.py \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --dataset_name openwebtext \
    --streaming \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 1 \
    --do_train \
    --max_steps 1000 \
    --output_dir output_dir/ \
    --block_size 512 \
    --save_steps 10 \
    --save_total_limit 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap "LlamaDecoderLayer" \
    --bf16 True \
```

2.    Kill training after a checkpoint has been saved. Then, resume training from the checkpoint with the resume_from_checkpoint training argument.

4.   Observed behavior: optimizer state is not loaded


### Expected behavior

Optimizer state should be loaded. 

The current code checks if a scheduler was saved in the directory: 
https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/trainer.py#L2444
but the current save_checkpoint method does not save the `self.lr_scheduler.state_dict()`:
https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/trainer.py#L2334

Also, currently `accelerate.utils.save_fsdp_optimizer` saves the optimizer as `optimizer.bin`
https://github.com/huggingface/accelerate/blob/a16b843a1b8626d571e62f7301e3a478be2c04b4/src/accelerate/utils/fsdp_utils.py#L142C75-L142C75

but the trainer checks for `optimizer.pt` so `load_fsdp_optimizer` isn't called.

Comments:
[MMS] Update docs with HF TTS implementation
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Following #24085, Transformers now contains an implementation of MMS-TTS. This PR updates the MMS docs to include a code-snippet using this implementation.

cc @osanseviero since this was something you flagged before, and also @Vaibhavs10 for viz


Comments:
_The documentation is not available anymore as the PR was closed or merged._
run_clm_no_trainer.py example - problem with most recent checkpoint loading
Author: TomerRonen34
Pull request: no

The example has code for finding the latest checkpoint, but accelerator.load_state isn't called.
https://github.com/huggingface/transformers/blob/1baeed5bdf3c58b723a6125632567f97bdf322c6/examples/pytorch/language-modeling/run_clm_no_trainer.py#L561C15-L561C15

Comments:
Hi @TomerRonen34, thanks for raising this issue! 

Can you make sure to follow the issue template and include: 
* A reproducible code snippet
* Details of the expected and observed behaviour including the full traceback if it exists
* Information about the running environment: run `transformers-cli env` in the terminal and copy-paste the output

Thanks, this has now been fixed via #25318 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MNLI benchmark, I see only only eval_accuracy, not seperated to m/mm = matched/mismatched
Author: ndvbd
Pull request: no

### System Info

N/A

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

MNLI benchmark, I see only only eval_accuracy, not seperated to m/mm = matched/mismatched

### Expected behavior

MNLI benchmark, I see only only eval_accuracy, not seperated to m/mm = matched/mismatched

Comments:
I have no Idea what you are talking about

Hi Arthur, the GLUE, for the MNLI dataset, has *two* metrics of accuracy: match accuracy, and mismatched accuracy. Kindly see here: https://openreview.net/pdf?id=rJ4km2R5t7

Now, when running the tranformer code run_gleu, for the mnli task, it only reports a single 'accuracy'.

We have at least 5 different files called `run_glue`, which are supposed to show examples of how to perform certain tasks, but are not absolute source of truth. If you don't see the metric you want being used, you should be able to adapt the code pretty easily for your needs.

As the doc mentions about the example folder: 
>  It is expected that they won’t work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs. To help you with that, most of the examples fully expose the preprocessing of the data. This way, you can easily tweak them.
Request support for RWKV-4-World model.
Author: SetoKaiba
Pull request: no
Labels: name: New model

### Model description

As RWKV-4-World is using a different tokenizer and vocabs, the current RWKV support in transformers is incompatible.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/StarRing2022/RWKV-4-World-1.5B
@StarRing2022

Comments:
@sgugger Is it possible to add the support for it? I'd like to use PEFT to fine tune the model.

We have implemented the function of fine-tuning the RWKV-World model using the peft library, and we will immediately publish it in HF

The relevant code will be provided under this issue

Please refer to：
https://github.com/StarRing2022/HF-For-RWKVWorld-LoraAlpaca
https://huggingface.co/StarRing2022/RWKV-4-World-7B

> Please refer to： https://github.com/StarRing2022/HF-For-RWKVWorld-LoraAlpaca https://huggingface.co/StarRing2022/RWKV-4-World-7B

@StarRing2022 Thank you for your great work.
Will you make a PR for it? Or will you just keep it a separate repo?
I'd like to see it PR back to hf transformers. And others can get your work easily.

https://github.com/StarRing2022/RingRWKV
I think this should be mentioned in the example github repo as well.

cc @younesbelkada 

Thanks，there are indeed some issues with the RWKV in HF format on the official transformers. Recently, we have found that there are issues such as CFG and sample_ Logits, PR is necessary, and we also hope that the official can provide improvements

I think that hf transformers is working with RWKV world ,just use the ringrwkv tokenizer(or the world tokenizer in the official repo) and it will work

Yes,We are trying to contact with HF transformers
problem
Author: Zlasejd
Pull request: no

### System Info

  File "/fsa/home/hqz_zhangjd/.conda/envs/newenv/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 986, in forward
    buffered_token_type_ids_expanded = buffered_token_type_ids.expand(batch_size, seq_length)
RuntimeError: The expanded size of the tensor (693) must match the existing size (512) at non-singleton dimension 1.  Target sizes: [16, 693].  Tensor sizes: [1, 512]

  2%|▏         | 2738/146754 [12:51<11:16:06,  3.55it/s]
 Every time I run  the file 'run_mlm_wwm.py' reported an error when it was used. This issue has not been resolved and the version has also been changed. We are currently using Torch 2.0.1 and Transformers 4.28.1. Can you help us solve it?

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

  File "/fsa/home/hqz_zhangjd/.conda/envs/newenv/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 986, in forward
    buffered_token_type_ids_expanded = buffered_token_type_ids.expand(batch_size, seq_length)
RuntimeError: The expanded size of the tensor (693) must match the existing size (512) at non-singleton dimension 1.  Target sizes: [16, 693].  Tensor sizes: [1, 512]

  2%|▏         | 2738/146754 [12:51<11:16:06,  3.55it/s]
 Every time I run  the file 'run_mlm_wwm.py' reported an error when it was used. This issue has not been resolved and the version has also been changed. We are currently using Torch 2.0.1 and Transformers 4.28.1. Can you help us solve it?

### Expected behavior

solve this problem.

Comments:
RuntimeError: The expanded size of the tensor (20) must match the existing size (101) at non-singleton dimension 0.  Target sizes: [20].  Tensor sizes: [101] 
Author: avnerbarr
Pull request: no

My goal is to be able to pass 100 (or 300) tokens (or some other number) as input and get back at most 16 additional tokens (or some other number I can control).


Code is strait forward:

```
vocab_size = model.config.vocab_size
input_ids = torch.randint(vocab_size, (1, 300), dtype=torch.long, device=device)
model.generate(
            input_ids,
            language_ids=language_ids,
            do_sample=False,
            num_beams=num_beams,
            max_length=max_length,
            repetition_penalty=1.6,
            pad_token_id=tokenizer.eos_token_id,
            num_return_sequences=num_sequences
        )
```

Can you elaborate on the semantics of what "max_length" refers to?

Per the documentation:

https://huggingface.co/transformers/main_classes/configuration.html#transformers.PretrainedConfig

**Parameters for sequence generation**

> max_length (int, optional, defaults to 20) – Maximum length that will be used by default in the generate method of the model.

If I am doing something wrong, please point me to how to control the behavior in the correct way 😄 

Side note:
I also played with the model config (so many tests and various messages):

```
model.config.max_length = 16 # other numbers were tried as well
```


Depending on the setup I could also get a message like this (same `generate` call as above):

> Input length of input_ids is 300, but ``max_length`` is set to 20.This can lead to unexpected behavior. You should consider increasing ``config.max_length`` or ``max_length``

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Have you solved this problem?