Bug in new version transformers 4.34.0-4.36.2
Author: JAX627
Pull request: no

### System Info

ver: transformers 4.34.0-4.36.2
problem: finetune chatglm3 model, finetune.py don't generate pytorch_model.bin file in output, as point out in https://github.com/THUDM/ChatGLM3/discussions/253#discussioncomment-7837093
it seems like problems in modeling_utils.py file, and it can be solved by pip install transformers==4.33.0, seems like higher version transformers not suitable for chatglm3 totally


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. download chatglm3-6b-32k model
2. pip install transformers 4.34.0-4.36.2
3. follow finetune steps in https://github.com/THUDM/ChatGLM3/tree/main/finetune_chatmodel_demo
4. after finish finetuning, there is no pytorch_model.bin file in output dir
5. pip install transformers==4.33.0
6. follow finetune steps in https://github.com/THUDM/ChatGLM3/tree/main/finetune_chatmodel_demo
7. after finish finetuning, there is the pytorch_model.bin file in output dir

### Expected behavior

solve the problem in new version transformers

Comments:
Hi @JAX627, thanks for opening an issue! 

By default, a `pytorch_model.bin` file will no longer be saved out as `safetensors` is now the default format. To save a pytorch file out instead, you can explicitly use the `safe_serialization` argument when calling `save_pretrained`:

```py
model.save_pretrained(checkpoint_name, safe_serialization=False)
``` 

pytorch 2.1.2, tokenizer 0.14.1, transformer 4.36.2. 

Using 

```
model.save_pretrained(checkpoint_name, safe_serialization=False)
```

The `pytorch_model.bin` is still missing. 

generated files

```
model.safetensors  optimizer.pt  rng_state.pth  scheduler.pt  trainer_state.json  training_args.bin
```


Issue resolved. The problem is that when constructing the trainer, `save_safetensors=False` should be set. Otherwise, the above `safe_serialization=False` will not work.  

https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/trainer#transformers.TrainingArguments.save_safetensors

Happy New Year!

@WilliamYi96 + amyeroberts
Many thanks for your information, helps a lot! 
I'm having the same issue when fine-tuning whisper models.
Do you maybe know: 
 1.  Does 'save_safetensors=False' also work for older transformers<4.34 - i.e. is it simply ignored or does it cause an error?
 2. Is there any possibility (tool, converter, code) to extract the (original) pytorch_model.bin from a model.safetensors file??
 many thanks for any hint!
   

Hi @welliX, happy new year! 

1. Yes, it works for older versions of transformers. It was added as an argument to `save_retrained` in #19175, which was part of v4.23. However, its first iterations were as an experimental feature and not guaranteed to be bug free. We strongly advise using the most recent versions of transformers, as many different issues in the saving and serialization of the models have been ironed out since then. 

2. You can simply load a model that was saved in safetensors and save it out again: 
```py
from transformers import LlavaForConditionalGeneration

checkpoint = "llava-hf/llava-1.5-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(checkpoint)

model.save_pretrained('new_llava_model', safe_serialization=False)
```

Dear Amy, 
many thanks for your speedy response; this was quite helpful!
I tried your proposal 2. and  in principle it does s.th., i.e. it produces following directory ;-) 
However with 6 *huuuge* pytorch*bin files:

>    1024 -rwxr-xr-x 1 akiessling users       1746 Jan  3 16:00 config.json*
>    1024 -rwxr-xr-x 1 akiessling users       3531 Jan  3 16:00 generation_config.json*
> 4766720 -rwxr-xr-x 1 akiessling users 4880137238 Jan  3 16:01 pytorch_model-00001-of-00006.bin*
> 4744192 -rwxr-xr-x 1 akiessling users 4857219539 Jan  3 16:03 pytorch_model-00002-of-00006.bin*
> 4744192 -rwxr-xr-x 1 akiessling users 4857219603 Jan  3 16:06 pytorch_model-00003-of-00006.bin*
> 4744192 -rwxr-xr-x 1 akiessling users 4857219603 Jan  3 16:08 pytorch_model-00004-of-00006.bin*
> 4744192 -rwxr-xr-x 1 akiessling users 4857219603 Jan  3 16:11 pytorch_model-00005-of-00006.bin*
> 3851264 -rwxr-xr-x 1 akiessling users 3942840241 Jan  3 16:13 pytorch_model-00006-of-00006.bin*
>    1024 -rwxr-xr-x 1 akiessling users      70128 Jan  3 16:13 pytorch_model.bin.index.json*

Is this what as expected?
So far my fine-tuning delivered a model dir/ with only a single file (and smaller) 'pytorch_model.bin' ,
which I than used directly for inference (following transcribe_audio.py from https://github.com/vasistalodagala/whisper-finetune).
Are all the six files necessary for inference? Can they be merged into one single pytorch_model.bin ?

many thanks again for your help!
    /Andi



Hi amyeroberts,
I've tried to load/use one of the 6 pytorch_model-*.bin  just as before the pytorch_model.bin model for infrerence - of course it is not working:

  File "/home/run/icsf/icsf_inference/index.py", line 618, in loadASRmodel
    ASRtranscriber=ASRinf.Init_ASRinference("openai/whisper-tiny", icsfitems['ASR_Activemodel'], 'English', 0)
  File "/home/run/icsf/icsf_inference/./whisper/ASRinference.py", line 48, in Init_ASRinference
    transcribe = loadmodel(args.ckpt_dir, args.device, tokenizer)
  File "/home/run/icsf/icsf_inference/./whisper/ASRinference.py", line 25, in loadmodel
    t = pipeline(task="automatic-speech-recognition",
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/__init__.py", line 741, in pipeline
    config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 1039, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 734, in __getitem__
    raise KeyError(key)
KeyError: 'llava'

Probably it's the better idea to use the (new) model.safetensors file directly for inference. I guess I have to use s.th. like
 use_safetensors=True
in the pipeline() function call, right? Or do I have to switch this somewhere else?

In my last try some weeks ago when doing that I got an error 
TypeError: AutomaticSpeechRecognitionPipeline._sanitize_parameters() got an unexpected keyword argument 'use_safetensors'

maybe it was because of a (still) older transformer version?

Kind regards, Andi



Hi @welliX  

>I've tried to load/use one of the 6 pytorch_model-*.bin just as before the pytorch_model.bin model for infrerence - of course it is not working:

just from the error messages, it looks like you're trying to load a llava model into the ASR pipeline. Llava is a vision-language model and so isn't compatible with this task. The llava checkpoint I provided in my example was just for demonstration. You'll want to use the checkpoint for whichever ASR model you were trying to get the pytorch files for.

> Probably it's the better idea to use the (new) model.safetensors file directly for inference. 

yes

> I guess I have to use s.th. like use_safetensors=True in the pipeline() function call, right? Or do I have to switch this somewhere else?

That shouldn't be necessary. Using safetensors is the default behaviour in the transformers library

> In my last try some weeks ago when doing that I got an error ... 
> maybe it was because of a (still) older transformer version?

I really couldn't tell you without know the code you were running and the versions. However, I suspect you'd get the same error now. The error is being raised in `_sanitze_parameters` which is a method responsible for allocating kwargs to different parts of the pipeline. This is because passing kwargs to the pipeline can be ambiguous - what is this argument supposed to be configuring? If it's raising an error, it means the pipeline doesn't know how to use the argument. In general, if you want to start customizing parts of the pipeline, it might be easiest to work with the code more directly e.g. [in this ASR example](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperForCausalLM.forward.example). 

Hi @amyeroberts,
many thanks for your detailed answer - yes, you are right, I'll try the (new) way via 
model.safetensors file now.
Hope it'll work out.

Another question: The whisper models are pretty huge. 
Thus when loading more than one model (for switching between different inference models) it often happens that CUDA is out of memory (see example below). 
Do you know perhaps an easy way how to free the CUDA memory of an earlier used model before loading a new one? 
I found the following but it doesn't really hAve a big effect:
  import torch, gc
  gc.collect()
  torch.cuda.empty_cache()

kind regards!

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 7.75 GiB of which 30.19 MiB is free. Process 71391 has 3.39 GiB memory in use. Including non-PyTorch memory, this process has 1.95 GiB memory in use. Of the allocated memory 1.80 GiB is allocated by PyTorch, and 2.84 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF


Hi @welliX, these are questions best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.