Repository: huggingface/diffusers

Issue:
AttributeError: 'CLIPTextModel' object has no attribute 'peft_config' when using default train_dreambooth_lora_sdxl_advanced.py but with checkpointing_steps < max_train_steps
### Describe the bug

When checkpointing at defined steps, such as every 50 steps I get a 'CLIPTextModel' object has no attribute 'peft_config'  while trying to save the model. This only happens when checkpointing_steps < max_train_steps, as the code wouldn't run if it was higher than the train steps

### Reproduction

python train_dreambooth_lora_sdxl_advanced.py --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" --dataset_name="./trainimages/" --caption_column=prompt --instance_prompt="a TOK person" --validation_prompt="a TOK person dressed as Yoda" --output_dir="huggy_v11" --caption_column="prompt" --mixed_precision="bf16" --resolution=1024 --train_batch_size=1 --repeats=1 --gradient_accumulation_steps=1 --gradient_checkpointing --train_text_encoder_ti --learning_rate=1 --text_encoder_lr=1 --lr_scheduler="constant" --snr_gamma=5.0 --lr_warmup_steps=0 --rank=32 --max_train_steps=1000 --checkpointing_steps=50 --seed="0" --optimizer="prodigy" --prodigy_safeguard_warmup=True --prodigy_use_bias_correction=True --adam_beta1=0.9 --adam_beta2=0.99 --adam_weight_decay=0.01

### Logs

```shell
Traceback (most recent call last):
  File "C:\Users\\Documents\GitHub\\train_dreambooth_lora_sdxl_advanced.py", line 2100, in <module>
    main(args)
  File "C:\Users\\Documents\GitHub\\train_dreambooth_lora_sdxl_advanced.py", line 1895, in main
    accelerator.save_state(save_path)
  File "C:\Users\\Documents\GitHub\\venv\lib\site-packages\accelerate\accelerator.py", line 2793, in save_state
    hook(self._models, weights, output_dir)
  File "C:\Users\\Documents\GitHub\\train_dreambooth_lora_sdxl_advanced.py", line 1320, in save_model_hook
    get_peft_model_state_dict(model)
  File "C:\Users\\Documents\GitHub\\venv\lib\site-packages\peft\utils\save_and_load.py", line 64, in get_peft_model_state_dict
    config = model.peft_config[adapter_name]
  File "C:\Users\\Documents\GitHub\\venv\lib\site-packages\torch\nn\modules\module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'CLIPTextModel' object has no attribute 'peft_config'
```


### System Info

- `diffusers` version: 0.25.0.dev0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.16
- PyTorch version (GPU?): 2.3.0.dev20240102+cu121 (True)
- Huggingface_hub version: 0.20.1
- Transformers version: 4.37.0.dev0
- Accelerate version: 0.23.0
- xFormers version: 0.0.21
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_


Discussion:
I am also experiencing this issue
Did you install `peft`? 

Cc: @linoytsaban 
> Did you install `peft`?
> 
> Cc: @linoytsaban

Yes, and the issue still persists.
Just ran into this issue as well using the script after 3 hours of training. Looks like it failed at the first checkpoint.

```
#!/usr/bin/env bash
!accelerate launch train_dreambooth_lora_sdxl_advanced.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" \
  --dataset_name="./training_data" \
  --output_dir="father_lora_v1" \
  --cache_dir="./dataset_cache_dir" \
  --caption_column="prompt" \
  --mixed_precision="fp16" \
  --instance_prompt="a photo of a TOK man" \
  --resolution=1024 \
  --train_batch_size=2 \
  --gradient_accumulation_steps=4   \
  --gradient_checkpointing \
  --snr_gamma=5.0 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1700 \
  --checkpointing_steps=717 \
  --validation_prompt="A photo of TOK in a suit" \
  --validation_epochs=50 \
  --learning_rate=1 \
  --text_encoder_lr=1 \
  --train_text_encoder_ti \
  --train_text_encoder_ti_frac=0.5 \
  --token_abstraction="TOK" \
  --num_new_tokens_per_abstraction=2 \
  --with_prior_preservation \
  --class_data_dir="./dataset" \
  --num_class_images=150 \
  --class_prompt="a photo of a man" \
  --rank=32 \
  --optimizer="prodigy" \
  --prodigy_safeguard_warmup=True \
  --prodigy_use_bias_correction=True \
  --adam_beta1=0.9 \
  --adam_beta2=0.99 \
  --adam_weight_decay=0.01 \
  --seed="0"
```

peft                              0.7.1


I will add that I did not run into this issue with an earlier run that did not use `train_text_encoder_ti` flag.

Resolving patch:
diff --git a/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py b/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py
--- a/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py
+++ b/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py
@@ -1316,13 +1316,15 @@ def save_model_hook(models, weights, output_dir):
                 if isinstance(model, type(accelerator.unwrap_model(unet))):
                     unet_lora_layers_to_save = convert_state_dict_to_diffusers(get_peft_model_state_dict(model))
                 elif isinstance(model, type(accelerator.unwrap_model(text_encoder_one))):
-                    text_encoder_one_lora_layers_to_save = convert_state_dict_to_diffusers(
-                        get_peft_model_state_dict(model)
-                    )
+                    if args.train_text_encoder:
+                        text_encoder_one_lora_layers_to_save = convert_state_dict_to_diffusers(
+                            get_peft_model_state_dict(model)
+                        )
                 elif isinstance(model, type(accelerator.unwrap_model(text_encoder_two))):
-                    text_encoder_two_lora_layers_to_save = convert_state_dict_to_diffusers(
-                        get_peft_model_state_dict(model)
-                    )
+                    if args.train_text_encoder:
+                        text_encoder_two_lora_layers_to_save = convert_state_dict_to_diffusers(
+                            get_peft_model_state_dict(model)
+                        )
                 else:
                     raise ValueError(f"unexpected save model: {model.__class__}")
 
@@ -1335,6 +1337,8 @@ def save_model_hook(models, weights, output_dir):
                 text_encoder_lora_layers=text_encoder_one_lora_layers_to_save,
                 text_encoder_2_lora_layers=text_encoder_two_lora_layers_to_save,
             )
+        if args.train_text_encoder_ti:
+            embedding_handler.save_embeddings(f"{output_dir}/{args.output_dir}_emb.safetensors")
 
     def load_model_hook(models, input_dir):
         unet_ = None


Test patch: