Repository: huggingface/diffusers
Pull request: Update train_dreambooth_lora_sd3.py
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #8594 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [ ] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [ ] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/c/discussion-related-to-httpsgithubcomhuggingfacediffusers/63)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/diffusers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sayakpaul 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @.

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Core library:

- Schedulers: @yiyixuxu 
- Pipelines:  @sayakpaul @yiyixuxu @DN6
- Training examples: @sayakpaul 
- Docs: @stevhliu and @sayakpaul
- JAX and MPS: @pcuenca
- Audio: @sanchit-gandhi
- General functionalities: @sayakpaul @yiyixuxu @DN6

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- transformers: [different repo](https://github.com/huggingface/transformers)
- safetensors: [different repo](https://github.com/huggingface/safetensors)

-->


Problem statement:
UnboundLocalError: tokenizer_one referenced before assignment in train_dreambooth_lora_sd3.py
### Describe the bug

I encountered an issue when trying to run train_dreambooth_lora_sd3.py. The script throws an UnboundLocalError because tokenizer_one is referenced before it is assigned. This prevents the training process from starting.

### Reproduction

Clone the repository.
Set up the environment as per the instructions.
Run the following command:

```
accelerate launch train_dreambooth_lora_sd3.py \
    --pretrained_model_name_or_path=$MODEL_NAME \
    --instance_data_dir=$INSTANCE_DIR \
    --output_dir=$OUTPUT_DIR \
    --mixed_precision="fp16" \
    --instance_prompt="a photo of sks dog" \
    --resolution=1024 \
    --train_batch_size=1 \
    --gradient_accumulation_steps=4 \
    --learning_rate=1e-5 \
    --lr_scheduler="constant" \
    --lr_warmup_steps=0 \
    --max_train_steps=500 \
    --seed="0"
```

Expected Behavior:
The script should initialize the tokenizers and proceed with the training process without any errors.

Actual Behavior:
The script fails with the following error:

```UnboundLocalError: local variable 'tokenizer_one' referenced before assignment```


### Logs

```shell
accelerate launch train_dreambooth_lora_sd3.py   --pretrained_model_name_or_path=$MODEL_NAME    --instance_data_dir=$INSTANCE_DIR   --output_dir=$OUTPUT_DIR   --mixed_precision="fp16"   --instance_prompt="a photo of sks dog"   --resolution=1024   --train_batch_size=1   --gradient_accumulation_steps=4   --learning_rate=1e-5   --lr_scheduler="constant"   --lr_warmup_steps=0   --max_train_steps=500   --seed="0"
/workspace/diffusers/src/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: `Transformer2DModelOutput` is deprecated and will be removed in version 1.0.0. Importing `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead.
  deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
06/16/2024 23:42:29 - INFO - __main__ - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: fp16

You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
You are using a model of type t5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:15<00:00,  7.69s/it]
Traceback (most recent call last):
  File "/workspace/diffusers/examples/dreambooth/train_dreambooth_lora_sd3.py", line 1686, in <module>
    main(args)
  File "/workspace/diffusers/examples/dreambooth/train_dreambooth_lora_sd3.py", line 1347, in main
    tokens_one = tokenize_prompt(tokenizer_one, args.instance_prompt)
UnboundLocalError: local variable 'tokenizer_one' referenced before assignment
Traceback (most recent call last):
  File "/workspace/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1097, in launch_command
    simple_launcher(args)
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 703, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/venv/bin/python', 'train_dreambooth_lora_sd3.py', '--pretrained_model_name_or_path=models/stable-diffusion-3-medium-diffusers', '--instance_data_dir=dog', '--output_dir=trained-sd3', '--mixed_precision=fp16', '--instance_prompt=a photo of sks dog', '--resolution=1024', '--train_batch_size=1', '--gradient_accumulation_steps=4', '--learning_rate=1e-5', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--max_train_steps=500', '--seed=0']' returned non-zero exit status 1.
```
```


### System Info

- 🤗 Diffusers version: 0.30.0.dev0
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Running on a notebook?: No
- Running on Google Colab?: No
- Python version: 3.10.12
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.23.4
- Transformers version: 4.41.2
- Accelerate version: 0.31.0
- PEFT version: 0.11.1
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: not installed
- Accelerator: NVIDIA RTX 6000 Ada Generation, 49140 MiB VRAM
- Using GPU in script?: Yes?
- Using distributed or parallel set-up in script?: No.

### Who can help?

@sayakpaul @yi

Hints from issue discussion:
the same error

Commit message: Update train_dreambooth_lora_sd3.py

Patch to review:
diff --git a/examples/dreambooth/train_dreambooth_lora_sd3.py b/examples/dreambooth/train_dreambooth_lora_sd3.py
index 8f831b50d07c..142dbcc4a2e0 100644
--- a/examples/dreambooth/train_dreambooth_lora_sd3.py
+++ b/examples/dreambooth/train_dreambooth_lora_sd3.py
@@ -1317,7 +1317,7 @@ def compute_text_embeddings(prompt, text_encoders, tokenizers):
     if not train_dataset.custom_instance_prompts:
         del tokenizers, text_encoders
         # Explicitly delete the objects as well, otherwise only the lists are deleted and the original references remain, preventing garbage collection
-        del tokenizer_one, tokenizer_two, tokenizer_three
+        # del tokenizer_one, tokenizer_two, tokenizer_three
         del text_encoder_one, text_encoder_two, text_encoder_three
         gc.collect()
         if torch.cuda.is_available():


Code review comments:
diff_hunk: @@ -1317,7 +1317,7 @@ def compute_text_embeddings(prompt, text_encoders, tokenizers):
     if not train_dataset.custom_instance_prompts:
         del tokenizers, text_encoders
         # Explicitly delete the objects as well, otherwise only the lists are deleted and the original references remain, preventing garbage collection
-        del tokenizer_one, tokenizer_two, tokenizer_three
+        # del tokenizer_one, tokenizer_two, tokenizer_three
path: examples/dreambooth/train_dreambooth_lora_sd3.py
text: @user1:
We can just remove this line no? 

@author:
done