Repository: huggingface/diffusers
Pull request: [docs] Clarify training args
Fixes #3038 to clarify what the `pretrained_model_name_or_path` argument expects during training.

Problem statement:
Expand documentation for LORA with locally stored models
**Description of LORA fine tuning does not cover all the bases**
I am fairly new to Stable Diffusion, but have some experience using other ML projects, so in general, configuring a dataset and fine tuning is not an issue, however, I run into a block here trying to finetune LORA on a local model, as the `--pretrained_model_name_or_path` switch expects a JSON file, but this behavior is undocumented for the script.

**A solution I'd like**
A chapter for the documentation in `diffusers/examples/text_to_image/README.md` that describes or at least links to other documentation on what file path should be specifies for using a locally downloaded model for the basis of LORA.

**Alternatives I've considered**
I've read through the code, and I can see that the `args.pretrained_model_name_or_path` is used in many function calls, so I assume people more familiar with Stable Diffusion would be able to find the documentation, but it is hardly ideal for someone a bit less knowledgeable.

**Additional context**
Nowadays many different SD models exist that one can get from the internet. Using LORA to fine tune one that is almost there for one's personal use case would be ideal from a time and wasted computational power perspective. Therefore having documentation on how to do LORA fine tuning on locally downloaded models (safetensors, ckpt) would be nice.

Hints from issue discussion:
Hey @nicemicro, 

Yes I agree we should try to improve our docs here. Maybe @sayakpaul you could take this in one go next week once you have some time to adapt the LoRA training script :-) 
I think it's unrelated to LoRA but related to mentioning the expectation of what `pretrained_model_name_or_path` should be in general, no? 

`pretrained_model_name_or_path` remains unchanged irrespective of LoRA. Or am I missing something? 

For clarification on the docs side, I'd like to defer to @yiyixuxu. 
Guess it could make sense to generally update the LoRA docs here a bit once we support LoRA text encoder training and LoRA A1111 format loading. Maybe just a PR similar to: https://github.com/huggingface/diffusers/pull/3068. Also cc @stevhliu @yiyixuxu 
@sayakpaul yes, you are right, it is not directly related to LoRA. If there is a documentation on what `pretrained_model_name_or_path` expects, I somehow did not find, and by extension, others might not find it either, so at least a link would be useful in my opinion.
I see what you are saying. 

All the pipelines (like `StableDiffusionPipeline`) inherit from [`DiffusionPipeline`](https://huggingface.co/docs/diffusers/api/diffusion_pipeline). There we have:

> Any diffusion pipeline that is loaded with [from_pretrained()](https://huggingface.co/docs/diffusers/v0.15.0/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained) will automatically detect the pipeline type, e.g. [StableDiffusionPipeline](https://huggingface.co/docs/diffusers/v0.15.0/en/api/pipelines/stable_diffusion/text2img#diffusers.StableDiffusionPipeline) and consequently load each component of the pipeline and pass them into the __init__ function of the pipeline, e.g. __init__().

And there you have: https://huggingface.co/docs/diffusers/v0.15.0/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained. 

Does this help? 

Cc @stevhliu 
Thanks for the feedback!

For all the training examples, I can clarify that setting the `MODEL_NAME` environment variable is really just specifying the argument to `from_pretrained.pretrained_model_name_or_path` and maybe also include a link to it.
@sayakpaul @stevhliu 
Thank you for the quick reactions!
I think if it specified what command/variable it uses from what other package, that would be helpful, and a direct link to the relevant documentation would be amazing!

Commit message: clarify training arg

Patch to review:
diff --git a/docs/source/en/training/controlnet.mdx b/docs/source/en/training/controlnet.mdx
index 7a5454107b83..94e3d969b80a 100644
--- a/docs/source/en/training/controlnet.mdx
+++ b/docs/source/en/training/controlnet.mdx
@@ -74,6 +74,7 @@ wget https://huggingface.co/datasets/huggingface/documentation-images/resolve/ma
 wget https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/controlnet_training/conditioning_image_2.png
 ```
 
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument.
 
 ```bash
 export MODEL_DIR="runwayml/stable-diffusion-v1-5"
diff --git a/docs/source/en/training/dreambooth.mdx b/docs/source/en/training/dreambooth.mdx
index 908355e496dc..7be5b2a6a248 100644
--- a/docs/source/en/training/dreambooth.mdx
+++ b/docs/source/en/training/dreambooth.mdx
@@ -60,7 +60,9 @@ DreamBooth finetuning is very sensitive to hyperparameters and easy to overfit.
 
 <frameworkcontent>
 <pt>
-Let's try DreamBooth with a [few images of a dog](https://drive.google.com/drive/folders/1BO_dyz-p65qhBRRMRA4TbZ8qW4rB99JZ); download and save them to a directory and then set the `INSTANCE_DIR` environment variable to that path:
+Let's try DreamBooth with a [few images of a dog](https://drive.google.com/drive/folders/1BO_dyz-p65qhBRRMRA4TbZ8qW4rB99JZ); download and save them to a directory and then set the `INSTANCE_DIR` environment variable to that path.
+
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument.
 
 ```bash
 export MODEL_NAME="CompVis/stable-diffusion-v1-4"
@@ -94,6 +96,8 @@ Before running the script, make sure you have the requirements installed:
 pip install -U -r requirements.txt
 ```
 
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument.
+
 Now you can launch the training script with the following command:
 
 ```bash
diff --git a/docs/source/en/training/instructpix2pix.mdx b/docs/source/en/training/instructpix2pix.mdx
index c485db6d6b20..ff34ec335656 100644
--- a/docs/source/en/training/instructpix2pix.mdx
+++ b/docs/source/en/training/instructpix2pix.mdx
@@ -74,8 +74,7 @@ write_basic_config()
 As mentioned before, we'll use a [small toy dataset](https://huggingface.co/datasets/fusing/instructpix2pix-1000-samples) for training. The dataset 
 is a smaller version of the [original dataset](https://huggingface.co/datasets/timbrooks/instructpix2pix-clip-filtered) used in the InstructPix2Pix paper.
 
-Configure environment variables such as the dataset identifier and the Stable Diffusion
-checkpoint:
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument. You'll also need to specify the dataset name in `DATASET_ID`:
 
 ```bash
 export MODEL_NAME="runwayml/stable-diffusion-v1-5"
diff --git a/docs/source/en/training/lora.mdx b/docs/source/en/training/lora.mdx
index 1c72fbbc8d58..d221b00fc755 100644
--- a/docs/source/en/training/lora.mdx
+++ b/docs/source/en/training/lora.mdx
@@ -50,7 +50,9 @@ Finetuning a model like Stable Diffusion, which has billions of parameters, can
 
 Let's finetune [`stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) on the [Pokémon BLIP captions](https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions) dataset to generate your own Pokémon.
 
-To start, make sure you have the `MODEL_NAME` and `DATASET_NAME` environment variables set. The `OUTPUT_DIR` and `HUB_MODEL_ID` variables are optional and specify where to save the model to on the Hub:
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument. You'll also need to set the `DATASET_NAME` environment variable to the name of the dataset you want to train on.
+
+The `OUTPUT_DIR` and `HUB_MODEL_ID` variables are optional and specify where to save the model to on the Hub:
 
 ```bash
 export MODEL_NAME="runwayml/stable-diffusion-v1-5"
@@ -138,7 +140,9 @@ Load the LoRA weights from your finetuned model *on top of the base model weight
 
 Let's finetune [`stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) with DreamBooth and LoRA with some 🐶 [dog images](https://drive.google.com/drive/folders/1BO_dyz-p65qhBRRMRA4TbZ8qW4rB99JZ). Download and save these images to a directory.
 
-To start, make sure you have the `MODEL_NAME` and `INSTANCE_DIR` (path to directory containing images) environment variables set. The `OUTPUT_DIR` variables is optional and specifies where to save the model to on the Hub:
+To start, specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument. You'll also need to set `INSTANCE_DIR` to the path of the directory containing the images. 
+
+The `OUTPUT_DIR` variables is optional and specifies where to save the model to on the Hub:
 
 ```bash
 export MODEL_NAME="runwayml/stable-diffusion-v1-5"
diff --git a/docs/source/en/training/text2image.mdx b/docs/source/en/training/text2image.mdx
index 70f8c003a787..dabb68397f78 100644
--- a/docs/source/en/training/text2image.mdx
+++ b/docs/source/en/training/text2image.mdx
@@ -72,7 +72,9 @@ To load a checkpoint to resume training, pass the argument `--resume_from_checkp
 
 <frameworkcontent>
 <pt>
-Launch the [PyTorch training script](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image.py) for a fine-tuning run on the [Pokémon BLIP captions](https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions) dataset like this:
+Launch the [PyTorch training script](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image.py) for a fine-tuning run on the [Pokémon BLIP captions](https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions) dataset like this.
+
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument.
 
 <literalinclude>
 {"path": "../../../../examples/text_to_image/README.md",
@@ -141,6 +143,8 @@ Before running the script, make sure you have the requirements installed:
 pip install -U -r requirements_flax.txt
 ```
 
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument.
+
 Now you can launch the [Flax training script](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_flax.py) like this:
 
 ```bash
diff --git a/docs/source/en/training/text_inversion.mdx b/docs/source/en/training/text_inversion.mdx
index 6e6971d7f119..168400a7f28e 100644
--- a/docs/source/en/training/text_inversion.mdx
+++ b/docs/source/en/training/text_inversion.mdx
@@ -83,7 +83,9 @@ To resume training from a saved checkpoint, pass the following argument to the t
 
 For your training dataset, download these [images of a cat statue](https://drive.google.com/drive/folders/1fmJMs25nxS_rSNqS5hTcRdLem_YQXbq5) and store them in a directory. 
 
-Set the `MODEL_NAME` environment variable to the model repository id, and the `DATA_DIR` environment variable to the path of the directory containing the images. Now you can launch the [training script](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion.py):
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument, and the `DATA_DIR` environment variable to the path of the directory containing the images. 
+
+Now you can launch the [training script](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion.py):
 
 <Tip>
 
@@ -121,6 +123,8 @@ Before you begin, make sure you install the Flax specific dependencies:
 pip install -U -r requirements_flax.txt
 ```
 
+Specify the `MODEL_NAME` environment variable (either a Hub model repository id or a path to the directory containing the model weights) and pass it to the [`~diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path`] argument.
+
 Then you can launch the [training script](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion_flax.py):
 
 ```bash


Code review comments:
diff_hunk: @@ -60,7 +60,9 @@ DreamBooth finetuning is very sensitive to hyperparameters and easy to overfit.
 
 <frameworkcontent>
 <pt>
-Let's try DreamBooth with a [few images of a dog](https://drive.google.com/drive/folders/1BO_dyz-p65qhBRRMRA4TbZ8qW4rB99JZ); download and save them to a directory and then set the `INSTANCE_DIR` environment variable to that path:
+Let's try DreamBooth with a [few images of a dog](https://drive.google.com/drive/folders/1BO_dyz-p65qhBRRMRA4TbZ8qW4rB99JZ); download and save them to a directory and then set the `INSTANCE_DIR` environment variable to that path.
path: docs/source/en/training/dreambooth.mdx
text: @user1:
While we're at it could we maybe improve this script to **not** download images from google drive but from Hugging Face datasets? 

I've uploaded them here: https://huggingface.co/datasets/diffusers/dog-example and you can use the same code as done here: https://github.com/huggingface/diffusers/tree/main/examples/textual_inversion#cat-toy-example

and then also update it here: 
https://github.com/huggingface/diffusers/tree/main/examples/dreambooth#dog-toy-example

and https://github.com/huggingface/diffusers/pull/3146/files#r1170252797 as well