DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev

LoRA Encoder (FLUX.1-Dev)

This model encodes LoRA models for FLUX into embedding vectors, unlocking the capabilities of the LoRA models.

Taking the LoRA model VoidOc/F.1_Animal_Forest_LoRA as an example, the LoRA encoder can be used in the following ways.

Method 1: LoRA Usage Inference

Given a LoRA model, with no additional information and using an empty prompt, the LoRA encoder can directly activate the LoRA's capabilities, allowing inference of its intended use.

Prompt: ""

Without LoRA Encoder	With LoRA Encoder

Method 2: Trigger-Free Activation of LoRA Capabilities

Activate the LoRA's capabilities automatically without needing to specify any trigger words.

Prompt: "a car"

Without LoRA Encoder	With LoRA Encoder

Method 3: LoRA Strength Control

An additional parameter scale is provided to control the influence of the LoRA on the generated image.

In the example below, the prompt is "a cat". When scale=1, the LoRA exerts maximum influence, resulting in an image showing both a character from Animal Crossing and a cat. When scale=0.5, the LoRA's influence is reduced, producing an image of a cat character from Animal Crossing. The optimal scale value depends on the specific LoRA model; we recommend using larger values for character-based LoRAs and smaller values for style-based LoRAs.

Prompt: "a cat"

scale=1	scale=0.5

Inference Code

git clone https://github.com/modelscope/DiffSynth-Studio.git  
cd DiffSynth-Studio
pip install -e .

import torch
from diffsynth.pipelines.flux_image_new import FluxImagePipeline, ModelConfig


pipe = FluxImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="flux1-dev.safetensors"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder_2/"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="ae.safetensors"),
        ModelConfig(model_id="DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev", origin_file_pattern="model.safetensors"),
    ],
)
pipe.enable_lora_magic()

lora = ModelConfig(model_id="VoidOc/flux_animal_forest1", origin_file_pattern="20.safetensors")
pipe.load_lora(pipe.dit, lora, hotload=True) # Use `pipe.clear_lora()` to drop the loaded LoRA.

# Empty prompt can automatically activate LoRA capabilities.
image = pipe(prompt="", seed=0, lora_encoder_inputs=lora)
image.save("image_1.jpg")

image = pipe(prompt="", seed=0)
image.save("image_1_origin.jpg")

# Prompt without trigger words can also activate LoRA capabilities.
image = pipe(prompt="a car", seed=0, lora_encoder_inputs=lora)
image.save("image_2.jpg")

image = pipe(prompt="a car", seed=0)
image.save("image_2_origin.jpg")

# Adjust the activation intensity through the scale parameter.
image = pipe(prompt="a cat", seed=0, lora_encoder_inputs=lora, lora_encoder_scale=1.0)
image.save("image_3.jpg")

image = pipe(prompt="a cat", seed=0, lora_encoder_inputs=lora, lora_encoder_scale=0.5)
image.save("image_3_scale.jpg")