LoRA text2image fine-tuning - remi349/sd_trained_3D_lora

These are LoRA adaption weights are for stabilityai/stable-diffusion-2-1. The weights were fine-tuned on the remi349/finetuning_dataset_for_3D_training dataset thanks to the library diffusers.

Intended uses & limitations

This model aims at generating images of isolated objects, compatible with 2D_to_3D models like Triposr or CRM. It was finetuned in order to create after a pipeline of prompt-to-3D model.

How to use

# First load the basic architecture and everything
import torch
from diffusers import StableDiffusionPipeline 
pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16)

# Then add the lora weights to the model stable diffusion 2
pipe.unet.load_attn_procs('ACROSS-Lab/PromptTo3D_sd_finetuned')
pipe.to("cuda")

# Then you can begin the inference process on a prompt and save the image generated
prompt = 'a rabbit with a yellow jacket'
image = pipe(prompt, num_inference_steps=30, guidance_scale=7.5).images[0]
image.save("my_image.png")

Limitations and bias

This model is a first try some hyperparameters tuning should be done, but for that we would need a solid automated benchmark.

Training details

The model finetuned model is Stable Diffusion 2. The data used to train this model is the dataset available on uggingface at 'remi349/finetuning_dataset_for_3D_training'. you can download it thanks to the command

from datasets import load_dataset
dataset = load_dataset("ACROSS-Lab/PromptTo3D_sd_dataset", split = 'train')

This dataset is a subset of the dataset Objaverse.

Collaboration

This model and dataset has been made in collaboration by Josué ADOSSEHOUN and Rémi DUCOTTET