BertChristiaens commited on
Commit
a9bdadd
1 Parent(s): a9142eb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -2
README.md CHANGED
@@ -9,8 +9,6 @@ Big thanks to `Google` for lending us TPUv4s to train this model on. Big thanks
9
  ## About the dataset
10
  To make this demo as good as possible, our team spend a lot of time training a custom model. We used the LAION5B dataset to build our custom dataset, which contains 130k images of 15 types of rooms in almost 30 design styles. After fetching all these images, we started adding metadata such as captions (from the BLIP captioning model) and segmentation maps (from the HuggingFace UperNetForSemanticSegmentation model).
11
 
12
- For the gathering and inference of the metadata we used the Fondant framework (https://github.com/ml6team/fondant) provided by ML6 (https://www.ml6.eu/), which is an open source data centric framework for data preparation. The pipeline used for training this controlnet will soon be available as an example pipeline within Fondant and can be easily adapted for building your own dataset.
13
-
14
  ## About the model
15
  These were then used to train the controlnet model to generate quality interior design images by using the segmentation maps and prompts as conditioning information for the model. By training on segmentation maps, the enduser has a very finegrained control over which objects they want to place in their room. The resulting model is then used in a community pipeline that supports image2image and inpainting, so the user can keep elements of their room and change specific parts of the image.
16
  The training started from the `lllyasviel/control_v11p_sd15_seg` checkpoint, which is a robustly trained controlnet model conditioned on segmentation maps. This checkpoint got fine-tuned on a TPUv4 with the JAX framework. Afterwards, the checkpoint was converted into a PyTorch checkpoint for easy integration with the diffusers library.
 
9
  ## About the dataset
10
  To make this demo as good as possible, our team spend a lot of time training a custom model. We used the LAION5B dataset to build our custom dataset, which contains 130k images of 15 types of rooms in almost 30 design styles. After fetching all these images, we started adding metadata such as captions (from the BLIP captioning model) and segmentation maps (from the HuggingFace UperNetForSemanticSegmentation model).
11
 
 
 
12
  ## About the model
13
  These were then used to train the controlnet model to generate quality interior design images by using the segmentation maps and prompts as conditioning information for the model. By training on segmentation maps, the enduser has a very finegrained control over which objects they want to place in their room. The resulting model is then used in a community pipeline that supports image2image and inpainting, so the user can keep elements of their room and change specific parts of the image.
14
  The training started from the `lllyasviel/control_v11p_sd15_seg` checkpoint, which is a robustly trained controlnet model conditioned on segmentation maps. This checkpoint got fine-tuned on a TPUv4 with the JAX framework. Afterwards, the checkpoint was converted into a PyTorch checkpoint for easy integration with the diffusers library.