kellibelcher commited on
Commit
9f4a001
1 Parent(s): 588a812

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +90 -35
README.md CHANGED
@@ -1,55 +1,80 @@
1
  ---
2
- license: creativeml-openrail-m
3
- datasets:
4
- - laion/laion400m
5
- tags:
6
- - stable-diffusion
7
- - stable-diffusion-diffusers
8
- - text-to-image
9
  language:
10
- - en
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  pipeline_tag: text-to-3d
 
12
  ---
13
 
14
  # LDM3D model
15
 
16
- The LDM3D model was proposed in ["LDM3D: Latent Diffusion Model for 3D"](https://arxiv.org/abs/2305.10853) by Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, Vasudev Lal.
17
 
18
- LDM3D got accepted to [CVPRW'23]([https://aaai.org/Conferences/AAAI-23/](https://cvpr2023.thecvf.com/)).
19
 
20
- For better results, do not hesitate to use [our new checkpoint](https://huggingface.co/Intel/ldm3d-4c) based on a slighlty different architecture.
21
 
22
- New checkpoints have been released:
23
- - [ldm3d-4c](https://huggingface.co/Intel/ldm3d-4c). A new version of LDM3D with higher quality results
24
- - [ldm3d-pano](https://huggingface.co/Intel/ldm3d-pano). LDM3D applied to panoramic images
25
- - [ldm3d-sr](https://huggingface.co/Intel/ldm3d-sr). Upscaler for LDM3D
26
 
27
- ## Model description
28
 
29
  The abstract from the paper is the following:
30
  This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the img2img pipeline to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences.
31
 
32
  ![LDM3D overview](model_overview.png)
33
- <font size="2">LDM3D overview taken from [the original paper](https://arxiv.org/abs/2305.10853)</font>
34
 
35
- ## Intended uses
36
 
37
- You can use this model to generate RGB and depth map given a text prompt.
38
  A short video summarizing the approach can be found at [this url](https://t.ly/tdi2) and a VR demo can be found [here](https://www.youtube.com/watch?v=3hbUo-hwAs0).
39
- A demo is also accessible on [Spaces](https://huggingface.co/spaces/Intel/ldm3d)
40
 
41
- ### How to use
42
-
43
- Here is how to use this model to get the features of a given text in PyTorch:
44
  ```python
45
 
46
  from diffusers import StableDiffusionLDM3DPipeline
47
 
48
  pipe = StableDiffusionLDM3DPipeline.from_pretrained("Intel/ldm3d")
49
- pipe.to("cuda")
50
 
 
 
 
 
 
51
 
52
- prompt ="A picture of some lemons on a table"
53
  name = "lemons"
54
 
55
  output = pipe(prompt)
@@ -62,13 +87,6 @@ This is the result:
62
 
63
  ![ldm3d_results](ldm3d_results.png)
64
 
65
-
66
- ### Limitations and bias
67
-
68
- For the image generation, limitations and bias are the same as the ones from [Stable diffusion](https://huggingface.co/CompVis/stable-diffusion-v1-4#limitations)
69
- For the depth map generation, a first limitiation is that we are using DPT-large to produce the ground truth, hence, other limitations and bias are the same as the ones from [DPT](https://huggingface.co/Intel/dpt-large).
70
-
71
-
72
  ## Training data
73
 
74
  The LDM3D model was finetuned on a dataset constructed from a subset of the LAION-400M dataset, a large-scale image-caption dataset that contains over 400 million image-caption pairs.
@@ -79,9 +97,46 @@ The fine-tuning process comprises two stages. In the first stage, we train an au
79
 
80
  ## Evaluation results
81
 
82
- Please refer to Table 1 and Table2 from the [paper](https://arxiv.org/abs/2305.10853) for quantitative results.
83
- The figure below shows some qualitative results comparing our method with (Stable diffusion v1.4)[https://arxiv.org/pdf/2112.10752.pdf] and with (DPT-Large)[https://arxiv.org/pdf/2103.13413.pdf] for the depth maps
84
- ![qualitative results](qualitative_results.png)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
 
86
  ### BibTeX entry and citation info
87
  ```bibtex
 
1
  ---
 
 
 
 
 
 
 
2
  language:
3
+ - en
4
+ tags:
5
+ - stable-diffusion
6
+ - stable-diffusion-diffusers
7
+ - text-to-image
8
+ model-index:
9
+ - name: ldm3d
10
+ results:
11
+ - task:
12
+ name: Latent Diffusion Model for 3D
13
+ type: latent-diffusion-model-for-3D
14
+ dataset:
15
+ name: LAION-400M
16
+ type: laion/laion400m
17
+ metrics:
18
+ - name: FID
19
+ type: FID
20
+ value: 27.82
21
+ - name: IS
22
+ type: IS
23
+ value: 28.79
24
+ - name: CLIP
25
+ type: CLIP
26
+ value: 26.61
27
+ - name: AbsRel
28
+ type: AbsRel
29
+ value: 0.0911
30
+ - name: RMSE [m]
31
+ type: RMSE-m
32
+ value: 0.334
33
  pipeline_tag: text-to-3d
34
+ license: creativeml-openrail-m
35
  ---
36
 
37
  # LDM3D model
38
 
39
+ The LDM3D model was proposed in the paper [LDM3D: Latent Diffusion Model for 3D](https://arxiv.org/abs/2305.10853.pdf), authored by Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, and Vasudev Lal.
40
 
41
+ LDM3D was accepted to the [IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR)](https://cvpr2023.thecvf.com/Conferences/2023) in 2023.
42
 
43
+ For better results, do not hesitate to use our new checkpoint, [ldm3d-4c](https://huggingface.co/Intel/ldm3d-4c) based on a slighlty different architecture.
44
 
45
+ The following new LDM3D model checkpoints have been released:
46
+ - [ldm3d-4c](https://huggingface.co/Intel/ldm3d-4c): A new version of LDM3D with higher quality results.
47
+ - [ldm3d-pano](https://huggingface.co/Intel/ldm3d-pano): LDM3D applied to panoramic images.
48
+ - [ldm3d-sr](https://huggingface.co/Intel/ldm3d-sr): Upscaler for LDM3D.
49
 
50
+ ## Model details
51
 
52
  The abstract from the paper is the following:
53
  This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the img2img pipeline to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences.
54
 
55
  ![LDM3D overview](model_overview.png)
56
+ <font size="2">LDM3D overview taken from the [LDM3D paper](https://arxiv.org/abs/2305.10853).</font>
57
 
58
+ ## Usage
59
 
60
+ You can use this model to generate an RGB image and depth map given a text prompt.
61
  A short video summarizing the approach can be found at [this url](https://t.ly/tdi2) and a VR demo can be found [here](https://www.youtube.com/watch?v=3hbUo-hwAs0).
62
+ A demo is also accessible on [Spaces](https://huggingface.co/spaces/Intel/ldm3d).
63
 
64
+ Here is how to use this model to get the features of a given text in PyTorch on both a CPU and GPU architecture:
 
 
65
  ```python
66
 
67
  from diffusers import StableDiffusionLDM3DPipeline
68
 
69
  pipe = StableDiffusionLDM3DPipeline.from_pretrained("Intel/ldm3d")
 
70
 
71
+ # On CPU
72
+ pipe.to("cpu")
73
+
74
+ # On GPU
75
+ pipe.to("cuda")
76
 
77
+ prompt = "A picture of some lemons on a table"
78
  name = "lemons"
79
 
80
  output = pipe(prompt)
 
87
 
88
  ![ldm3d_results](ldm3d_results.png)
89
 
 
 
 
 
 
 
 
90
  ## Training data
91
 
92
  The LDM3D model was finetuned on a dataset constructed from a subset of the LAION-400M dataset, a large-scale image-caption dataset that contains over 400 million image-caption pairs.
 
97
 
98
  ## Evaluation results
99
 
100
+ ### Quantitative results
101
+
102
+ The table below shows the quantitative results of text-conditional image synthesis on the 512 x 512-sized MS-COCO dataset with 50 DDIM steps.
103
+
104
+ |Method |FID ↓|IS ↑ |CLIP ↑ |
105
+ |------------|-----|------------|------------|
106
+ |SD v1.4 |28.08|34.17 ± 0.76|26.13 ± 2.81|
107
+ |SD v1.5 |27.39|34.02 ± 0.79|26.13 ± 2.79|
108
+ |LDM3D (ours)|27.82|28.79 ± 0.49|26.61 ± 2.92|
109
+
110
+ Our model is on par with the Stable Diffusion models with the same number of parameters (1.06B). IS and CLIP similarity scores are averaged over 30k captions from the MS-COCO dataset.
111
+
112
+ The following table shows the evaluation results of depth evaluation comparing LDM3D and DPT-Large with respect to ZoeDepth-N that serves as a reference model.
113
+
114
+ |Method |AbsRel|RMSE [m]|
115
+ |---------|------|--------|
116
+ |LDM3D |0.0911|0.334 |
117
+ |DPT-Large|0.0779|0.297 |
118
+
119
+ The results shown above can be referenced in Table 1 and Table 2 of the [LDM3D paper](https://arxiv.org/abs/2305.10853.pdf).
120
+
121
+ ### Qualitative results
122
+
123
+ The figure below shows some qualitative results comparing our method with [Stable Diffusion v1.4](https://arxiv.org/pdf/2112.10752.pdf) and with [DPT-Large](https://arxiv.org/pdf/2103.13413.pdf) for the depth maps ![qualitative results](qualitative_results.png).
124
+
125
+ ## Ethical Considerations and Limitations
126
+
127
+ For image generation, the [Stable Diffusion](https://huggingface.co/CompVis/stable-diffusion-v1-4#limitations) limitations and biases apply. For depth map generation, a first limitiation is that we are using DPT-large to produce the ground truth, hence, other limitations and biases from [DPT](https://huggingface.co/Intel/dpt-large) are applicable.
128
+
129
+ ## Caveats and Recommendations
130
+
131
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
132
+
133
+ Here are a couple of useful links to learn more about Intel's AI software:
134
+ * [Intel Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch)
135
+ * [Intel Neural Compressor](https://github.com/intel/neural-compressor)
136
+
137
+ ## Disclaimer
138
+
139
+ The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.
140
 
141
  ### BibTeX entry and citation info
142
  ```bibtex