some quesetions about panorama images generation
Hi, first of all, thank you very mush for sharing ldm3d tech, the results are very impressvie! I'm just getting into the 3D filed and have some questions about panoramas generation.I hope you can help me.
First, I followed the guidance of ldm3d huggingface space and generated some panoramas images. but I found that the performace of panoramas generated by ldm3d-pano checkpoints is not as good as that in the demo(https://www.youtube.com/watch?v=3hbUo-hwAs0). In the generated settings, the height is 1024, the width is 2048, and the prompt text is "360 view of a beautiful garden in CG style", use default values for other parameters. However, the result style is not CG style and geometric deformation is serious. Could you tell me some suggestions about generating better quality panoramas?
Second, The dataset used in the original paper was LAION-400M, which does not contain panorama images. So the ldm3d-pano model was trained on another dataset that contains panoramas image, depth image and caption tuple?
Three, I found some bugs in the generated panorama images, and that is, inconsistencies at the stitching. Have you considered how to deal with this bug, whether to solve it directly from the model structure or through post-processing?
Looking forward to your reply!!!
Thank you for the interest in our work!
Regarding your first comment: we are not using the same method to display than the one used in the youtube demo. The demo used in Spaces is lighter and more appropriate for a large-scale use compared to the demo you saw on YouTube. Have you maybe tried to play with the resolution (maybe smaller resolution? the one we trained on is 512 by 1024.). I am not sure what CG style is, but LDM3D was finetuned on Stable Diffusion 1.5 so it leverages the dataset it was trained on. Do you get good results with SD1.5 when using "CG style"?
Regarding the 2nd I addressed this by creating the model card with all the relevant info
For the 3rd and last question: we are aware of the stitching issue and thinking on how to address with this. However, its not among our current priorities