SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our study, while exploratory, has broad implications. We discuss a few of them below.
Understanding Generative Models. We highlight important properties and capabilities in generative models when not trained with explicit supervision. This insight could potentially lead to the development of more efficient and effective training methods for generative models.
Figure 11: Quantitative evaluation of normal variations following relighting. Normals are calculated under 16 distinct lighting conditions from [11] for 214 test scenes for each normal prediction methods. The inner product between normals under each condition and the overall mean is computed. Ideally, this should be 1, indicating normals' consistency. The boxplots illustrate these values, with regression methods showing an average change of 8 degrees, with outliers up to 14 degrees. High similarity scores for StyleGAN indicate its normals' robustness to lighting changes.
Figure 12: This figure shows the failure in generation of surface normal directions using StyleGAN-XL on ImageNet with 10,000 unique generated images. The images in the middle row resemble the surface normals in the bottom row but miss details that [28] predicted. The differences in architecture between StyleGAN-XL and the more appropriate StyleGAN-2 make it difficult to leverage this model. Either an alternate strategy is required or the latent space in this model is not easy to manipulate.
Intrinsic Images from other Generative Models. We have demonstrated that StyleGAN can easily extract representations of important intrinsic images. This makes our work the first to achieve this, and that too by simply adding an offset to a pretrained generative model. Even if the current method has a performance gap compared to SOTA monocular predictors, this understanding that generative models can capture such intrinsic properties provides a new avenue for research. Our findings raise compelling questions for further investigation. We suspect that this property may also be true for other generative models. Is this the case, or is it a result of specific design choices in StyleGAN? We think other generative models "know" intrinsic images, too, but don't currently know how to elicit that knowledge.
Robust Intrinsic Images. One of the potential advantages of our approach is deriving intrinsic images that are inherently more robust, especially against variations like lighting changes (Figure 10). One might use this to bootstrap a better StyleGAN predictor from a non-robust, but SOTA, predictor.
Guidance from SOTA Monocular Predictor. Currently, we rely on monocular predictors for offset guidance. However, we use limited guidance, consisting of just 2000 examples. We anticipate that future models may not require such guidance.
GAN Inversion. To build a practical intrinsic image predictor from a StyleGAN, one needs an inverter that is (a) accurate and (b) preserves the parametrization of the image space by latents – can this be built?
What Makes Good Image Representation? Finally, our findings provide insights into what makes a "good" image representation. Our discovery of fundamental image intrinsics such as depth, albedo, and normals in the latent space of a StyleGAN suggests that meaningful image representations are those that can capture these intrinsic properties, as highlighted in Barrow and Tenenbaum's work [6]. StyleGAN clearly 'knows' the intrinsics that are well-understood in the vision and graphics community. Are there other intrinsics that it 'knows' but that we have overlooked? The materials literature, particularly studies like Motoyoshi et al. [38] and Sharan et al. [51], suggest that many other types of intrinsic images may exist. Therefore, our search method could potentially uncover unknown intrinsic images.
This material is based upon work supported by the National Science Foundation under Grant No. 2106825 and by gifts from Amazon and Boeing.
[66] Ning Yu, Guilin Liu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro, Larry S Davis, and Mario Fritz. Dual contrastive loss and attention for gans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6731-6742, 2021. 5
[67] Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3
[68] Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 5, 6, 9, 16, 17
[69] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2
[70] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv preprint arXiv:2010.09125, 2020. 3
[71] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021. 2, 3
[72] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153, 2023. 3
[73] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv:1608.05442, 2016. 7
[74] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference on Computer Vision (ECCV), 2020. 2
[75] Peiye Zhuang, Oluwasanmi Koyejo, and Alexander G Schwing. Enjoy your editing: Controllable gans for image editing via latent space navigation. arXiv preprint arXiv:2102.01187, 2021. 2, 4
In Figure 13, we demonstrate the segmentation quality with increase in number of labeled examples. We also provide additional qualitative figures for intrinsic image predictions from StyleGAN – normals in Figure 14, depth in Figure 15, albedo-shading decomposition in Figure 16, and segmentation of lamps and pillows in Figure 17, segmentation of windows and paintings in Figure 18 and segmentation of beds in Figure 19. In our experiments, we generated three-channel output for each depth, shading, and segmentation from StyleGAN. We took the mean for each of them to get the final single-channel estimate. We also provide additional examples of robustness against lighting changes for the segmentation task in Figure 20.
Figure 13: Segmentation Quality vs. Labeled Dataset Size: The graph illustrates how segmentation quality for 'pillows' and 'windows' classes improves with increasing labeled images, plateauing around 2,000 images. Note that all images are seen only once. While the potential exists for further refinement, the core insight is StyleGAN's inherent capability to generate these segmentations without explicit training.
Figure 14: Additional Normal generation.
Figure 15: Additional Depth Estimation Comparison.
Image
Albedo-G
Albedo-R
Shading-G
Shading-R
Residual-G
Residual-R
Figure 16: Additional Results for Albedo-Shading Recovery with StyleGAN.
Figure 17: Further segmentation of lamps on the left and pillows on the right.
Figure 18: Window segmentation on the left and painting segmentation on the right.
Figure 19: Bed segmentation comparison.
Figure 20: Additional examples for robustness against lighting for segmentation.