Title: IntrinsiX: High-Quality PBR Generation using Image Priors
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Synthesis

Diffusion

Intrinsic Images

Rendering

Texturing

Deep Learning
Abstract: We introduce IntrinsiX, a novel method that generates high-quality intrinsic images from text description.
In contrast to existing text-to-image models whose outputs contain baked-in scene lighting, our approach predicts physically-based rendering (PBR) maps.
This enables the generated outputs to be used for content creation scenarios in core graphics applications that facilitate re-lighting, editing, and texture generation tasks. 
In order to train our generator, we exploit strong image priors, and pre-train separate models for each PBR material component (albedo, roughness, metallic, normals).
We then align these models with a new cross-intrinsic attention formulation that concatenates key and value features in a consistent fashion. 
This allows us to exchange information between each output modality and to obtain semantically coherent PBR predictions.
To ground each intrinsic component, we propose a rendering loss which provides image-space signals to constrain the model, thus facilitating sharp details also in the output BRDF properties. 
Our results demonstrate detailed intrinsic generation with strong generalization capabilities that outperforms existing intrinsic image decomposition methods used with generated images by a significant margin.
Finally, we show a series of applications, including re-lighting, editing, and for the first time text-conditioned room-scale PBR texture generation.
We will release the code and the pre-trained model weights.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to generate intrinsic properties images (including albedo, roughness & metallic, and normal) from a given text description. The method uses the pretrained FLUX.1-dev model as the generative backbone and finetunes it using 3 separate LoRAs for each intrinsic property. After separate finetuning, the method finetunes the LoRAs jointly using a cross-intrinsic attention and a re-rendering loss with importance-based lighting direction sampling. The paper showcases the applications of the proposed method and conducts extensive experiments to show the superiority of the proposed method against baseline methods and ablation studies to show the effectiveness of the proposed techniques.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed two-stage training strategy with the cross-intrinsic attention (though this attention is alike the commonly-used cross-view or cross-frame attention and lacks true novelty) and the re-rendering loss is somewhat novel: Though the re-rendering loss is vastly used in previous CNN-based intrinsic image decomposition methods such as [33], it is the first time (as far as I know) that it is used in a diffusion-based framework. The proposed importance-based lighting direction sampling for training is also interesting and effective.\n2. The paper proposes an efficient pipeline that takes only 26h to train on a single A100 GPU and 12s to generate an image on a single A6000 GPU, and also gives state-of-the-art results in the task of intrinsic image generation (though it is a new task proposed by this paper).\n3. Extensive quantitative and qualitative comparisons and ablation studies.\n4. The authors provide an entire section to showcase the applications of the proposed method, including relightable image generation and PBR texturing.\n5. Good presentation and easy-to-follow paper.\n\nWeaknesses:\n1. The quantitative evaluation may not be very rigorous. \n  - Firstly, the paper uses FID to compare the quality of the generated albedo image across different baselines. However, the testing dataset contains only 100 (for in-distribution) + 40 (for out-of-distribution) images, which are far less than the common requirements of computing FID, i.e., 10,000 images. The small amount of images for computing FID may render the evaluate unreliable. Is it a good practice to use 100 and 40 images to compute FID? If not, why not use more images?\n  - Secondly, I am a little bit confused about the definition of \"in-distribution\" and \"out-of-distribution\" in L263-L272. While the sampled images are \"in-distribution\" regarding the proposed method as it is trained on InteriorVerse, are they still \"in-distribution\" regarding the comparing methods, including IID, RGBX, and ColorfulShading, as these baselines may not be trained on InteriorVerse? Could the authors provide a more in-depth discussion of \"in-distribution\" and \"out-of-distribution\", as the ability to handle \"out-of-distribution\" is stated as a major difference between the proposed method and baseline methods (L33, L44, L90, L297)?\n2. The statement about the inherent ambiguity of intrinsic image decomposition is not persuasive. Intuitively, generating PBR images from mere text inputs is more ambiguous than from an input shaded RGB image, as even the scene structure is not determined yet. So I fail to understand why \"the inherent ambiguity of decomposing an RGB image into its intrinsic properties makes it difficult to match the PBR distribution for out-of-domain samples\" (L90, L298) and how does the proposed method overcome this difficulty using an even more ambiguous task.\n3. The above weakness also motivated me to question the proposed task, intrinsic image generation. I think in essence it is similar to the traditional intrinsic image decomposition, except that the input is changed to a text description instead of an image. Personally, I prefer to use an image as the input in many cases as I may want to edit an exisiting image instead of generating a new image from scratch. Thus, I think the proposed task is not very useful (the texturing task in Section 4 is also actually achieved by finetuning the model to enable it to condition on an image). Could the authors compare the usefulness between the proposed intrinsic image generation task and the traditional intrinsic image decomposition task? Also, what is the possible outcome of using the proposed techniques for intrinsic image decomposition (i.e., changing the input from text to image)?\n\nTrivia:\n1. Typos:\n  - L129: \"with utilizing\" -> \"with\" or \"utilizing\". \n  - L274: \"A-PP\" -> \"A-PQ\".\n  - L337: \"the preserve\" -> \"to preserve\".\n2. Eq. (3) seems incorrect. E.g., the BRDF evaluation should be related to the incoming lighting direction $\\omega_i$ and the cosine term should be related to the surface normal.\n\nQUESTIONS:\nCorresponding to the weaknesses above, I summarize the questions below:\n1. The paper uses FID to compare the quality of the generated albedo image across different baselines. Is it a common practice to use 100 and 40 images to compute FID? If not, why not use more images?\n2. Could the authors provide a more in-depth discussion of \"in-distribution\" and \"out-of-distribution\", as the ability to handle \"out-of-distribution\" is stated as a major difference between the proposed method and baseline methods (L33, L44, L90, L297)?\n3. Why \"the inherent ambiguity of decomposing an RGB image into its intrinsic properties makes it difficult to match the PBR distribution for out-of-domain samples\" (L90, L298) and how does the proposed method overcome this difficulty using an even more ambiguous task?\n4. Could the authors compare the usefulness between the proposed intrinsic image generation task and the traditional intrinsic image decomposition task?\n5. What is the possible outcome of using the proposed techniques for intrinsic image decomposition (i.e., changing the input from text to image)?\n\nI will raise my rating once the authors provide clarifications to these questions.\n\nLIMITATIONS:\nRegarding the rendered image: The method does not discriminate between foreground and background regions. Sometimes relighting the background region like a plain surface feels weird. Also, the renderer cannot render cast shadows as the depth information is absent. These can be left for future work to make the rendered image more realistic.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of text-conditioned PBR generation using a pretrained generative model. The authors adapt an image generation model with LoRA for the text-based generation of different PBR components. This results in three sets of LoRA weights: one for albedo, one for normals, and one for specularity (channel-packing roughness and metalness). Naturally, these models are not aligned and produce inconsistent outputs. To address this, the authors introduce a second stage where traditional self-attention is replaced by cross-task attention. In this setup, the activation for a given component is computed based on its correlation with the activations of other components (including itself). Additionally, a rendering loss is applied in image space between the re-rendered ground truth and the rendering of the generated samples, using a single point light source. The model produces high-quality, sharp results that are consistent across PBR properties. Finally, the authors demonstrate that their method can be successfully applied to scene texturing via score distillation sampling.\n\nSTRENGTHS AND WEAKNESSES:\n## Paper strengths\n- I find the explanation of why joint learning (with cross-talk between different material properties) leads to unsatisfactory results quite interesting. As the authors hypothesize, since attention mechanisms are correlation-based, they tend to encourage similar features, which may hinder diversity among outputs.\n\n- The quality of the results are good and material property maps remain sharp.\n\n- I also appreciate the extension to more complex scenes which does not look like a trivial addition.\n\n## Task definition\n- The formulation of the problem is a bit clumsy. For example, the phrase “we formulate PBR generation as a generative approach” (l.91) is redundant, PBR generation is inherently a generative task. And since the downstream task is not material decomposition, it's already framed as a conditional generation task by definition.\n\n- The authors claim that generating PBR maps “from scratch” helps alleviate the ambiguity of the task. First, “from scratch” is a somewhat odd choice of words, especially considering the model is not unconditional. More importantly, it's unclear how this approach resolves ambiguity. One could argue that the task is simply easier, since it does not rely on an RGB reference. In fact, any prediction in the parameter space can result in a plausible output under the BRDF analytic model. In contrast, I would argue that decomposition methods (conditioned on RGB images) tackle a more complex problem and suffer from inherent bias, precisely because they are forced to meet specific constraints. The task chosen in this paper, text-conditioned generation, is less constrained: the model is free to generate any plausible decomposition, as long as it aligns with a coherent scene. It’s not about explaining the properties of a specific image, but rather about sampling a plausible material configuration without regard to a particular appearance. In that sense, the comparison with decomposition methods is not entirely fair.\n\n## Novelty and scope\n- The authors claim: “For the first time, we directly generate PBR maps from text as input in a probabilistic diffusion process.” This is a rather bold statement. Indeed, text prompts have already been used for PBR generation in \"Diffusion-based G-buffer Generation and Rendering\" (Xue et al. arXiv 2025). In the context of single-material generation, this has been explored in works such as MatFuse (Vecchio et al., CVPR 2024), ControlMat (Vecchio et al., 2023), and DreamPBR (Xin et al., arxiv 2024).\n\n- The idea of modeling the joint distribution has also been addressed before, notably in ControlMat, although it targets single materials. In fact, they first train a text-conditioned model and then add a ControlNet on top. If joint learning of the distribution is indeed so beneficial, this naturally raises the question of whether adding a ControlNet, or any other form of image conditioning on the resulting model would in fact yield a more powerful decomposition. Compared to a model trained directly for decomposition from the start.\n\n## Related works\n- I am wondering whether the cross-task attention is similar to how Tune-A-Video (Wu et al., ICCV 2023) models attention between the input frames? I think it’s important to position this work with respect to such approaches, which also rely on cross-task mechanisms to model interactions. See also Material Picker (Ma et al., SIGGRAPH 2025) which uses a video diffusion model to learn interactions between PBR maps. There currently seem to be no discussion positioning this part of the contribution. Also, it would be valuable if the authors positioned their approach relative to the literature on cross-task attention mechanisms. This is common mechanism found in the multi-task literature: for example in MTFormer (Xu et al., ECCV 2022). \n\n- On line 201, I don’t think it is accurate to say the light direction is “obtained”; rather, it is chosen as the reflected direction with respect to the outgoing direction. It would be good to mention that a similar rendering loss has been applied in prior works, e.g., SVBRDF capture (Deschaintre et al., Siggraph 2018), as well as the fact that they enforce sampling of the reflected direction. \n\n## Qualitative results\n- Overall, the generations look good, but my concern is whether the model is simply outputting random properties that remain internally consistent within image segments. Since the model is conditioned on text prompts alone, what needs to be verified is whether it has actually learned the correct distribution of object properties: whether it understands which objects should be metallic or non-metallic, and what their reflectance characteristics should be (glossy or rough).\n\n- Based on the results in Figure 12, I'm not so convinced it does so very well. Plausible roughness values are difficult to assess and quite subjective. However, the metallic maps can be more easily judged. For instance, several object parts are labeled with high metalness values despite being clearly non-metallic: the horse’s tail, the guitar soundhole, the spacesuit helmet visors (both), the guitar player’s eyes, and the windows of the wooden cabin. Similarly, in Figure 13, the water and tree in the Taj Mahal scene are predicted as metallic, while the sword and key are assigned rather low metalness values.\n\n## Quantitative results\n- Regarding the metric in Table 1, I am not very convinced by the results of the user study (particularly for the rendered images, specularity assessment, and coherence with the input prompt). I notice that the uncertainty in the results is greater than the difference between the baseline and the proposed method. Therefore, I do not fully agree with the conclusions drawn in Section 5.1.\n\n- Regarding the FID metric: since the albedo maps are generated via text prompts, they are naturally not aligned with the ground-truth albedo annotations. As a result, the authors resort to using the FID score. However, for the FID evaluation, only 100 images are used for the ID set and 40 for the OOD set. Is this sample size truly representative for a metric like FID? I believe that such distribution-based measures typically require larger sample sizes to be reliable.\n\n## Minor comments:\n- On line 97, authors said they \"align them [each modality] by leveraging cross-intrinsic\". The term align is a bit vague; perhaps learning the joint distribution would be more accurate\n\n- On line 201, the sentence \"Then, we obtain the light direction as the reflectance \\omega_i\", shouldn't it be \"as the reflected direction \\omega_i\" instead?\n\n## Typos:\n\n- line 54: \"pysically\"\n- line 67: It is grammatically incorrect to use citation numbers within sentences like \"They typically model the diffusion process following [19] or [36]\". Same on line 84 with \"generate PBR textures of entire 3D scenes following [7].\" and line 108 with \"Unfortunately, existing datasets, such as [34, 71, 46]\".\n\n- Line 124: I find \"paired datasets for each intrinsic property\" to be imprecise\n\n- Line 129: Simplifying the grammatical structure would help fix the sentence \"Our method with utilizing a pre-trained image prior does not require extensive PBR datasets\"\n\nQUESTIONS:\n- Wouldn't you say there are also inherent limitations to relying on text-based conditioning? I remain puzzled about the practical need for such a prompt-based approach. Text prompts in diffusion models have been shown to be relatively imprecise, as they tend to map to broad regions in the appearance space; as shown in Material Palette (Lopes et al., CVPR 2024) and TexSlider (Guerrero-Viu et al., SIGGRAPH 2024).\n\n- Regarding the importance sampling, here the strategy used is rather simplistic, shouldn’t directions other than the reflected direction also be sampled? It seems natural that sampling this direction exclusively would prevent the model from learning rougher properties.\n\n- On line 294, authors say that baselines perform worse because they rely on indoor datasets. Again, I am convinced that the problem tackled by these methods is a different that the one proposed in this paper. Anyhow, I think this still raises the question of whether the qualitative results stem from model size or from the fact that the model is learning the joint distribution. I wonder if the generative capability depends primarily on model size. If the authors had compared their approach with a smaller model, such as Stable Diffusion (v1 or v2), we could better understand whether the improvements come from model capacity or the problem formulation.\n\n- Have the authors investigated which layers are most critical for adapting the model to the downstream task? Is fine-tuning all layers necessary, and how effective is this relative to the model size? In the current setup, how many parameters are trained when applying LoRA?\n\n- Regarding the dataset size ablation (line 937), the fact that larger dataset sizes yield worse results sounds counterintuitive. Do you have an explanation for why this occurs? Isn’t this a limitation if the model’s learning capacity doesn’t scale well with more data? How does the model behave when a new domain is added to the training set, especially for the two tasks trained with fewer examples (normals and albedo)?\n\n- In the scene texture examples, I find the roughness and metallic values rather flat, that is, the values seem clustered around an average. Do you have any idea why this might be?\n\n- Regarding the image-space loss, how is backpropagation handled through all the steps? Also, have you tried rendering with multiple light directions or even an environment map during inference?\n\nLIMITATIONS:\nThe limitations section does not provide an in-depth analysis of the method’s failure modes. Instead, it mainly explains how a larger dataset could improve results. I believe a deeper understanding of the method’s real limitations and how future research could address them would be very beneficial, and this aspect is definitely missing from the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IntrinsiX, a novel approach for generating high-quality intrinsic images directly from text descriptions. Unlike traditional text-to-image models that produce RGB images with baked-in lighting effects, IntrinsiX generates physically-based rendering (PBR) maps (albedo, roughness, metallic, and normal maps) using a probabilistic diffusion process. The method involves a two-stage training process: first, training intrinsic priors for each PBR component separately using LoRA layers on curated datasets, and then aligning these models via cross-intrinsic attention and a new rendering loss with importance-based light sampling. This ensures semantically coherent and detailed PBR predictions. The core contributions of this work include being the first to directly generate usable PBR maps from text, decomposing strong image priors into intrinsic components, and combining cross-intrinsic attention with a novel rendering objective.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. It's the first method to directly generate PBR maps from text, addressing the limitations of existing text-to-image models that only output RGB images with baked-in lighting effects.\n2. The two-stage training process with cross-intrinsic attention and novel rendering loss is well-designed and effectively combines the strong image prior of pretrained T2I models with intrinsic component generation.\n3. The paper provides comprehensive evaluations with quantitative metrics and qualitative comparisons against baselines. The user study further supports the claims about the quality of the generated PBR maps.\n\nWeaknesses\nThe method requires specific datasets for training each PBR component and may struggle with more diverse or complex scenarios beyond the training data. The paper only uses 40 images for out-of-distribution evaluation, which may not reflect the true generalization ability of the method.\n\nQUESTIONS:\nHow was the decision made to use a rank of 64 in the LoRA training? Were other values explored during the experimentation process? If so, how did they compare in terms of performance and efficiency? If not, what was the reasoning behind choosing 64 without exploring other options?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework for generateing PBR maps directly from text prompts.  It leverages a pretrained flow model (Flux.dev1) and employs LoRAs for effective fine-tuning.  To further enhance the quality of generated PBR maps, the authors use a cross-view/multi-view attention which enables effective data interaction across different attributes of PBR maps (e.g., albedo, roughness/metallic, normal) on batch dimension. The paper also apply a rendering loss to enhance the modeling of PBR map distributions.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper is well-written and easy to follow. The figures are intuitive and helpful for understanding the method.\n- The proposed method shows promise in generating high-quality PBR maps from text prompts, especially showing improvements in generating clean albedo maps.\n\n### Weaknesses\n\n- Although the proposed method produces cleaner albedo maps, it still faces challenges in accurately handling reflections. Additionally, the generated normal maps frequently appear noisy. Consistency among different attributes (albedo, roughness/metallic, normal) is also lacking, as each attribute often presents disparate details.\n\nThe paper is motivated by the limitations of existing methods in decomposing RGB images into distinct PBR maps, a task inherently ambiguous and ill-posed. The authors opt for generating PBR maps directly from text prompts instead. However, for practical real-world applications, decomposing or generating PBR maps from RGB images aligns more closely with common workflows, such as image editing and relighting. Therefore, the primary advantage of the paper, generating cleaner PBR maps, is simultaneously its principal limitation regarding practicality in real-world scenarios. Nonetheless, the overall approach shows promising results, particularly given the limited training data. The proposed solution is logically sound and well-justified. Considering these points, I am leaning towards borderline acceptance.\n\nQUESTIONS:\n- Exactly how many images are used for training the albedo and normal maps? The mention in Line 130 of \"20 synthetic examples of albedo and normal maps\" is somewhat unclear.\n- Does the cross-view/multi-view attention mechanism introduce new attention layers (e.g., new linear layers for queries, keys, and values)? If not, reusing the existing attention layers might interfere with pretrained priors due to altered attention behavior, potentially complicating the second-stage training process that still only employs LoRAs.\n\nLIMITATIONS:\nPlease refer to Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}