MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models
Authors: Jonas Belouadi
Tamy Boubekeur
Adrien Kaiser
Venue: ICLR 2026 Poster (2026)
Keywords: generative models
procedural materials
appearance modeling
multimodal learning
program synthesis
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We train large multimodal language models for procedural material synthesis using a novel multimodal program synthesis framework.

Abstract: Material node graphs are programs that generate the 2D channels of procedural materials, including geometry such as roughness and displacement maps, and reflectance such as albedo and conductivity maps. They are essential in computer graphics for representing the appearance of virtual 3D objects parametrically and at arbitrary resolution. In particular, their directed acyclic graph structure and intermediate states enable a modular, interpretable workflow for interactive appearance modeling. However, creating such graphs remains challenging and typically requires professional training. While recent neural program synthesis approaches attempt to simplify this process, they solely represent graphs as textual programs, failing to capture the inherently visual-spatial nature of node graphs that makes them accessible to humans. To address this gap, we present MultiMat, a multimodal program synthesis framework that leverages large multimodal models to process both visual and textual graph representations for improved generation of procedural material graphs. We train our models on a new dataset of production-quality procedural materials and combine them with a constrained tree search inference algorithm that ensures static correctness while efficiently navigating the program space. Our experimental results show that our multimodal program synthesis method is more efficient in both unconditional and conditional graph synthesis with higher visual quality and fidelity than text-only baselines, establishing new state-of-the-art performance.

Published as a conference paper at ICLR 2026
MULTIMAT: MULTIMODAL PROGRAM SYNTHESIS FOR PRO- CEDURAL MATERIALS USING LARGE MULTIMODAL MODELS
Jonas Belouadi
Tamy Boubekeur, Adrien Kaiser Adobe Research, France [boubek, aeriserr com
University of Mannheim, Germany jonas. ban.abelouadi @uni mannieimnneemm...mm
ABSTRACT
Material node graphs are programs that generate the 2D channels of procedural materials, including geometry such as roughness and displacement maps, and reflectance such as albedo and conductivity maps. They are essential in computer graphics for representing the appearance of virtual 3D objects parametrically and at arbitrary resolution. In particular, their directed acyclic graph structure and intermediate states enable a modular, interpretable workflow for interactive appearance modeling. However, creating such graphs remains challenging and typically requires professional training. While recent neural program synthesis approaches attempt to simplify this process, they solely represent graphs as textual programs, failing to capture the inherently visual-spatial nature of node graphs that makes them accessible to humans. To address this gap, we present MULTIMAT, a multimodal program synthesis framework that leverages large multimodal models to process both visual and textual graph representations for improved generation of procedural material graphs. We train our models on a new dataset of production- quality procedural materials and combine them with a constrained tree search inference algorithm that ensures static correctness while efficiently navigating the program space. Our experimental results show that our multimodal program synthesis method is more efficient in both unconditional and conditional graph synthesis with higher visual quality and fidelity than text-only baselines, establishing new state-of-ie satee--f-hh--att the-art performance.
INTRODUCTION
Procedural materials have become increasingly important in modern 3D content creation, offering artists greater control and flexibility in designing surface appearances for digital assets. Unlike traditional image-based textures, which are constrained by fixed resolutions and limited editability, pro- cedural material modeling tools like Adobe Substance Designer (Adobe, 2025c) or Blender (Blender, 2025) leverage node-based graphs to generate textures programmatically. This enables resolution- independent exccution, high-level parametric control, and non-destructive editing workflows that have proven valuable in industries such as game development, film production, and VR/AR applica- tions (Musgrave et al., 2002). More specifically, a procedural material is defined as a directed graph where nodes represent texture generators (e.g., noise functions, patterns) or filtering operations (e.g.. blurs, color adjustments), and edges encode the flow of data between these operations, ultimately producing the texture maps required by physically-based rendering (PBR) models (Pharr al., 2016) (cf. Figure 1). However, the complexity of crafting these procedural material graphs presents substantial barrier to entry, creating a pressing need for automated and semi-automated approaches to support material artists at all levels of proficiency.
With recent advances in neural program synthesis (Huynh & Lin, 2025), procedural material synthesis has become increasingly feasible. MATFORMER pioneered this direction with multi-stage transformer- based model for unconditional generation with Adobe Substance Designer (Guerrero et al., 2022). Building on this foundation, Hu et al. (2023) extended the approach to support conditional synthesis, enabling applications such as inverse rendering (Patow & Pueyo, 2003), i.e., generating procedural materials that match the appearance of captured or rendered images. More recently, VLMATERIAL demonstrated that large language models (Zhao et al., 2025) can effectively perform end-to-er procedural material synthesis (Li al., 2025a). However, these approaches share fundamental

---

Published as a conference paper at ICLR 2026
Purameter A
Rendering
Material Engine
Engine
Generranrr
Surface Mesh
Parameter Ser S Maps
Rendering
Filler
Engine
Nodes
Procedural Material Graph
Albedo
Nommal
Metalis
Figure I: Procedural materials offer powerful control over the appearance of 3D objects through few high-level parameters. Here, production-gen -ggadd example (left) with the images obtained using two distinct parameter sets A and B (right).
limitation: they generate node graphs as text-only programs without access visual feedback during synthesis. This contrasts sharply with how human artists work, who create procedural materials by manipulating node graphs through arguably more intuitive visual interface, illustrated in Figure (left). Without visual feedback, models must rely solely on textual representations to reason about complex spatial relationships and visual outcomes, task that becomes increasingly difficult as material complexity grows. To address this limitation, we propose a novel multimodal program synthesis paradigm based large multimodal models (Yin al., 2024) that incorporates visual feedback throughout the generation process, more closely mirroring human creative workflows. We demonstrate that this approach, which we refer as MULTIMAT, outperforms previous state-of-the-art methods (cf. $6). Our key contributions are as follows:
1. We introduce MULTIMAT, a novel procedural material synthesis approach that incorporates visualizations of intermediate graphs, including node states, into its context. This multimodal feedback loop improves material quality substantially compared to text-only baselines.
2. Investigating intermediate states enables real-time validation of each generated node. This allows to develop tree search algorithm that backtracks upon encountering invalid states, enabling more efficient inference than prior methods, which often produce invalid graphs.
3. We implement a transpiler that converts between Adobe Substance Designer formats and compact representation suitable for language modeling while supporting the complete feature set. This enables training on larger datasets and the generation of more complex materials than previous approaches, which examined only limited subsets of Designer's capabilities.
2 RELATED WORK
Large Language Models for Program Synthesis Our work builds upon recent advances in neural program synthesis (Parisotto et al., 2017; Devlin et al., 2017: Thakoor et al., 2018; Ye al., 2021; Ellis et al., 2021). Traditional program synthesizers require formal specifications and employ search or logical derivation to produce programs that provably satisfy these specifications (Alur et al., 2013). Recently, large language models have demonstrated impressive capabilities in this domain (Huynh Lin, 2025: Li et al., 2025b; Lozhkov et al., 2024: Li et al., 2023b; Rozière et al., 2023; Fried et al., 2023; Li et al., 2022; Chen et al., 2021). However, current research predominantly targets high-resource programming languages such as Python, Java, or JavaScript (Zan et al., 2023; Huynh & Lin, 2025). In contrast, work synthesizes graphics programs, which pose unique challenges due to domain-specific requirements and considerable data scarcity, establishing as a distinct research area.
Graphics Program Synthesis Deep learning approaches have shown strong performance in synthesizing graphics programs that compile to visual outputs (Ellis et al., 2018; 2019; Ganin et al., 2018). This progress has been accelerated by the emergence of large multimodal models, particularly vision-language models that bridge visual and textual domains (Alayrac et al., 2022; Liu et al., 2023; Belouadi et al., 2024b; Kulits et al., 2024: Li & Ellis, 2024; Kapur et al., 2025; Lin et al., 2025: Xu et al., 2025). The field encompasses both controlled experimental settings using domain-specine languages (Ellis et al., 2018; Tian et al., 2019; Sharma et al., 2018; Cámara et al., 2023; Kulits et al., 2024; Kapur et al., 2025; Wen al., 2025) and practical applications. Notable examples include
2

---

Published as a conference paper at ICLR 2026
Multimodal Program Tree 7
Material
Ga
Materiul Engine
Engine
Rendering
G/1
Image
Vati
Engine
MultiMat
Definition
New Node
"..
Transpiler
Pinished
Parameter Oplimization
Procedural
Material
<bos>
Program
Inpul
Generate
Validate
Oulpul
Figure Architecture overview of MULTIMAT during inference. The system constructs multimodal program tree T by iteratively generating node definitions. At each step r, the system derives graph G, of valid nodes along with corresponding intermediate outputs I, by traversing T, which may contain both valid and invalid nodes, to generate the next node 1r+1. When transpilation and execution succeed, the system advances with an updated graph Gr+1 and outputs 1,+1- If errors occur, it reverts to previous state (Gst. The generation process initiates from either input image or unconditionally using a beginning-of-sequee optimization (cf. $6.2), the final procedural material can be applied any target geometry for rendering.
token (<bos>). Following optional parameter
systems for generating scientific figures using TikZ (Belouadi et al., 2024a;b; 2025; Laurençon
et al., 2024; Laurençon et al., 2024; Tong et al., 2024; Zhang et al., 2025) and automating data visualization (Mackinlay, 1986; Roth et al., 1994; Luo et al., 2021; Wu et al., 2024: Voigt et al., 2024). However, these approaches generate code designed for text-based editing and therefore do not face the unique circumstances of node graphs in procedural material synthesis that our work addresses.
Procedural Material Synthesis Procedural material modeling is one of the most challenging domains in graphies program synthesis. The combination lengthy, complex material programs and severe data scarcity creates unique obstacles for learning-based approaches (Li et al., 2025a; 2024). Existing methods primarily focus on inverse procedural material modeling by synthesizing graphs that reproduce a given target appearance (Hu et al., 2023) unconditional generation to create diverse, novel materials without specific targets (Guerrero et al., 2022). related line of work optimizes parameters of existing material graphs match image targets by transpiling them into differentiable programs (Shi et al., 2020; Hu et al., 2022; Li et al., 2023a). As discussed in $1, previous generative approaches limited text-only representations, limitation address in this work.
3 BACKGROUND ON PROCEDURAL MATERIALS
As indicated in $1, procedural materials are directed acyclic graphs G, executed by material engine to produce raster images representing the physical properties of materials. These so-called material maps define surface characteristics, e-g., albedo, roughness, normal (tangent space orientation), that enable photorealistic rendering when applied to 3D objects, with their appearance controlled through small set of high-level parameters (cf. Figure 1). The internal structure of material graph G comprises nodes (v1, ......, VN) connected by edges that define the flow of image data. Each node v; functions as either generator that creates new image content or filter that transforms existing images from upstream nodes. Common node operations include noise generation, blending, and mathematical transformations, which collectively produce intermediate image outputs / (ú,, f2,.... .iN). The behavior of each node is governed by parameters that may be discrete or continuous scalars or vectors, providing fine- mre-graided control over the final material appearance.
Professional material authoring tools such as Blender and Adobe Substance Designer enable artists to construct and modify procedural material graphs through visual interfaces (cf. Figure 1). Users can interactively add or remove nodes and edges while adjusting node parameters to achieve desired visual effects. Among these tools, Adobe Substance Designer stands out for its particularly expressive node graph system, which MULTIMAT specifically targets. It offers advanced capabilities for creating complex material appearances through features like function graphs and pixel processors. Function graphs allow parameters to be controlled through custom operations input values, while pixel
3

---

Published as a conference paper at ICLR 2026
COMPACTSBS Visualization
COMPACTSBS
Detokenizer
dagendanc::
Gy
MuLIMar
/lazt trandeen..k
Graph
Iogur:
mode:
abselitte:
85
Visian Encoder
Tokenizer
Yrel
Mixed
Figure 3: Visualization of the two conditioning approaches used by MULTIMAT for generating node definition V/+1. In the graph-conditioned approach (1), MULTIMAT processes the graph G, visual representation similar to human perception. In the mixed-conditioned approach (2), MULTIMAT receives G, as a multimodal program where <img> tokens are replaced with their corresponding vision encoder representations from 4.
processors enable users to define specialized computational graphs that operate on individual pixels using sequences of atomic mathematical operations. These sophisticated capabilities make automated procedural material synthesis particularly challenging problem in this domain.
4 THE MULTIMAT MODEL & ARCHITECTURE
Figure 2 illustrates complete model pipeline. At its core, MULTIMAT is vision- langunl model, trained for synthesizing procedural material graphs. It accepts images as input for inverse procedural material synthesis and supports unconditional generation. Unlike previous approaches, MULTIMAT generates nodes topologically, ensuring each node precedes all nodes connects to. This enables an iterative generation process detailed below that can provide continuous visual feedback the model, verify the validity of intermediate outputs, and recover from errors automatically in certain cases.
4.1
MULTIMODAL. PROGRAM SYNTHESIS
Given a partially generated material graph G, = (V1, v2..... with nodes v; at generation step 1, the topological ordering of nodes allows for visualizing intermediate node states, similar to visual editing environments that target humans. This enables an iterative generation loop where MULTIMAT generates one node ciinitionn-including node parameters and connections to previous nodes time that is processed accordingly before the generation continues. After generating node Fr41 in an intermediate text format (cf. $5), combine it with the existing node definitions (Vi,..., and feed them to a transpiler, which compiles the intermediate representations back to a format the material engine understands. We then use the material engine visualize the state of node v,- Upon successful transpilation and execution, V/+1 is appended to the graph G This updated state, including the visualized intermediate outputs I,. is fed back to the model to generate the subsequent node Vr+2 (cf. Figure 2). If execution or transpilation fails, we discard the current v, and resample, or backtrack further in case of repeated errors (cf. $4.2). We explore two complementary approaches for representing G, and I, as multimodal programs to the model, as visualized in Figure 3:
Mixed Conditioning Starting with textual representation of C, (cf. $5), we enhance each node v; with an additional field containing visualized intermediate state. This creates multimodal program where the model processes textual tokens interleaved with image patch embeddings (cf. Figure 3). To manage the increased context size from image embeddings, we omit node parameters (which are implicitly encoded in the visualizations) but explicitly include node output type information (e.g., grayscale or color) that the model cannot infer from the visualization alone.
Graph Conditioning
This approach more closely mirrors human visual experience by conditioning MULTIMAT solely on visualization of the entire graph G, with embedded intermediate visual outputs I, as shown in Figure 3. The model generates subsequent node Vr-1 using only this complete visual context, without explicit access to underlying textual node definitions.
4

---

Published as a conference paper at ICLR 2026
(I)
(2a)
(2b)
(3)
Figure 4: Visualization of our inference algorithm as a tree search. Tree nodes represent generated node definitions, and edges represent possible continuations. The algorithm proceeds as follows: generation continues until an invalid state (X) encountered (1), triggering backtracking to the previous node; from this point, if valid node (v) is generated, normal generation resumes (2a), but invalid outputs persist (2b), the algorithm backtracks further until a valid path is found (3).
At their core, both approaches remain autoregressive language models, and MULTIMAT can be trained by minimizing a cross-cnte crosspyy objective:
==
log p( Vr,s Vrycss Gr, (,x;x),
where Vr.s the expected token of in our intermediate text format position s, vr,ss represents previous tokens, x denotes the input conditions (which can be empty for unconditional generation), and represents the model parameters.
4.2 INCREMENTAL TREE SEARCH
Another advantage of topological node ordering is the ability to validate node definitions incrementally during generation. By invoking our transpiler and material engine at each step, we can detect syntactic and semantic errors immediately rather than waiting until the entire graph is complete. When erroneous node definition is encountered, first discarding and resampling the problematic node, and errors persist, inferring deeper structural issues by reversing further back in the generation sequence. Specifically, we discard the ((--1) most recently generated nodes, where i represents the current backtracking iteration. This approach effectively transforms our generation process into an incremental tree search on tree T of valid and invalid nodes (cf. Figure 4), systematically exploring the solution space to discover valid programs. This incremental validation approach identifies invalid outputs much faster than previous approaches, which require sampling complete programs before validation can commence.
execute adaptive backtracking strategy:
4.3
AUTOMATIC ERROR REPAIR
Through systematic analysis of failure cases, identified recurring erгor patterns that could be repaired automatically: (1) removal extraneous parameters that are specified for node types that do support them, and (2) automatic insertion of conversion nodes to resolve type mismatches between connected nodes. For instance, when a color output erroneously connected to a grayscale input, we automatically insert appropriate grayscale conversion node. Conversely, when grayscale output feeds into a color input, we insert gradient map node to perform type conversion. These repair mechanisms increase the proportion valid generations without requiring additional sampling steps.
DATASET
To support the training and evaluation MULTIMAT, we collect procedural materials from Adobe's Substance 3D Assets Repository (Adobe, 2025a). Unlike previous work that either focuses basic graphs utilizing only a subset of Substance Designer features (e.g., lacking complex nodes such as pixel processors or function graphs; Guerrero al., 2022: Hu et al., 2023) or targets other tools with more limited capabilities (Li et al., 2025a), our approach supports the complete feature set. This comprehensive coverage enables us to collect over 6 6000 unique materials, substantially more than existing datasets. Table 1 summarizes key characteristics of our dataset compared to prior work.
5

---

Published as a conference paper at ICLR 2026
Human-Readable Graph Representation Sub-
Models
Size
Max Nodes
Feature Set
Program
stance Designer's native file format (SBS) has not been designed for human readability, containing verbose XML structures, embedded binary data, legacy metadata, and other implementation details, which makes direct lan- guage modeling impractical. To address this, we develop bidirectional transpiler that converts between SBS and compact, human-readable YAML-based representation with topological node order, which we call COMPACTSBS. Unlike previous approaches that support only partial feature sets (Guerrero et al., 2022; Hu al., 2023), transpiler preserves the complete functionality of Substance graphs with programs that are, on average, over 80% shorter. Models operate exclusively in COMPACTSBS, with outputs transpiled back to SBS for execution. We provide representative examples in Figure 3 and complete program listings in Appendix A.
MATFORMER
2820
400!
Subset
Designer
MAT. (COND)
4.667
S 80
Subset
Designer
VLMATERIAL
3663
30
Limited
Blender
MULTIMAT
6878
128
Complete
Designer
Upper bound in complex filtering pipeline, actual could be less.
Table I: Comparison of training data of MATFORMER (Guerrero al., 2022), conditional MATFORMER (Hu et al., 2023), VLMATERIAL (Li al., 2025a), and MULTIMAT (ours). We assembled the largest dataset with the most comprehensive set of features.
Graph Preprocessing Our preprocessing pipeline standardizes graphs for the PBR workflow, focusing on five essential texture maps: base color, normal, roughness, metallic, and height. We trace backwards from these outputs to identify all contributing nodes, pruning unconnected components and other output maps. Graphs containing embedded bitmap graphics and SVGs are excluded to keep graphs fully procedural. We further filter out graphs exceeding 128 nodes and flatten hierarchical structures by inlining nested subgraphs and custom author dependencies into the main graph. Non-atomic nodes from the standard Substance Designer library remain as external references.
6 EXPERIMENTS
We build MULTIMAT models upon the QWEN2.5VL (7B) vision-language model which leverages a late fusion approach to combine image and text tokens (Bai et al., 2025). We train and evaluate separate models for unconditional generation (cf. $6.1) and inverse procedural material synthesis (cf. $6.2). We also conduct a human evaluation (cf. $6.3). Across all model variants, consistent maximum sequence length of 192 tokens. The training setup consists of 5 epochs using ADAMW (Loshchilov & Hutter, 2019), learning rate of 5e-5, and batch size of 128. To ensure diversity in our generated outputs, we set the inference sampling parameters to a temperature of 0.8 and top-p value of 0.95. We provide examples Figure and Appendix A. We ablate incremental tree search in $7.
maintain
6.1 EVALUATION OF UNCONDITIONAL. GENERATION
For unconditional generation, the mixed conditioning variant, MULTIMAT (Mixed), embeds node previews 140 x 140 resolution, resulting in 25 patch embeddings per image. For the graph conditioning variant, MULTIMAT (Graph), graph visualizations can utilize up to 6 144 tokens, with larger images downscaled accommodate this limit. We generate 100 outputs per model for evaluation.
Baselines For text-only procedural material synthesis. VLMATERIAL represents the current state-of- the-art approach. However, its elender-specific training makes direct comparison with our method difficult. We therefore create VI.AATEIAAL (SBS) by retraining a VTLMATERIAL-SSYle model on our dataset for fair comparison. Unlike the objective in equation (1), VLMATERIAL, is trained to generate complete graphs in a single pass. However, during inference, we can still validate nodes as they are generated and roll back upon detecting irreparable errors (cf. $4.2) or repair them after generation completes (cf. $4.3). This means the progression from VLMATERIAL (SBS) to MULTIMAT (Mixed) to MULTIMAT (Graph) represents comparable, gradual shift from complete graph generation toward iterative node generation. Since VLMATERIAL (SBS) does not receive any images in the unconditional setting, we base it the larger and more powerful text-only model QWEN3 (8B; YANG AL.,

---

Published as a conference paper at ICLR 2026
Models
DSIM,
C
STYLEJ
KID;
ROUGE-L
NER
VLMATERIAL (SBS)
31.344
65.678
3.211
14.976
1.621
16.933
MULTIMAT (Mixed)
34.922
66.737
3.199
3.675
2.194
12.388
MULTIMAT (Graph)
36.609
67.907
3.178
2.801
2.037
17.046
VLMATERIAL" (SBS)
31.348
65.867
3.126
27.862
MULTIMAT" (Mixed)
40.258
69.687
3.093
17.792
MULTIMAT' (Graph) 40.367
70.114
3.046
14.886
Table 3: System-level scores x 100 for conditional (inverse) generation, without (top) and with (bottom) parameter optimization. Bold and underlined values indicate the best and second-best scores for each metric column, respectively. Arrows indicate metric directionality. ROUGE-L and NER scores remain unchanged by parameter optimization and are shown only once. MULTIMA (Graph) and MULTIMAT (Graph) achieve the best overall performance.
MULTMAr* (Mixed)
83.3%
16.7%
VLMATERIAL" + (SBS)
MULTIMAR (Graph)
81.1%
18.9%
VLMATERIAL." (SBS)
MULTIMAT (Graph)
58.3%
41.7%
MULIMAT" (Mixed)
Figure 5: Human preferences for model outputs diverging bar chart. MULTIMAT* (Graph) the most preferred model overall, while VLMATERIAL" (SBS) is consistently the least preferred.
2025A), giving it slight advantage over our models. While graphics program synthesis research typically also benchmarks against proprietary large language models such as GPT-4o (OpenAl et al., 2024) or CLAUDE 4 (Anthropic, 2025), which have demonstrated competitive performance related domains (Belouadi et al., 2024a;b; 2025; Rodriguez et al., 2025), these models' unfamiliarity with COMPACTSBS and inability to produce valid SBS output preclude their inclusion as baselines.
Metrics Our multimodal task permits diverse evaluation schemes for automatic evaluation. To evaluate the visual quality of generated materials, we compute the Kernel Inception Distance (KID; Binkowski et al., 2018), which compares the distribution of generated material maps with material maps from our dataset. To detect degenerate low KID scores due to memorization of training data (a legitimate concern given our relatively small dataset), also calculate ROUGE-L scores (Lin, 2004) between the COMPACTSBS representation of our generated materials and the training set (with masked effective
parameters). This metric computes the longest common subsequence and serves memorization indicator (Hans et al., 2024). Notably, we specifically require consecutive subsequences due to COMPACTSBS's limited syntactic diversity, which could otherwise produce misleading matches. To measure efficiency, we introduce the Node Error Ratio (NER), defined as the average ratio between discarded nodes and the total number of generated nodes.
Results Table 2 presents the system- level metric scores for our evaluation. MULTIMAT (Graph) leads in visual quality with the lowest KID score, outperforming MULTIMAT (Mixed) by over 4pp (percentage points) and VI- MATERIAL (SBS) by more than 1pp. This considerable gap in performance suggests that the better the visual repre- sentations are aligned with human cre- ative workflows, the better the results- an intuitive but important finding. All models exhibit minimal memorization, with ROUGE-L scores showing that no more than 4% of any generated sequence matches contiguous segment from the training data. Nonetheless, both MULTIMAT variants demonstrate approximately 1.5pp lower copying rates compared to VLMATERIAL (SBS), suggesting slightly better generalization. Regarding efficiency, MULTMMAT (Mixed) excels with
Models
KIDJ
ROUGE-L
NER
VLMATERIAL (SBS)
14.155
3.641
14.846
MULTIMAT (Mixed)
6.752
2.195
8.923
MULTIMAT (Graph) 2.365
1.915
15.024
Table 2: System-level scores x 100 for unconditional gen- eration. Bold and underlined values indicate the best and second-best scores for each metric column, respectively. Arrows indicate metric directionality. MULTIMAT (Graph) achieves the best overall performance.
7

---

Published as a conference paper at ICLR 2026
the lowest NER, achieving a 6pp improvement over the other models. Both MULTIMAT (Graph) and VLMATERIAL (SBS) show comparable NER scores around 15%. For MULTIMAT (Graph), these errors primarily due to OCR-like errors in reading node names and function types embedded as text in graph images. In contrast, we attribute the errors in VLMATERIAL (SBS) to fundamental difficulties in understanding graph structures. Despite these limitations, the error rates remain within acceptable bounds for practical applications, and MULTIMAT (Graph) emerges as the best overall model.
6.2 EVALUATION OF CONDITIONAL GENERATION
As in prior work (Hu et al., 2023; Li et al., 2025a), we train inverse MULTIMAT variants that learn to generate procedural materials from rendered images. These models follow the same training procedure as their unconditional counterparts, with one key modification: each training example preceded by 512 x 512 rendering of itself, which adds 324 additional image patches to the model context. During inference, the model takes an image input and generates a corresponding procedural material. We reserve 100 examples from our data as held-out test data for evaluation.
Baselines Analogously $6.1, we adapt VLMATERIAL for inverse rendering with SBS and use it as baseline. Since an image input is now required for VLMATERIAL (SBS), we also base it on QWeN2.5v1. (7B) instead of QWEN3 (8B) and train it using the same method as MULTMAT.
Parameter Optimization To further refine generated materials, we apply gradient-based opti- mization using differentiable rendering. This approach has proven effective for optimal parameter estimation (Shi al., 2020: Hu et al., 2022; Li et al., 2023a; Hu et al., 2023). We employ DiffMat (Shi et al., 2020; Li et al., 2023a), a widely adopted differentiable renderer for Designer materials, to optimize the generated graphs against the input images. Models using this refinement step are denoted as MULTIMAT and VI.ATTEIIAI, respectively.
Metries addition to the metries from $6. I, we evaluate reconstruction quality by rendering the generated materials and comparing them to the input images using perceptual similarity metrics. Specifically, we measure cosine similarity between CLIP image embeddings (Radford et al., 2021; Hessel et al., 2021), compute STYLE Loss loss (STyLE: Gatys et al., 2016) as the L1 distance between Gram matrices of VGG features, and calculate DREAMSIM (DSIM; Fu et al., 2023), a learned perceptual similarity metric designed to align with human judgments.
Results Table 3 presents the system-level metric scores for conditional evaluation. The perceptual similarity metrics consistently demonstrate that MULTMAT (Graph) achieves the highest fidelity to input images, with MULTIMar (Mixed) performing second-best and VLMATERIAL (SBS) ranking last. For example, DREAMSIM scores are 36.609, 34.922, and 31 31,344, respectively, a ranking that mirrors our unconditional evaluation results. Parameter optimization yields substantial improvements in perceptual similarity, with MULTIMAT" (Graph) and MULTIMAT" (Mixed) showing average gains of 6% and 8%, respectively. In contrast, VLMATERIAL* + (SBS) exhibits minimal improvement (only 1 1%), suggesting its outputs deviate too far from the input for parameter optimization to be effective. Interestingly, while parameter optimization improves perceptual similarity, KID scores increase. This could occur because optimization aligns outputs more closely with the test set, which represents only a subset of the training distribution, potentially increasing distance from the full distribution. Nevertheless, both MULTIMAT and MULLIMT' variants outperform VTMATEIIA (SBS) and VLMATERIAL* (SBS) on KID by over 10pp, respectively. The remaining metrics reinforce trends from unconditional evaluation. ROUGE-L scores do not exceed 2% (indicating minimal memorization), and MULTIMAT (Mixed) produces the fewest errors. Overall, MULTIMAT (Graph) and its optimized variant, MULTIMAT" (Graph), deliver the strongest performance across metrics.
6.3 HUMAN EVALUATION
To corroborate our automatic evaluation results, we conduct human evaluation. We employ comparative annotation (Thurstone, 1927) and focus on the image reconstructioon/ineerse rendering case, which allows for intuitive human assessment (cf. Figure 6). Annotators receive triplets of rendered generated materials from VLMATERIAL* + (SBS), MULTIMAT (Mixed), and MULTIMAT (Graph) and identify which output best and least resembles the input image. Following Hu al.
8

---

Published as a conference paper at ICLR 2026
Input
VLMATERIAL" (SBS)
MULTIMAr" (Mixed)
MULrIMAT" (Graph)
Figure 6: Qualitative examples for inverse procedural material modeling following the setup of our human evaluation in $6.3. The leftmost column shows input materials from graphs filtered during preprocessing, making these particularly challenging test cases. MULTIMAT* (Mixed) consistently outperforms VLMATERIAL* (SBS), while MULTIMAT* (Graph) achieves the best results overall. Additional examples, including failure cases, are provided in Appendix A.
(2023); Li et al. (2025a), we generate multiple programs (N 40) per model and image, selecting the result with the highest DREAMSIM score the final candidate. We test 33 input materials from graphs filtered during preprocessing (e.g., due to excessive length), which represent particularly challenging cases. Eight expert annotators with extensive procedural material experience assess each triplet in randomized order. As shown in Figure 5, annotators rank VLMATERIAL" (SBS) considerably lower than MULTIMAT (Mixed) and MULTIMAT* (Graph), and prefer MULTIAr (Graph) over MM (Mixed). These findings align with our automatic evaluation rankings and demonstrate our approach's ' effectiveness in generating perceptually similar materials.
7 ANALYSIS & DISCUSSION
Our comparisons demonstrate that model performance improves steadily as the degree graph visualization increases, with MULTIMAT (Graph) achieving the highest performance overall (cf. Tables & 3: Figure 5). This finding aligns with how humans interact with procedural materials- through visual node graph interfaces- and validates established UX design principles in this domain.

---

Published as a conference paper at ICLR 2026
The qualitative examples in Figure 6 fur- ther illustrate this trend, with VLMATERIAL"
Models
Deletions
Conversion,
(SBS) struggling to generate faithful outputs, indicating that purely text-based approaches not ideal for expressive node graph sys- tems like Designer. This limitation persists even with more powerful base models, our unconditional generation experiments con- firm. Beyond architectural improvements, our tree search algorithm enables more ef- ficient graph generation; without it, models may have to resort to sampling complete out- puts for validation (the inference approach used by previous methods), which is expen- sive. For instance, disabling tree search causes NER of VLMATERIAL (SBS) to deteriorate further from 14.846 to 33.953, highlighting how search strategy can improve inference without further training. The impact of automatic error repair is more nuanced, as shown in Table 4. Only approximately 1% of nodes generated by MULTIMAT contain hallucinated parameters, and fewer than 6.5% require conversion. In contrast, VLMATERIAL exhibits nearly double the scores for both repair mechanisms. This difference demonstrates that VLMATERIAL requires considerably more repair than our models and supports our claim that our models possess better understanding of graph structures. Notably, since corrections are not fed back to the models, these results reflect their intrinsic generation capabilities.
VLMATERIAL (SBS)
2.71
12.26
MULTIMAT (Mixed) 1.18
3.51
MULTIMAT (Graph) 1.1
6.49
Table 4: Percentage of nodes repaired through pa- rameter deletion conversion node insertion in our unconditional and conditional evaluations. Bold and underlined values indicate the best and second-best scores for each metric column, respectively. Arrows indicate metric directionality, Our MULTIMAT models require the least amount of repair.
8 CONCLUSION
We present MULTIMAT, a multimodal program synthesis framework and model suite that generates procedural materials by incorporating visual feedback throughout the generation process. Our key insight is that procedural material graphs are inherently visual-spatial programs, and treating them such leads to substantial improvements over text-only approaches. By conditioning on visual intermediate states-either interleaved with text (mixed conditioning) or as complete graph visualizations (graph conditioning)- models achieve consistent improvements over text-only baselines. Our incremental tree search algorithm further enhances generation efficiency by validating nodes as they are created and backtracking upon errors. While we demonstrate MULTIMAT specifically for procedural material synthesis, we hope its general principles will inspire further research at the intersection of computer graphics, program synthesis, and multimodal AL.
The development of procedural material graph synthesis approaches is currently constrained by limited training data availability. We plan to address this challenge through self-learning techniques (He al., 2020; Wei al., 2021) that leverage our unconditional models to generate synthetic supervised training data by rendering outputs and subsequently training conditional models on this expanded data. Additionally, we aim develop a unified model trained across multiple node graph systems to investigate potential transfer learning benefits (Pan & Yang, 2010). Beyond methodological advances, our models offer promising practical applications: conditional models could extract material graphs directly from photographic regions, while unconditional models could power intelligent auto-completion features user interfaces. Furthermore, our methodology naturally extends to related domains such vector graphics synthesis (Wu et al., 2023; Polaczek et al., 2025; Rodriguez et al., 2025; Yang et al., 2025b), where visual editing interfaces are similarly prevalent.
Future Work
Limitations Although our models and baselines use the same or similar base models, they generate graphs in fundamentally different ways, resulting in considerable differences in training efficiency. Text-only models like VI.MATERIAL. can process entire graphs as single training examples, whereas MULTIMAT must adapt the visual context for each individual node, effectively processing training examples one node at a time. This difference leads to much longer training times: while VLMATERIAL completes training in few hours on 8 x A000 80GB GPUs, MULTIMAT models require several days on the same hardware despite being trained on a comparable number of tokens. Nevertheless, since the amount of procedural materials is very small (regardless of the dataset), training times remain within acceptable bounds in absolute terms, despite the relative differences between methods. Additionally, both approaches achieve more similar throughput during inference.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We ensure that all procedural materials collected for model training are properly licensed and explicitly permit such usage, thereby preventing any copyright infringement. In adherence to this principle, we specifically exclude Substance 3D Community Assets (Adobe, 2025b) from our training data due to licensing restrictions. While we acknowledge the use of generative models in preparing this manuscript, their application is strictly limited to writing assistance, such as paraphrasing, spell checking, and synonym suggestions.
ACKNOWLEDGEMEENS
We thank the Adobe Substance 3D team for providing access to the Substance 3D Assets Repository and the Substance Automation Python API. We also thank our annotators for their valuable time. This work was conducted while the first author was intern at Adobe Research, France.
REFERENCES
Adobe. Substance 3D Assets. https://substancedddaddb wtw.aaaseeeeccmsettttssssss 2025a.
Adobe. Substance 3D Community Assets. http:://uubstaaccidd.adoeee comm://ccc...ooo...cocooooccccii cow.communiccooooooooooommm.cc.o ty- assets, 2025b.
Adobe. Substance 3D Designer. 2025c.
http:://www..dddde.cc.oopppoop/s htmm,
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and others. Flamingo: a visual language model for few-shot learning. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrrviiet fet. ddbbuuimbbbbsusbss...
Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax- guided synthesis. In 2013 Formal Methods in Computer-Aided Design, pp. 1-8, 2013. doi: 10.1 10.1109/FMCAD.. 201 SET.CAD.2113.6699300 3.6799885.
Anthropic. System card: Claude Opus 4 & Claude Sonnet 4, 2025. URL https://ww-cdn. pddf.
anthropic.ccmm/22bb444babbb 22002//0000005000002222222220330
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and others. Qwen2.5-VL. techhiiaal report, 2025. URL https: /tt::://axxvvvvee wwwrxiv.orr//iie abs//50b/2222 13923.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. AutomaTikZ: Text-guided synthesis scientific vector graphics with TikZ. In The Twelfth International Conference Learning Representations, Vienna, Austria, May 2024a. URL https:: http:::///ppeeeviiie net/ fo
Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ. In The Thirtty-ightt Annual Conference on Neural Information Processing Systems, Vancouver, Canada, December 2024b. URL https: //openreview het:/forum2dd-beLLFOOC
Jonas Belouadi, Eddy 1lg, Margret Keuper, Hideki Tanaka, Masao Utiyama, Raj Dabre, Steffen Eger, and Simone Paolo Ponzetto. TikZero: Zero-shot text-guided graphics program synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Honolulu, Hawaii, October 2025.
Mikolaj Birikowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https: //openreview. net ///formmmmddrr110002c
11

---

Published as a conference paper at ICLR 2026
Blender. Blender. https://ww..blent org. 2025.
Javier Cámara, Javier Troya, Lola Burgueño, and Antonio Vallecillo. On the assessment of generative Al in modeling tasks: an experience report with chatgpt and UMI.. Softw. Syst. Model., 22(3))781- 793, 2023. doi: 10.1007/S10270--0220 105-5. URL http:::tttoioorrgg//eoooo wwwpoio.ogg//... 1067/ 52
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. Evaluating large language models trained on code, 2021.
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural program learning under noisy I/O. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning. volume of Proceedings of Machine Learning Research, pp. 990-998. PMLR, 06-11 Aug 2017. URL https:tt////.. wtt.:httts:/ppceeeeeeeee http//proceeeng..///// mlr. www.u..reess///// hee//dvvlinnna/ html.
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. Thirty-second Conference on Neural Information Processing Systems, pp. 6062-6077 2018. URL http: //pperrs/ppp.......///// www..ninccc...o 1aaper/7845-leamiin to- dww-d-dawwa----- images.
infer-graphics-peee programs
Kevin Ellis, Maxwell Nye. Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with a REPL. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox. and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:ttpps//ppoopeeeteeeeee http://rrrreiiiinn...iiiisspppps nwwrips.e....m.. cc/paper_ httppp///....a.a...///////////// Paaeaaaaa------..... pdf.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2021, pp. 835-850, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383912. doi: 10. 00.114///45348.34454000
URL https: //oii//oi/// org/ 1145 3454488
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: generative model for code infilling and synthesis, In The Eleventh International Conference on Learning Representations, 2023. URL
https://openrrevie wwtpeeeviiiievefforeeeeeeeee net/forum?id-howb-bo IbM6ELL
Stephanie Fu, Netanel Yakir Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. DreamSim: Learning new dimensions of human visual similarity using synthetic data. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL fh1k7.
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. Synthesizing programs for images using reinforced adversarial learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1666-1675. PMLR, 10-15 Jul 2018. URL https::/prooce httpps//prooeeedings mlr. www.mmees./m./t..t...t..t.w ppwpssv80/eeeeepppppppe ///////00///nnnn///// aaaain18..hmm...
Leon A. Gatys, Alexander Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomir Mëch, Tamy Boubekeur, and Niloy J. Mitra. MatFormer: generative model for procedural materials. ACM Trans. Graph., 41)4), July 2022. ISSN 0730-0301. doi: 10.1145/3528223335228 3533017. URL https:///ooi (iiidi..og//ii 1145/ 3530173.
12

---

Published as a conference paper at ICLR 2026
Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, and Tom Goldstein. Be like goldfish, don't memorize! mitigating memorization in generative LLMs. In The Thirtty-ightth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview net net/ffrunmi htt///ooommmddd/y/oae
Jumxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. Revisiting self-training for neural sequence generation. In Proceedings of ICLR, 2020. URL https://openrrviert wwt.wiewwwweeennnr
id=S]gdnAVKDH.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Bras, and Yejin Choi. CLIPScore:
reference- free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.cmmg 595n URL httph::t//aaeee http:s/://aaaanhhlooo/oo hrrpollooy.rgg///// 2rg/2221. mannlppmmii.995
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. Node graph optimization using differentiable proxies. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393379, doi: 10.1145535282333533733 URL https://doi org/10. 1145/3528233. 3530733.
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. Generating procedural materials from text or image prompts. An SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, New York, NY. USA, 2023, Association for Computing Machinery. ISBN 9798400701597. doi: 10. 10.1145/388433..3591 3591520.
URL https://doi org/10. 114411/4//3388..
Nam Huynh and Beiyu Lin. Large language models for code generation: A comprehensive survey of challenges, techniques, evaluation, and applications, 2025. URL https://axxiv.org 2503.0 01245.
Shreyas Kapur, Erik Jenner, and Stuart Russell. Diffusion on syntax trees for program synthesis. In The Thirleenth International Conference on Learning Representations, 2025. URL https: // //openreview net/ http/fooummiddeee forum?idd=wN3KKXXXA5
Peter Kulits, Haiwen Feng, Weiyang Liu, Victoria Fernandez Abrevaya, and Michael J. Black. Re-thinking inverse graphics with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856 URL. http:///pppppeeeeeeeee//orrrrree
Hugo Laurençon, Leo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https://opeereviiii wtwpeenrwwieeeeeeeeoeeeeeeeeeeee peet/forummiddddt/WFyy222
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and léo Tronchon. Building and better understanding vision-language models: insights and future directions, 2024. URL https: //ar:iv////ivv httaaarxivviogr... org/abs/ 2408. 12637.
Beichen Li, Liang Shi, and Wojeiech Matusik. End-to-end procedural material capture with proxy-free mixed-integer optimization. ACM Transactions on Graphics (TOG), 42(4):1-55 2023a.
Beichen Li, Yiwei Hu, Paul Guerrero, Milos Hasan, Liang Shi, Valentin Deschaintre, and Wojeiech Matusik. Procedural material generation with reinforcement learning. ACM Trans. Graph., 43(6). November 2024. ISSN 0730-0301. doi: 10.1145/3687979, URL https:://///ooorrg 3687979.
1145/
Beichen Li, Rundi Wu, Armando Solar-Lezama, Changxi Zheng, Liang Shi, Bernd Bickel, and Wojciech Matusik. VLMaterial: Procedural material generation with large rision-languar models. The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview www.riiwwwnntttonniiiiieee met//forun?iddd:leebbbbhht!
13

---

Published as a conference paper at ICLR 2026
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhski.. Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Joel Lamy-Poirier. Joao Monteiro, Nicolas Gontier, Ming- Ho Yee, and 39 others. StarCoder: may the source be with you! Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL https: https://openreview www./eoouumidd-oooooooo Reproducibility Certification.
Wen-Ding Li and Kevin Ellis, Is programming by example solved by LLMs? In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
//openreview. http:/vvwweeeeeettonriiiiiiieeee heee/forumnidddccsyyss....
Wen-Ding Li, Darren Yan Key, and Kevin Ellis. Toward trustworthy neural program synthesis. In ICLR 2025 Workshop on Human-Al Coevolution, 2025b. URL https://openreviier /
forum?id-HPIvbIJGNY)
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Ré mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, and 7 others. Competition-level code generation with AlphaCode. Science, 378(6624):10922-0009 1097, dec 2022. doi: 10. 1266//iencc..abg 1588. URL http:::/ddoo.oogg/// 1126%2Fscience 1158.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguisties. URL
www.nooooooooooooooooooooooooooo oog.oogg/o.o........ 1013.
Yunlong Lin, ZiXu Lin, Kunjie Lin, Jinbin Bai, Panwang Pan, Chenxin Li, Haoyu Chen, Zhongdao Wang, Xinghao Ding, Wenbo Li, and Shuicheng YAN. JarvisArt: Liberating human artistic creativity via intelligent photo retouching agent. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL /tttpss:/oppervviiii www. mettforum?id- XPL.992270O.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview. wet meet/forumiid=w0Hrrce
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. URL https://openreviee hwtpevwwwww.e..eeeeeeeeeeeeee net/forum?iddBllee forummidd-BBggiiiic""
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, and 47 others. StarCoder 2 and The Stack v2: The next generation, 2024. URL https: //arxiv. org/ abs/2402 19173.
Yuyu Luo, Nan Tang, Guoliang Li, Chengliang Chai, Wenbo Li, and Xuedi Qin. Synthesizing natural language visualization (NL2VIS) benchmarks from NL2SQL benchmarks. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD '21, pp. 1235-1247, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383431. doi: 1445/3448801634552266 URL https://doi http:/doo..oogoo//// 10. 1145/3448816. 3457261.
Jock Mackinlay, Automating the design of graphical presentations of relational information. ACM Trans. Graph 10-1:1,0-141, April 1986. ISSN 0730-0301. doi: 10.1145/22949222951 URL
https://doi. http://doo.oog/1///////// 10. 1145 01445//2949... 22950.
F. Kenton Musgrave, Darwyn Peachey, Ken Perlin, Steven Worley, and David S. Ebert. Texturing and modeling: A procedural approach, Third Edition. Morgan Kaufmann series in computer graphics and geometric modeling. Morgan Kaufmann Publishers Inc., 3rd edition, 2002. ISBN 978-1558608481.
OpenAL, Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 400 others. GPT-40 system card, 2024. URL https: /ttp://aaxiv..v/.a www.ai.wgg..aaa..a../. argg/bbs/2411 21276.
14

---

Published as a conference paper at ICLR 2026
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions Knowledge and Data Engineering. 22(10))1345-1333 2010. doi: 10.1109/TKDE:2099.199
Emilio Parisotto. Abdel rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli, Neuro-symbolic program synthesis. In International Conference on Learning Representations, 2017. URL https://openrett
met//forum?id=r10WWeee.
Gustavo Patow and Xavier Pueyo. A survey of inverse rendering problems. Computer Graphics Forum, 22(4):663-687, 2003. doi: http://doi.org/10.111111 1t0.://0000111111146666002200000
200.00000000000001106000600 URL
https://onninelibrrary wi wwtp/////mb///.../11111
1467 8659. 00003300716..0
Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 3rd edition,
November 2016. ISBN 978-0-12-8088 ---80044--0.
Sagi Polaczek, Yuval Alaluf, Elad Richardson, Yael Vinker, and Daniel Cohen-Or, NeuralSVG: An implicit representation for text-to-vector generation, 2025. URL https://arxiv. orgo /
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763, PMLR, 18-24 Jul 2021. URL https://proceedings www.mprreeeeeeeeee pptppp/eweeee/eeppp.ppppppppp 30d 1awidd22a... html.
Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. StarVectur Generating scalable vector graphics code from images and text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16175-16186 86, June 2025.
Steven F. Roth, John Kolojejchick, Joe Mattis, and Jade Goldstein. Interactive graphic design using automatic presentation knowledge. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '94, pp. 112-117, New York, NY, USA, 1994. Association for Computing Machinery. ISBN 0897916506. doi: 10.1 00.1145/191666.191777 1145/1916666.... 191719.
URL. https:
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, and 6 others. Code LLaMA: Open foundation models for code, 2023.
Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. CSGNet: Neural shape parser for constructive solid geometry. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/ 2018. 8.00888 URL httt::///penaccess. thecvf. / /Saaaa Setppe__v___2220 2018 www.ppppppp.... html.
5515-5523.
Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. Match: Differentiable material graphs for procedural material capture. ACM Transactions on Graphics (TOG), 1-55 2020.
Shantanu Thakoor, Simoni Shah, Ganesh Ramakrishnan, and Amitabha Sanyal. Synthesis of programs from multimodal datasets. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 18/IAAI 18/EAAI 18. AAAI Press, 2018. ISBN 978-1-57735-800--2 URL https://aasa prt.aaaiiorrg/pppp/113333: synthesis- of programs- from- mul timodal -daallltst........
Louis Leon Thurstone. law of comparative judgment. Psychological Review, 34())2273-22 1927.
doi: 10310077/07702888 URL hatpps://dol weooo.orge .rg. 10. 1037 /0093/7/007788...
15

---

Published as a conference paper at ICLR 2026
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3D shape programs. In International Conference on Learning Representations, 2019. URL /otps///e///ereeee..e.oe www.eeeeoorddddyllliloooo
Shengbang Tong, Ellis Brown II, Penghao Wu, Sanghyun Woo, Adithya Jairam Iyer, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
URL https: httpss////ppeeeieeeeee nett
Henrik Voigt, Kai Lawonn, and Sina Zarrieß. Plots made quickly: An efficient approach for generating visualizations from natural language queries. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 12787-12793, Torino, Italia, May 2024. ELRA and ICCL. URL https://acllantholll org/ /224. www.--rrinnnnn.. 1119.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. International Conference on Learning Representations, 2021.
URL https:/
net/ forum?id-rC8s 44 6kaH.
Chao Wen, Jacqueline Staub, and Adish Singla. Program synthesis benchmark for visual programming in XLogoOnline environment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 158 12-15888, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-891 19-8-8899777-1111 76-251-0. doi: 10. 10.18653/v1/2225.eee
long.766 769. URL //aps:://ccll anthology. org /tttss///ccaatthoooyyooo////// acl long..looog..22 769.
Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. IconShop: Text-guided vector icon synthesis with autoregressive transformers. ACM Trans. Graph., 42(6), December 2023. ISSN 0730-0301. doi: 10. 145/36116186444 URL https:///oiii. wocolii.orrg/// 10. 1145/3618364.
Yang Wu, Yao Wan, Hongyu Zhang, Yulei Sui, Wucai Wei, Wei Zhao, Guandong Xu, and Hai Jin. Automated data visualization from natural language via large language models: An exploratory study. Proc. ACM Manug. Data 2(3), May 2024. doi: 10.1145/3654992. URL https://doi. org/10 10. 1145/3654992.
Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, and Ivan Vulić, Visual planning: Let's think only with images, 2025. URL https://arxis: /arxxv.xvvvvaa wwwpxiiv.orggeeo arg/abs/2505. 11409.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. Qwen3 technical report, 2025a. URL ht http:://arxiv.orgge
220.. 09388.
Yiying Yang. Wei Cheng, Sijin Chen, Xianfang Zeng, Fukun Yin, Jiaxu Zhang, Liao Wang, Gang Yu, Xingjun Ma, and Yu-Gang Jiang. OmniSVG: A unified scalable vector graphics generation model, 2025b. URL httpss::/axxvvvvviv /ttp///i....//b///eeeeae// .62663.
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. Optimal neural program synthesis from multimodal specifications. In Aarii-Frannine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1691-1704, Punta Cana, Dominican Republic, November 2021. Association for Computational I Linguistics. doi:
10. 10.18653/v1/2211 wtt22222..innnnseeeeee findinngs-ennlp..... 146. URL. emnlp. 146.
2021. findings-
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 1 1(122)nwac403, 11 2024. ISSN 2095-5138. doi: 10. 10.10///sr//waw44033 URL https:://olt org/ htt.://o...rr......////////aaaaa
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large language models meet NL.2Code: A survey. In Proceedings of the
16

---

Published as a conference paper at ICLR 2026
st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 7443-7464, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1/116655//1/2233.allllorggg
URL https:://cclaattolllooo/ http:/ooooogyoggg///..a---oogggo 411.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang. Aleksei Timofeev, Mingze Xu, Hong-You You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, and 4 others. MM1.5: Methods, analysis & insights from multimodal LLM fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL. https: //openreview ew hett/formm?ide Hd-niiddHHHHAAA....
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong. Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and others. A survey of large language models, 2025. URL https:/ /ttpa/aatrgaaa//////aa/aaaaaaaaa 18223.
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXAMPLES
In Figure 7 we provide additional qualitative examples. MULTIMAT" (Mixed) consistently surpasses VLMATERIAL (SBS), while MULTIMAT' (Graph) demonstrates the strongest results overall. Figure complements Figures 6 & 7 by showcasing failure cases where our models struggle to produce faithful outputs, though notably, the outputs from MULTIMAT" (Graph) and MULTIMA' (Mixed) still demonstrate superior representation of the input compared to VLMATERIAL (SBS). Beyond these conditional generation examples, Figure presents unconditional samples generated by MULTIMaT (Graph), which exhibit high visual quality with realistic material properties. Adjacent to these rendered materials, we visualize their underlying material graphs the same format used as model input. In Figure 10, we show a graph in COMPACTSBS representation to give an impression of the structure of our format.
18

---

Published as a conference paper at ICLR 2026
Input
VLMATERIAL. (SBS)
MUltIMar" (Mixed)
MULTIMAT" (Graph)
Figure 7: Additional qualitative examples for inverse procedural material modeling following the setup of our human evaluation in $6.3. The leftmost column shows input materials from graphs filtered during preprocessing, making these particularly challenging test cases. MULTIMrr' (Mixed) consistently outperforms VLMATERIAL* (SBS), while MULTIMAT (Graph) achieves the best results overall.
19

---

Published as a conference paper at ICLR 2026
Input
VLMATERIAL" (SBS)
MULTIMAT" (Mixed)
MULTIMAr" (Graph)
Figure 8: Representative failure cases from the same challenging subset in Figures 6 & 7. All models struggle to reproduce the intricate patterns in these examples, though MULTIMAT" (Graph) and MMTTIMT (Mixed) still outperform VITMATERAAL* (SBS).
20

---

Published as a conference paper at ICLR 2026
Render
Graph
Figure Example materials generated unconditionally by MULTIMAT (Graph), shown alongside their corresponding procedural graphs
21

---

Published as a conference paper at ICLR 2026
contennt:
oxeaton: 9:23
CRPACESION:
odeer 9.79,
Saction connccioon
aoda: u14
8.33
feeedte:: bleed mn:
becott peaicter
accual Incuaiiyy 9.1
Roda: A29
aoda: a13
pereay:
Elenttrgecde: dec A0E
mradeass: 3OAA
59
calemwitck: felte
Sacttion: recterd
oogaaaaa (4. 4]
comacttion::
perua::
Fieettir::
Fowtl: Darricarallt
connacteoaa
con semectioess compoattiiaa:
paraaa
opeciig:
smssladion (0-10, 8-82, 9-10, 9-4]
Idi cut
Saletion: recsuee
Daecten: cra.t
mode: .3
compoertiio: 27
levelca
Boocdaa: tupkpess grapecala http://apppp.....e
daccies:
Jursa::
Catesion: soest Flonts
Ddlae R.Y, 5-8)
soda:
AG4
Ldi orrjus
compotattr:
connections
dinect: IALE
conauctional
daution: clanci
Neeet: 9.5
ouppasttstur: 21
Salctton recuerd
depeddecy:
Faectie:: ert..ooot
Ioateciity:
comectionns
cmppaaradd:
Seccii:: Flors
get_tett Teeteetiy
Factiia: lovals
sooen: 114
Inpect:
victical cdbet: Crua
1d: orijut
picma:
Amectie:
serven:
taragrm ragge
deyeaderor:
Saction: flones
cionactine
pacma:
14: sutpet
suder
calpot
Flictios Foorts
Gat, Glaic
Sepaddecc::
20:
Fanctio:: aas on
52
posttcon
sutgin
get dtems: beight pesicies
moda:
Sheve::
fieecti:: vactort
marp 41
compue and olke
edituc: (6-is,
Eleadingedee SCRIS
satelada (-2.A, S.R, B.N, 1.1]
speci
(O. <3
cvtractiion
Lomittadg
soda:
coart fleuss:
Sewelaaiow: (0.33, 8-33, 0.30, 9-41
cour Slees: 3.5
fmuction: gr:
622
Figure 10: Complete example of a graph in COMPACTSBS format. This listing shows the full representation of the material partially illustrated in Figure 3.
22


Reviews:
[{"review_id": "eRBxYbGZQW", "reviewer": "Reviewer_nJcx", "summary": "This paper introduces MultiMat, which is a multimodal VLM trained for the accurate, efficient synthesis of 2D procedural material graphs (PMGs). MultiMat is the first procedural material approach that explicitly leverages both the textual and visual graph representations of PMGs to improve generation quality. The textual representation used in MultiMat is a new, simplified YAML-based description for PMGs, which is more readable and considerably (~80%) shorter than the native XML descriptions used by the underlying generation software (here, Adobe Substance Designer). During generation, MultiMat enforces an iterative, topologically-sorted generation procedure that verifies the graph after each node output, to ensure validity and allow efficient backtracking through constrained tree search. MultiMat also contributes a new PMG dataset based on the Substance Designer Assets Repository; this dataset is larger in size and scope than its predecessors.", "questions": "1. Why was it necessary/desirable to flatten hierarchical structures in the professional PMGs, and what (if any) are the ramifications of having done so? Do the substructures conflict with the tree search and/or the YAML representation? I ask because hierarchical subgroups are very useful for human designers, as they merge complex blocks into higher-level concepts whose internal details can be abstracted away once implemented; by the logic posited in this paper, I imagine that this would be similarly useful for the model once the graphs reach sufficient complexity (which is certainly possible within 128 nodes).\n2. Rather than 1 node at a time, would it be possible for each iteration to generate a set of nodes all residing at the same topological sort depth? What would the potential advantages or challenges of such an approach be, relative to a single node?\n3. Were the error patterns of Section 4.3 identified once (or a few times) through manual intervention, then hardcoded into the pipeline? Would there be any room to integrate feedback, such as online identification of failure modes and/or potential workarounds for them? \n4. To my understanding, the iterative generation scheme provides information about the (partial) graph's validity and state visualization/description; however, it does not evaluated or provide feedback on e.g., whether the generated image is likely to lead to the target image under conditional generation. It seems like that would be an interesting extension though; would it be possible to evaluate and/or include any such information to shape the trajectory of the generation procedure? Do there exist any heuristics to judge the suitability of a partial solution for this domain?\n\n-- Minor Comments --\nFig 1. The graph shown here emphasizes the possible complexity of professional PMGs, but does not illustrate the \"intuitive\" nature of these visualizations as you claim in e.g. l.68; it may be worth selecting a different graph here to make your point.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper achieves SOTA performance across a variety of metrics on a challenging and impactful problem in computer graphics. Moreover, the paper achieves this by capitalizing on a remarkably simple observation about a piece of affiliate data that has traditionally been overlooked in PMG generation: the visual graph representation is crucial for human interaction with PMGs, so why shouldn't it be beneficial for VLMs as well? \nOther domains beyond PMGs are likely to benefit from the notions in this work, such as reframing their data or adopting the iterative, verifiable graph generation with backtracking and error correction.\nThe paper is generally well written, including a substantive description of the domain, the related work and several comparisons to relevant baselines.", "weaknesses": "1. The paper does not specify details about the data format or the algorithms used to train the QWen model for MultiMat realization.\n2. The baseline setup in 6.1 is unclear -- is VLMaterial embedded within the current paper's pipeline, as an exchange for the MultiMat block? I assumed that it was evaluated as a standalone VLMaterial model with full-graph generation, albeit trained over a different dataset. I was surprised, then, to read l. 412 describing the degradation of the NER score for VLMaterial(SBS) when tree search was turned off, since I never expected it to be on. A clearer description of the baseline setup would be very helpful.\n3. Did you perform any ablation studies to test the efficacy/impact of your components? For example, does the fragmented node generation procedure affect the model's global reasoning ability over PMGs? This might be observable under the conditional generation task: is MultiMat is able to reproduce the desired target image equally well using iterative node generation vs single-shot graph generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "30mPz2b4M5", "reviewer": "Reviewer_d96n", "summary": "This paper presents MultiMat, a framework for synthesizing procedural material graphs using large multimodal models on both visual and textual data. Arguing that material creation is an inherently visual-spatial task, the authors show that incorporating visual intermediate states improves results over text-only methods. The system is validated on a new dataset from Adobe Substance Designer through extensive experiments, including unconditional and image-guided synthesis, ablations, and comparisons to recent text-only and multimodal baselines.", "questions": "- Could you offer a more formal definition of the multimodal objective function, with equations showing how image and text data are fused during generation? It would be helpful to know if attention is shared and whether the loss is contrastive or generative.\n\n- Since the evaluation is confined to Substance Designer, could a case study on a different visual programming language be provided to substantiate the claims of generality, even if the experiment is on a smaller scale?\n\n- For the retrained baselines like VLMaterial and MatFormer, please clarify which features were omitted or unsupported. How might these omissions impact the fairness of the performance comparisons?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper's central contribution is a multimodal synthesis paradigm that incorporates visual feedback of intermediate states during graph generation. This method addresses a key limitation in text-only approaches by more closely aligning with human creative workflows.\n\n2. A new, comprehensive dataset of production-level procedural materials is collected and used for evaluation. The experiments are rigorous, benchmarking against strong baselines on both unconditional and conditional synthesis tasks and demonstrating superior performance over text-only models across multiple metrics.\n\n3. The work includes systematic ablation studies and an analysis of failure cases, which add credibility by clarifying the model's capabilities and current limitations. The paper also presents practical error recovery strategies, such as incremental tree search and automatic repair.\n\n4. The exposition is clear and well-supported by figures that effectively visualize the model architecture, workflows, and experimental results. The inclusion of concrete output examples, such as visual graphs and code listings, helps to illustrate the system's real-world performance.", "weaknesses": "- The paper's contextualization is limited by its failure to engage with recent, directly relevant literature. The Related Work section omits several key advances in multimodal program synthesis (e.g., [1], [2], [3] ) and recent visual programming benchmarks.\n\n- The methodology lacks mathematical formalism, which hinders reproducibility and theoretical interpretation. Crucial elements like the multimodal training objective, loss functions, negative sampling strategy, and the method for combining image and text embeddings are not precisely specified. Algorithmic descriptions, such as the backtracking process, also lack the clarity of formal pseudocode.\n\n- Missing citations to several directly relevant works: As detailed below, the references omit numerous recent multimodal and vision-language approaches to program synthesis and visual programming benchmarks [4][5].\n\n- The experimental analysis is not sufficiently thorough. The ablation studies are limited and do not provide quantitative evidence for the impact of individual components like automatic error repair. The paper also needs a clearer discussion on the fairness of baseline comparisons, a deeper analysis of the trade-off between efficiency and accuracy, and more comprehensive qualitative comparisons.\nPlease provide some rendered videos.\n\n- Claims of the method's generality are conjectural, as the entire pipeline and dataset are specific to Substance Designer. The paper provides no experimental evidence or case studies to demonstrate the framework's adaptability to other graph-based creative environments.\n\n[1]. Optimal Neural Program Synthesis from Multimodal Specifications\n\n[2].  Synthesis of Programs from Multimodal Datasets\n\n[3]. Program Synthesis Benchmark for Visual Programming\n\n[4]. JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent\n\n[5]. Visual Planning: Let's Think Only with Images", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnH74c9Yy0", "reviewer": "Reviewer_tCBc", "summary": "The paper proposes MultiMat, a multimodal framework for synthesizing procedural material node graphs. Unlike prior text-only methods, MultiMat leverages both textual and visual representations during generation, integrating visual feedback to better mimic human material design workflows. It introduces (1) a multimodal feedback loop using intermediate graph visualizations, (2) an incremental tree search with automatic error repair for valid program generation, and (3) a new dataset of production-quality materials from Adobe Substance Designer.", "questions": "1. How well does MultiMat generalize to unseen node types or non-Designer graph systems?\n2. Could the incremental tree search approach be applied to other program synthesis domains (e.g., 3D modeling or animation)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel multimodal approach combining visual and textual cues for program synthesis.\n2. Strong empirical results, showing clear gains in KID, CLIP, and DreamSim metrics.\n3. Creation of a large, comprehensive dataset supporting full Substance Designer features.", "weaknesses": "1. Training inefficiency: requires per-node multimodal context updates, leading to much longer training times.\n2. Evaluation primarily limited to one domain (Substance Designer), with generalization to other systems unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "juQCrqgn95", "reviewer": "Reviewer_Zyod", "summary": "This paper introduces MultiMat, a multimodal program synthesis framework that generates procedural material node graphs using large multimodal models. Unlike previous text-only approaches, MultiMat incorporates visual feedback by processing both textual representations and visualizations of intermediate graph states. The authors propose two conditioning methods (mixed and graph-based) and implement a constrained tree search algorithm for efficient inference. Experiments on a dataset of 6,878 production-quality Adobe Substance Designer materials demonstrate superior performance over text-only baselines in both unconditional and conditional (inverse rendering) generation tasks.", "questions": "The paper relies entirely on automatic metrics. Given that visual quality is subjective, human evaluation comparing outputs from different methods would be valuable.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1: The key insight of treating procedural material graphs as visual-spatial programs is intuitive and well-motivated, aligning with how human artists interact with these tools.\n\n2: The collection of 6,878 materials with complete feature set support represents a significant improvement over previous datasets.", "weaknesses": "1: The authors acknowledge that MultiMat requires \"several days\" on 8×A100 GPUs compared to \"a few hours\" for text-only baselines. This ~10-20× difference in training time is a significant practical limitation that deserves more discussion about potential optimizations.\n\n2:  All training and testing use the same material engine (Substance Designer). How would the approach transfer to other procedural material systems like Blender's shader nodes?\n\n3: Lack of ablation study. What is the contribution of each component (visual feedback, tree search, error repair)? Ablation experiments would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)