VLMaterial: Procedural Material Generation with Large Vision-Language Models
Authors: Beichen Li
Rundi Wu
Armando Solar-Lezama
Changxi Zheng
Liang Shi
Bernd Bickel
Wojciech Matusik
Venue: ICLR 2025 Spotlight (2025)
Keywords: generative model
procedural material
appearance modeling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Procedural materials, represented as functional node graphs, are ubiquitous in computer graphics for photorealistic material appearance design. They allow users to perform intuitive and precise editing to achieve desired visual appearances. However, creating a procedural material given an input image requires professional knowledge and significant effort. In this work, we leverage the ability to convert procedural materials into standard Python programs and fine-tune a large pre-trained vision-language model (VLM) to generate such programs from input images. To enable effective fine-tuning, we also contribute an open-source procedural material dataset and propose to perform program-level augmentation by prompting another pre-trained large language model (LLM). Through extensive evaluation, we show that our method outperforms previous methods on both synthetic and real-world examples.

Published as a conference paper at ICLR 2025
VLMATERIAL: PROCEDURAL MATERIAL GENERA- TION WITH LARGE VISION-LANGUAGE MODELS
Beichen Li', Rundi Wu², Armando Solar-I.ezama' Changxi Zheng", Liang Shi!, Bernd Bicke!*. Wojeiech Matusik
MIT CSAIL, CColubbia University, 'ETH Zürich, *Googge Research
ABSTRACT
Procedural materials, represented functional node graphs, ubiquitous in computer graphics for photorealistic material appearance design. They allow users to perform intuitive and precise editing achieve desired visual appear- ances. However, creating a procedural material given an input image requires professional knowledge and significant effort. In this work, we leverage the abil- ity to convert procedural materials into standard Python programs and fine-tune a large pre-trained vision-language model (VL.)) generate such programs from input images. To enable effective fine-tuning, we also contribute an open-source procedural material dataset and propose to perform program-leve augmentation prompting another pre-trained large language model (LLM). Through exten- sive evaluation, we show that our method outperforms previous methods on both synthetic and real-world examples.
INTRODUCTION
High-quality materials are essential to creating digital 3D assets in computer graphics applications such video games, movies and AR/VR. Image-based materials use several texture maps repre- sent material attributes (e.g.. albedo, normal, and roughness). While extensively used in the render- ing pipeline, they are not easily editable and limited by the image resolution. As an alternative, procedural materials have become particularly popular in modern 3D) content creation workflows due to an interpretable, controllable, and peplluiionnidependeed material representation. design and content creation tools, e.g., Blender (Blender (024a) and Adobe Substance 3D (Adobe) 2024), a procedural material typically defined as directed computation graph of pre-defined functions. More specifically, the material graph consists of nodes representing texture generators or filtering operators and edges representing the information flow, producing the required material attributes of an appearance shading model (see Figure rigght right for example). Users can easily edit the material by adjusting the parameters of each node or manipulating the edges.
modern 3D>
Complex procedural material graphs can contain dozens of nodes, which requires professional skills and significant effort to create manually. This has motivated the research on inverse procedural material modeling, i.e., automatically generating procedural material match the appearance of a captured or rendered input image. Early works retrieve an artist-created material graph from an
online repository and focus on optimizing its node parameters (Hu al. 2019 Shi al., 2020)
Hu et al., 2022a; Li et al., 2023 Hu et al. 20226). MatFormer (Guerrero et al. 2022) first pro- posed to generate the entire material graph by training a multi-stage transformer based model that
sequentially generates nodes, edges, and parameters. Hu et al. (2023) later extended the model for conditional generation from text or image prompts. Instead of training a similar, custom network from scratch, we explore the fact that procedural material graphs in Blender (Blender] (024aa) can be transpiled into standard Python programs and fine-tune a pre-trained large vision-language model (VLM) to generate such programs directly.
A straightforward approach is to directly prompt pre-trained VLMs to generate procedural material programs from images. However,
find that existing commercial VLMs (e.g., GPT-40-miní (Ope- nAi (024b)) struggle on this task (see Ta as procedural material programs are domain- specifcc and underrepresented in the training data of VL.Ss. Therefore, fine-tune pre-trained VLMM

---

Published as a conference paper at ICLR 2025
Input images
Application of predicted materials
Post Post-editing
Before edit
Right wall
Dranser
After edit
Curtain
Chairs
Given single imput images (left), our model generates corresponding procedural materials (top right) that can be directly applied to 3D scene (middle). The generated materials can be easily edited, e.g., by changing some node parameters, to achieve desired visual appearance (bottom right).
Figure I:
on pairs of images and procedural material programs. The scarcity of publicly available procedu- ral materials, however, poses fundamental challenge. Inspired by FunSearch (Rorera-Parrdes ett al., (2024), we leverage pre-trained VLMs to perform program-level data augmentation followed by carefully designed node parameter augmentation. This data augmentation strategy extends our dataset from ~ 1.6K examples to ~ 550K examples, significantly improving the matching quality of predicted materials.
Through extensive evaluation, we show that our method outperforms previous single-image proce- dural material generation methods on both synthetic and real-world examples. As shown in Figure our generated procedural materials can be directly applied to a 3D scene with full editability. Fur- thermore, previous works are developed for proprietary software, e.g., Adobe Substance 3D (Adobe) 2024]. thus precluded from releasing training data. We contribute the first open-source procedural material dataset in Blender (Blender (0224) to promote future research in this area. Our dataset and code are available at wothttps:/// thub. com mi t gfx./////// /V...te.... a
RELATED WORK
Inverse procedural material modeling. thesizing image- based texture maps (Li et al. 2017 2018 Deschaintre et al. 2018 Gao al. 2019
Most previous work material generation focus syn-
Guo et al. 2020 2021; Henzler et al. 2021 Vecchio et al. 2021|22222 by Zhou & Kalantari 202T
Martin et al. 2022, Zhou et al. 2022 2023 Inverse procedural material modeling. instead, aims to generate procedural material programs from captured images. Early works along this direction assume known procedural material graph and use either learning-based or optimization-ba proaches to find the best matching parameters (Hu et al.) 2019 Shi et al., 2020 Hu al.) 2022a Li et al. 2023 Hu et al. 2022b). For example, Hu et al. (2019) trains a CNN for each material graph to regress the best parameters that match the input image. MATch (Shi et al. 2020) uses gra- dient descent to optimize the parameters by making the graph execution in Substance (Adobel 2024) differentiable, which later extended cover most available material node types for end-to-end differentiability (Hu et al. [222aa] Li et al.) 2023). More recent works learn to predict or refine both the graph structure and parameters of procedural materials (Guerrero af.) 2022, Hu et al., 2023] Huang et al.) 2024).. MatFormer (Guerrero et al., (2022) flattens the material graphs into custom token sequences and trains three Transformers (Vaswani et al., 2017) to generate nodes, edges, and parameters for each node sequentially. Hu et al. (2023) subsequently conditions MatFormer input text descriptions or images. [i et al. (2024a) further applies reinforcement learning (RL) fine-tuning to improve parameter predictions. Instead of training custom network from scratch, we leverage that procedural material graphs in Blender (Blender (2024)) can be transpiled into Python programs and fine-tune VLM for image-conditioned program generation.
ap-
2

---

Published as a conference paper at ICLR 2025
Visual program synthesis. Our work is also related to visual program synthesis, i.e., inferring the underlying programs from input visual signals. Prior works under this topic span many domains, such synthesizing drawing programs from images (Johnson et al. 2017 Ellis et al.] 2018 2019)
Tian et al.
2019
Ellis et al., 2023), converting raster images into SVGs (Reddy et al. [202]] Car-
lier et al.,
2020 Rodriguez al.] (023), reconstructing CSG trees from 3D shapes (Sharma al.,
2018 Du et al.
2018; Kania et al. 2020; Jones et al. 2020 Cy et al. 2022), and inferring CAD
construction programs from 3D shapes (Wu al. 2021| Xu et al. 202 Li et al. 2022 Xu al.,
2023). In contrast to some of these works that define their own domain specific language, we focus on generating procedural material programs that are already widely used in the 3D design industry.
LLMs for procedural modeling. Recent Large Language Models (LLMs) like GPT-4o (OpenAT, 2024a), LLaMA (Touvron al. (023) and Gemini (Team et al.) 2023) have demonstrated impres- sive results in code generation and program synthesis (Romera-Paredee et al. 2024] Olausson et al.)
2023). As procedural modeling system essentially defines domain specific language, some cent works explore procedural modeling using LLMs (Sun et al., 2023 Huang et al.] 2024] Yang et al. 2024 Yamada et al. 2024). 3D-GPT (Sun et al. 2023) uses an LLM as a planner to combine pre-defined procedural generators to create a 3D scene corresponding to the given text description. BlenderAlchemy (Huang et al. 2024) leverages VLMs iteratively refine a procedural modeling program to achieve the design intent of the user. These works all use the zero-shot reasoning ability LLMs to modify existing procedural modeling programs. Instead, we focus on generating proce- dural material programs from scratch based on the input images and find that state-of- sfae-of-the-asd art VLMs fail to perform this task reliably via zero-shot prompting. As a solution, we fine-tune VLM on our collected procedural material dataset with carefully designed data augmentation scheme and show a significant performance improvement.
3 BACKGROUND
Procedural materials are functional programs that produce photorealistic material appearances on objects after rendering, typically visualized as node graphs in industrial rendering software. A procedural material M is mathematically equivalent to directed acyclic computation graph = C=(VE)) Each node v E V represents a parameterized texture generation or filtering opera- for su = [( h) that receives zero or multiple input arguments =i, and generates one or more outputs - Its functionality additionally controlled by a set of heterogeneous node parameters & P2.... that are either continuous (e.g., floats, vectors, colors, and color ramps) discrete (e.g., integers, categories, and Booleans) by nature. Each edge e E È represents the data flow from an output connector of one node to an input connector of another node. Notably, adjacent nodes with several input and output connectors can be linked by multiple edges. A user can easily edit a pro- cedural material in graphical interface by adding or removing nodes, changing edge connections, and tweaking node parameters, to produce a range of appearances.
Among several industry-standarrd procedural material systems in modern 3D software (Adobe 2024 Blender 2024a Epic 2024), we tackle the inverse design problem of Blender procedural materials due well-established open-source community and mature Python API. Blender procedural materials utilize texture nodes to generate primitive noise or geometric patterns (e.g., bricks, checker, Voronoi, and fractal noise), shader nodes to specify surface properties (e.g.. roughness, normals, and metallicity), and converter nodes for color mapping and general mathematical operations. The node graph of an example Blender procedural material predicted by our model is shown in Figure 2 Furthermore, Blender allows users to package node subgraph into a custom macro-node with user- specified inputs and outputs, referred to as node group. Utilizing various node groups, procedural materials in Blender represent sophisticated appearances with compact graph structures.
4 METHOD
Figure 2 illustrates the workflow of our method. Given an input image capturing the material ap- pearance flat surface, we use fine-tuned large VLM to predict Blender procedural material in Python code. Executing the code in Blender yields the shader node graph of the material, which is rendered into the output image under point light source. The procedural materials predicted by our model preserve full editability albeit under the constraint of pre-defined node and parameter types from Blender (Blender. 2024b) and users. Following previous works (Shi al. 2020
Hu et al.)
3

---

Published as a conference paper at ICLR 2025
VLM
Execute
Render
Input
Python code
Shader node graph
Output
Mitiraal odt Trm
Figure 2: Workflow overview. Given input image, our fine-tuned VLM to predict procedural material in the form of Python code. Executing the code Blender yields user-editable shader node graph, whose rendered appearance closely matches the input image.
2023
Li et al., 2.23). we aim to achieve a close perceptual match with the input image instead of pixel -perfect reproduction.
Fine-tuning a large VLM requires a comprehensive procedural material dataset with diverse mate- rial appearances and graph structures. Prior work (Hu et al. (203) has developed such dataset for Adobe Substance 3D (Adobe 2024) materials but their dataset can not be released due to proprietary intellectual property. To address the lack of an open-source counterpart, we specially curate a dataset containing over 550K Blender procedural materials (Section 4.1 and 4.2). The remaining subsec- tions describe our fine-tuning process (Section 4.3) and post-optimization algorithm (Section 4.4)
4.1 DATA COLLECTION
We first collected 3,663 free Blender materials from three online sources: 1) 2,411 materials in BlenderKi an online repository of Blender 3D assets; 2) the 60 base materials of Infini- gen (Raistrick et al., 2023), procedural 3D scene generation framework; 3) 1,192 materials from individually published Blender procedural material packe] These materials are manually created by experienced technical artists and cover a wide variety of material categories. Then, we conduct the following curation process before adding each material to training data:
Node graph clean-up. We run depth-first search (DFS) from the material output node to identify nodes that participate in output appearance calculation, removing other redundant nodes and edges. The material is dropped if valid output node does not exist or the node graph is too large (with more than 30 nodes). addition, each node group the material is treated as a custom node type distinguished by its functional signature. To prevent the VLM from struggling to memorize too many custom node types, we iteratively expand the smallest node groups in these materials into corresponding subgraphs until reaching the graph size limit.
Program validation. We subsequntlyy transpile the remaining materials into Python code and ren- der the resulting appearances. This step filters out materials with excessively long code sequences (more than 2,048 tokens after tokenization) degenerate renderings, including empty images, uni- form colors, or textures lacking meaningful spatial features. We heuristically measure the complex- ity of material using the compressed JPEG file size of its rendered image. By sorting the materials with descending JPEG file sizes, we observe that most materials below 12KB exhibit undesirable texture quality. We thus apply this heuristic threshold to discard unwanted, meaningless materials from the dataset.
4.2 DATA AUGMENTATION
The scarcity of high-quality artist-created procedural materials has been a fundamental challenge for training generative models (Hu al.) 2023). Blender materials are no exception- tte curation process above leaves 1,640 materials in our dataset, which we found severely inadequate for VLM fine-tuning. Hu al. (2023) proposes to create appearance variations of existing materials by ran- domly perturbing node parameters. While this effectively introduces color and pattern changes,
https:/ http:///eypittteee otures. www.geddennoooom www//oeddooo/onddooo //ubhi httpss//nerrs77772.ggooom /oaehod
4

---

Published as a conference paper at ICLR 2025
Algorithm MCMC-based local parameter search.
1: i O, M < M, 1 (0,1+((111)))
- 2: M + 1.1++1
Initialize the result
3: while i < Nuurs do
4:
Randomly sample M's node parameters to obtain M
D See Appendix A.3 for details
5: I' + L[T, R(M"))
6: if < then
7:
M M', 1+
D Accept better solution
8:
else
9:
Set M
M', I + l' small probability Pace
Otherwise, accept stochastically
10:
end if
11:
if I < 1r then
12:
M <- M, ++
Update the best solution
13:
end
14: i+1
15:
end while
the sampled variants retain the same node graph structure, which hardly prevents the model from overfitting to a limited set of graph structures.
Our data augmentation strategy combines the augmentation of graph structures and parameters. On the graph structure level, we augment the 1.6K artist-created materials with diverse program struc- tures using an approach similar to the evolutionary workflow in FunSearch (Romera-Paredes al., 2024). Specifically, we manually select a subset of 870 complex materials with rich texture de- tail from the artist-created materials into sample pool. The evolution process involves selecting random program pairs from the sample pool and performing "genetic crossovers" using commer- cial pre-trained L.M (GPT-40-mini), which is prompted to generate novel programs using elements from the selected programs. The LLM can restructure the example programs and modify their node parameters as nceded. We then validate the LLM-generated programs and add executable programs to the sample pool, where each program pair contributes up to four valid programs out of 20 tri- als. Our evolutionary workflow produces 50.4K programs with distinct structures, more than 30x the number of artist-created materials. We provide the complete LLM prompt in Appendix A..I After graph structure augmentation, we conduct parameter-space augmentation following Hu et al. (2023). creating another 10x more program variations from random parameter perturbations (see Appendix A. for implementation details). The augmented dataset eventually comprises over 550K materials, more than 330x the initial size.
4.3
VLM FINE- TUNING
Our augmented procedural material dataset enables the adaptation of pre-trained VLMs to image- conditioned procedural material generation tasks. We employ the latest version of VA-NeXT LLi
et al. 2024b) from the LLaVA model family (Liu et al., 2023b 2024), series of open-source molti-modal LLMs trained on public-domain visual question answering (VQA) data. The model architecture consists of CLIP ViT/L-14 vision encoder (Radford et al. 2021). LLaMA 3 8B language decoder (Dubey al. (2024), and an MLP projector that bridges embedding spaces of the encoder and decoder networks. Additional training-related details are available in Appendix A.2
4.4
POST-OPTIMIZATION
After predicting a plausible procedural material with the VLM, we can further improve the percep- tual match against the input image by refining its node parameters. This post-optimization step is commonly used in previous works concerning Adobe Substance 3D materials (Shi et al., 2020 Hu
al. 2022a 2023 Li et al. 2023). In particular, they leverage differentiable implementation of the node graph to backpropagate analytical gradients from a perceptual loss based on the Gram ma- trices of VGG features (Gatys et al. 2015). However, Blender procedural materials currently do not have differentiable counterparts, nor is it feasible to train individual differentiable proxies (Hu et al. 2022a) for hundreds of built-in and custom node types.
5

---

Published as a conference paper at ICLR 2025
Table I: A quantitative comparison of method with several baselines in-distribution (Blender) and oitoof-distribution (Substance and real images) datascts. For a fair comparison, the numbers for our approach here before post optimization. Red: best score, Orange: second best score.
Program correctness f
Dataset
Method
Style loss +
SWD &
CLIP
GPT-4o-mini
0.041
3,478
0.728
0.294
Cond. MatFormer
0.034
2.828
0.686
Blender
BlendcrAlchemy
0.027
2.381
0.777
Nearest Neighbor
0.026
2.123
0.778
Ours
0.019
1.760
0.856
0.911
GPT--4omini
0,039
3.722
0.680
0.227
Cond. MatFormer
0,033
2.366
0.736
Substance
BlenderAlchemy
0.029
2.727
0.690
Nearest Neighbor
0.027
2.546
0.720
Ours
0.026
2.283
0.762
0.890
GPT-4o-mini
0.035
4.252
0.642
0.248
Cond. MatFormer
0.028
2.872
0.684
Real
BlendcrAlchemy
0.025
2.682
0.700
images
Nearest Neighbor
0.021
2.487
0.700
Ours
0.025
2,417
0.722
0.870
Instead, we opt for a gradient-free, stochastic local parameter search based on the Markov Chain Monte Carlo (MCMC) algorithm (Geyer 1992 Diaconis 2009). which has been applied to inductive program synthesis (Schkufza et al. 2013 Let (() be predicted procedural material with node parameters & that yields an output rendering image
R(M). L(I, be the perceptual loss function between the input image and the prediction J. Our MCMC algorithm translates into the pseudocode in Algorithm|] [ We run MCMC sampling for each predicted material for Nusr 200 iterations, accepting a worse sample in each iteration a small probability Pace 0.05.
EXPERIMENT
EVALUATION SETUP
Datasets. We evaluate method in-distribution and out-of-distribution test images collected from three different sources: 1) 44 synthetic materials from Blender (Blender 2024a), randomly selected and separated from our training dataset (before augmentation); 2) 64 synthetic materials from Substance (Adobe 2024), which are randomly sampled from the evaluation set of (Hu et al.,
3) 64 real photographs gathered from Shi al. (2020) and Zhou et al. (2023), captured using smart phone cameras. We note that the node graph system of Adobe Substance 3D is more expressive than Blender and capable of creating more intricate material appearances. Therefore, although the Substance test set comprises synthetic images, we consider them an out-of-distribution test set harder than Blender.
2023;
Baselines. We compare our method four baselines: GPT-4o-mini (OpenAl 2024b), nearest neighbor retrieval, Conditional MatFormer (Hu et al.) 2023) and BlenderAlchemy (Huang et al., 2024). For GPT-4o-mini (OpenAl 2024b), we directly query the model with the input image and a carefully designed prompt (see Appendix A.4). For nearest neighbor retrieval, we retrieve the nearest material from our original training dataset based on the style loss (Gatys) 2015) between its rendered image and the input image. Conditional MatFormer (Hu et al.) 2023) trains custom transformer on 446K procedural material programs (after parameter augmentation) from Substance Source (Aooee, 2024h. Since its code is not publicly available, we asked the authors to their trained model on our test sets. BlenderAlchemy (Huang et al. 2024) starts with an initial procedural material program and iteratively queries VLM to modify the program to better match the input image. We run their method starting from the nearest neighbor retrieval result and process VLM queries with GPT-4o-mini. We compare all methods without performing post-optimization.

---

Published as a conference paper at ICLR 2025
Conditional MatFormer
Blender Alchemy
Nearest Neighbor
Input
Ours
GPT-4o-mini
Blender
Substance
Real images
Figure 3: A qualitative comparison of our method with several baselines. The input images are shown in the leftmost column. Re-rendered images from our generated procedural material pro- grams best match the input.
Evaluation protocol. Similar to Hu et al. 2023), model and stop once obtaining
generate most N programs using N valid programs. We then pick the program with the lowest
style loss (Gatys 2015) between the input and rendered images the final result. Following Sec- tion 4.1, a generated program is considered valid it successfully executes and produces an output rendering above the 12KB JPEG file size threshold. For fair comparison, we evaluate GPT-4o- mini and Conditional MatFormer using the same protocol. We use N 50 and K 20 for the following quantitative and qualitative results. Additional comparison results under tighter sample budget (N = 20, K =
4) are included in Appendix B
Evaluation metric. To measure the perceptual similarity between the rendered image of gener- ated program and the input image, we use three metrics-style loss (Gatys 2015), sliced Wasserstein distance (SWD) (Bonneel et al. 2015), and CLIP cosine similarity (Radford al.) 202 lb. The style loss widely adopted metric for texture distance. Following prior work (Hu et al. 2023), we compute the style loss using the LI difference of Gram matrices of VGG features (Gatys et al. 2015) plus the LI difference of 16 x 16 downsampled images (weighted by 0.1). SWD calculates the distance between images based the optimal transport theory (Villani al. 2009). which

---

Published as a conference paper at ICLR 2025
Table 2:
Ablation study. parameter aug. removes the graph parameter augmentation, and struc- ture aug. further removes the graph structure augmentation. + post opr. optimizes graph parameters our generated programs using MCMC.
Program correctness f
Dataset
Method
Style loss
SWD
CLIP
structure aug. parameter aug.
0.032
2.757
0.750
0.603
0.022
1.965
0.839
0.897
Blender
Ours + post opt.
0.019
1.760
0.856
0.911
0.015
1.701
0.851
structure aug.
0.033
3.084
0.727
0.487
parameter aug.
0.030
2.482
0.744
0.767
Substance
Ours + post opt.
0.026
2.283
0.762
0.890
0.020
2.184
0.748
structure aug.
0.030
3.308
0.688
0.342
Real images
parameter aug.
0.028
2.682
0.707
0.713
Ours
0.025
2.417
0.722
0.870
post opt.
0.018
2.309
0.732
has been demonstrated for neural texture synthesis (Heitz et al. 2021). In addition, the CLIP metric measures the semantic difference textures as complement.
Aside from appearance matching, we also evaluate the model's ability to produce valid procedural material programs. Inheriting the evaluation protocol above, assuming we run the model n I N times and get & K valid programs for test image, define correctness the fraction of valid program predictions. This metric only applies to our method and GPT-4o-mini, since Conditional MatFormer represents procedural materials customized token sequences and implements an inference-time grammar checker to ensure the generated program is executable.
as k/n, i.e.,
5.2 RESULTS
We report quantitative comparison results in Table and show qualitative examples in Figure and Figure 5 GPT-4o-mini (OpenAl 2024b) struggies to produce valid material programs. Condi- tional MatFormer (Hu et al. 2023) performs reasonably well the Substance dataset (on which was trained) but shows limited generalization to the other test sets. Nearest neighbor retrieval ob- tains competitive quantitative scores, which agrees with the findings in previous work (Hu al., 2023). However, most retrieved materials are visually sub-optimal against the input. Blender- Alchemy (Huang al. 2024) often produces results that do not deviate much from the starting programs. Moreover, it incurs considerable running cost since each program generation requires dozens of multi-modal queries to the GPT model.
Our method outperforms the baselines on all three test datasets. The generated appearances are structurally and semantically close the input images. To demonstrate the practical usability of our generated materials, apply them to 3D meshes and show the rendering results in Figure We also conduct a user study among professional artists and domain researchers to validate how generated materials substitute artist's creation process (see Appendix for details).
5.3 ABLATION STUDY
In Table 2 and Figure 4 we compare our method against ablated versions without data augmenta- tion and validate the post-optimization algorithm. Both graph structure augmentation and parameter augmentation are crucial to improving the matching quality on in-distribution and outtof-distribution test sets. Specifically, structure augmentation guides the model to predict more accurate texture pat- terns, while parameter augmentation enhances the matching of color and other appearance details. As for program correctness, model has significantly higher probability of generating valid programs after fine-tuning using the fully augmented dataset. In addition, our results are further improved by the post-optimization step using MCMC local parameter search. For generated pro-
8

---

Published as a conference paper at ICLR 2025
Ours + +post opt.
Ours
Ours structure aug.
Input
Ours
parameter aug.
Blender
Substance
Real images
Figure 4: Visual examples for ablation study.
grams with sufficient capacity represent the input image, the post-optimization algorithm refines the overall color and spatial structures, achieving noticeably better alignment with the input.
DISCUSSION AND CONCLUSION
Figure demonstrates few failure cases of our method where the intricate texture pattern of the input image exceeds the expressiveness of existing Blender node types (including artist-created node groups) within a reasonable node graph size (e.g., we use limit of 30 nodes). In such cases, our model may struggle to predict the best material graph structure to approximate the input texture patterns. Extending the node graph system with custom node groups from a broader range of artist- created materials will alleviate the limitation in expressiveness. An alternative is to increase the graph size limit and the VLM's context window, although this requires additional engineering efforts in data augmentation generate more feature-rich materials using LLM.
Future works may address the challenge of generating complex visual programs using VLMs from several orthogonal directions. Including more artist-created materials and performing data augmen- tation using next-genenn next-generation LLMs with advanced tfathuht (CoT) reasoning may improve the complexity and diversity of training data. Further, it is possible represent Blender procedu- ral materials with a compact domain-specifn language (DSL) based on handcrafted rules or library learning. This will produce considerably shorter transpiled programs than Python and simplify VLM fine-tuning. Moreover, an inference-time syntax checker can be implemented for the DSL to mask invalid tokens and guarantee the correctness of generated programs. Finally, incorporating visual feedback through RL fine-tuning (Li et al., 2024a) is an effective approach to enhance visual qual- ity. Employing an image-space reward function bridges the gap between token accuracy and visual similarity, potentially resulting in better generalization to unseen images.
In conclusion, we present a single-image procedural material generation method that fine-tunes pre-trained large VLM to predict Blender procedural materials in Python code format. Our ap- proach harnesses the embedded knowledge the VLM through domain specialization, enabled by the first open-source procedural material dataset with more than 550K image-program pairs. We report superior perceptual matching quality on distrii-tttiiitoonnnn synthetic and real input images, outperforming existing models trained proprietary materials or leveraging zero-shot VLM infer- ence. We hope the release of our dataset and model will benefit future research on visual program synthesis and inverse procedural graphics.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partially funded by an unrestricted gift from Google.
REFERENCES
Adobe. Adobe Substance 3D Designer, U.L.ttpss//ww..addee....oopppppp
substance3d.
Blender. Blender a 3D modelling and rendering package. Blender Foundation, 2024a. URL
http:/ www. blender.e.r.o .rg
Blender. Blender Python API. Blender Foundation, 2024b. URL https://docs.blencce
hTg/api/curres
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51:22-45, 2015.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. Deepsvg: A hierarchical generative network for vector graphics animation. Advances in Neural Information Processing Systems, 33:16351-16366 2020.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869// 2023.
Valentin Deschaintre, Mïika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. Single- image svbrdf capture with a rendering-aver aware deep network. ACM Transactions on Graphics (ToG), 37(4):1-15, 2018.
Bulletin of the Ameri- Persi Diaconis. The markov chain monte carlo revolution. can Mathematical Society, 46:179-205, 2009. URL
50273- 09773-9999-88-1128--8
Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojciech Matusik. Inversecsg: Automatic conversion of 3d models to csg trees. ACM Transactions on Graphics (TOG), 37(6):1-16, 2018.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2407:2240, 2024.
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graph- ics programs from hand-drawn images. Advances in neural information processing systems, 31, 2018.
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar- Write, execute, assess: Program synthesis with repl. Advances in Neural Information Processing Systems, 32, 2019.
Kevin Ellis, Lionel Wong. Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua Tenenbaum. Dreamcoder: growing generalizable, interpretable knowledge with wake-slecp bayesian program learning. Philosophical Transactions of the Royal Society A, (81(2251)002200500 2023.
Epic. Unreal Engine. Epic Games, 2024.
Duan Gao, Xiao Li, Yue Dong, Picter Peers, Kun Xu, and Xin Tong. Deep inverse rendering for high-resolution svbrdf estimation from arbitrary number of images. ACM Trans. Graph., 38 14):134-1, 2019.
Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Advances in neural information processing systems, 28, 2015.
Leon A Gatys. A neural algorithm of artistic style. arXiv preprint arXiv: 508.06576. 2015.
10

---

Published as a conference paper at ICLR 2025
Charles J Geyer. Practical markov chain monte carlo. Statistical science, pp. 473 473-483, 1992.
Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomír Méch, Tamy Boubekeur, and Niloy J Mi- Matformer: generative model for procedural materials. arXiv preprint arXiv:2207.0104 2022.
Jie Guo, Shuichang Lai, Chengzhi Tao, Yuelong Cai, Lei Wang, Yanwen Guo, and Ling-Qi Yan. Highlight-aware two-stream network for single-image svbrdf acquisition. 2021.
Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. Materialgan: Re- flectance capture using generative svbrdf model. arXiv preprint arXiv:2010.00114, 2020.
Eric Heitz, Kenneth Vanhoey, Thomas Chambon, and Laurent Belcour. A sliced wasserstein loss for neural texture synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9412-9420, 2021.
Philipp Henzler, Valentin Deschaintre, Niloy Mitra, and Tobias Ritschel. Generative modelling of brdf textures from flash images. arXiv preprint arXiv:2102.11861, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688 09680. 2021.
Yiwei Hu, Julie Dorsey, and Holly Rushmeier. A novel framework for inverse procedural texture modeling. ACM Transactions on Graphics (ToG), 38(6):1-14, 2019.
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. Node graph optimization using differentiable proxies. In ACM SIGGRAPH 2022 conference proceedings, pp- 1-9, 2022a.
Yiwei Hu, Chengan He, Valentin Deschaintre, Julie Dorsey, and Holly Rushmeier. An inverse procedural modeling pipeline for svbrdf maps. ACM Transactions on Graphics (TOG), 41(2): 1-17, 2022b.
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. Generating procedural materials from text or image prompts. In ACM SIGGRAPH 2023 Conference Pro- ceedings, pp. 1-11, 2023.
Ian Huang, Guandao Yang, and Leonidas Guibas. Blenderalchemy: Editing 3d graphics with vision- language models. arXiv preprint arXiv:2404.17672, 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reason- ing. In Proceedings of the IEEE international conference on computer vision, pp. 2989-2998, 2017.
R Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. Shapeassembly: Learning to generate programs for 3d shape structure synthesis. ACM Transactions on Graphics (TOG), 39(6): 2020.
Kacper Kania, Maciej Zieba, and Tomasz Kajdanowicz. Desg-net-unsupere discovering of con- structive solid geometry tree. Advances in neural information processing systems, 33:8776-8788, 2020.
Beichen Li, Liang Shi, and Wojciech Matusik. End-to-end procedural material capture with proxy- free mixed-integer optimization. ACM Transactions on Graphics (TOG), 42(4):1-15, 2023.
Beichen Li, Yiwei Hu, Paul Guerrero, Milos Hasan, Liang Shi, Valentin Deschaintre, and Wojciech Matusik. Procedural material generation with reinforcement learning, ACM Trans. Graph., 43(6), November 2024a. ISSN 0730-0301. doi: 10.1145/3687979 URL https:://///..... //doi. org.or////. 10.
1145/3687979
11

---

Published as a conference paper at ICLR 2025
Bo Li, Kaichen Zhang. Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger 1lms supercharge multimodal ca- pabilities in the wild, May 2024b. URL https://llava-w. githubb Bacichuu.ioo/eee ic/blog/
2024-05-20 10-100111 aalaavva-in next--eett---ee -teett-sttee ronger- 1ms/
Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J Mitra. Free2cad: Parsing freehand drawings into cad commands. ACM Transactions on Graphics (TOG), 41(4) 2022.
Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Modeling surface appearance from single pho- tograph using self-augmented convolutional neural networks. ACM Transactions on Graphics (loG), 36(4):1-11, 2017.
Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. Materials for masses: Svbrdf acquisi- tion with single mobile phone image. In Proceedings of the European conference on computer vision (ECCV), pp. 72-87, 2018.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
1lava v1. wtw.://wwwoo/oo///2/00/000000000 01-00 llaval next//
I Loshehilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,
Rosalie Martin, Arthur Roullier, Romain Rouffet, Adrien Kaiser, and Tamy Boubekeur. Materia: Single image high-resolution material capture in the wild. In Computer Graphics Forum, vol- ume 41, pp. 163-177. Wiley Online Library, 2022.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang. Jianfeng Gao, and Armando Solar- Lezama. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations, 2023.
OpenAl. GPT-4o, 2024a. URL https:
OpenAl.
GPT-4o-mini, 2024b.
URL
gpt 40 mini cedvancing--ostteeeeeee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021.
Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, et al. Infinite photorealistic worlds using pro- cedural generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12630-12641, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
pp.
Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J Mitra. Im2vec: Synthesizing vector graphics without vector supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7342-7351, 2021.
12

---

Published as a conference paper at ICLR 2025
Juan A Rodriguez, Shubham Agarwal, Issam H Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, and Marco Pedersoli. Starvector: Generating scalable vector graphics code from images. arXiv preprint arXiv:2312.11556 2023.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468-475 2024.
Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superoptimization. ACM SIGARCH Computer Architecture News, 41(1):305-316 2013.
Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. Csgnet: Neural shape parser for constructive solid geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5515-5523, 2018.
Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. Match: Differentiable material graphs for procedural material capture. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould. 3d-gpt: Procedural 3d modeling with large language models. arXiv preprint arXiv:2310. 12945, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11800 2023.
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenen- baum, and Jiajun Wu. Learning infer and execute 3d shape programs. arXiv preprint arXiv:1901.0288 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv: /397/, 2023
Mikaela Angelina Uy. Yen- Yu Chang, Minhyuk Sung, Purvi Goel, Joseph G Lambourne, Tolga Birdal, and Leonidas Guibas. Point2cyl: Reverse engineering 3d objects from point clouds to extrusion cylinders. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11850-11860, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL Talee//eeee2SSeeeeeee100003o3o03
occpss//e files/paper/2017/
Giuseppe Vecchio, Simone Palazzo, and Concetto Spampinato. Surfacenet: Adversarial svbrdf estimation from a single image. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12840-12844 2021.
Giuseppe Vecchio, Rosalie Martin, Arthur Roullier, Adrien Kaiser, Romain Rouffet, Valentin De- schaintre, and Tamy Boubekeur. Controlmat: A controlled generative approach to material cap- ture. ACM Transactions on Graphics, 43(5):1-17, 2024a.
Giuseppe Vecchio, Renato Sortino, Simone Palazzo, and Concetto Spampinato. Matfuse: control- lable material generation with diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4429-4438. 2024b.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer- aided design models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6772 6782, 2021.
13

---

Published as a conference paper at ICLR 2025
Xiang Xu, Pradeep Kumar Jayaraman, Joseph G Lambourne, Karl DD Willis, and Yasutaka Fu- rukawa. Hierarchical neural coding for controllable cad model generation. arXiv preprint arXiv:2307.00149, 2023.
Xianghao Xu, Wenzhe Peng, Chin-Yi Cheng, Karl DD Willis, and Daniel Ritchie. Inferring cad modeling sequences using zone graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6062-6070, 2021.
Yutaro Yamada, Khyathi Chandu, Yuchen Lin, Jack Hessel, Ilker Yildirim, and Yejin Choi. L3go: Language agents with chain-of-3d-thoughh for generating unconventional objects. arXiv preprint arXiv:2402.09055. 2024.
Yue Yang. Fan-Yur Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 16227-16237, 2024.
Xilong Zhou and Nima Khademi Kalantari. Adversarial mngle-image svbrdf estimation with hybrid training. In Computer Graphics Forum, volume 40, pp. 315-325. Wiley Online Library, 2021.
Xilong Zhou, Milos Hasan, Valentin Deschaintre, Paul Guerrero, Kalyan Sunkavalli, and Nima Khademi Kalantari. Tilegen: Tileable, controllable material generation and capture. In SIGGRAPH Asia 2022 conference papers, pp. 1-9, 2022.
Xilong Zhou, Milos Hasan, Valentin Deschaintre, Paul Guerrero, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Nima Khademi Kalantari. Photomat: A material generator learned from single flash photos. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 DATA AUGMENTATION
Graph structure augmentation. We prompt GPT-4o-mini generate novel programs using the template below based on pairs of example programs.
You are familiar with creating procedural materials using Blender's a Python APT. You will be given Python code block delimited by triple backticks, which contain two functions that define the node trees of two procedural materials in and respectively. Your task is to write Python function "shader_materie material based on the provided functions. Write your code following the guidelines below.
Blender. The functions are named "shader_materian "shader_material_
that creates a new
Code template: 'pyyhhn
import bpy
def shader_material (material: bpy. Moy.types.Material material suriall.use_oddes True
nodes naterial.node.. nodes
links = material.node_tt wew.oddeerreeeineeee
# Create nodes YOUR CODE HERE
Create links to connect nodes # YOUR CODE HERE
# Set parameters for each node
# YOUR CODE HERE
Rules:
1. Create no more than 30 nodes. Only use node types and parameter fields referenced in the provided functions. Change parameter values as needed.
2. Make sure your code can be correctly executed in Blender 3.3. Refer to the Blender Python API documentation for valid node types and parameters.
3. Try to generate materials with complex and semantically meaningful appearances. Avoid generating materials that lack structure or look too similar to the provided examples.
4. Follow the format and coding style of the example functions.
Do not add new code blocks or comments.
5. Simply reply with code. Exclude any additional text explanations.
Parameter augmentation. Given a procedural material with a prescribed graph structure, we ran- domly sample its node parameters around their initial values to generate appearance variations. The collection of sampled parameters includes 1) unoccupied input and output connectors of each node with well-defined default values, e.g., the color input to principled BSDF shader node; 2) node- specific attributes, such as the operation type math node and the interpolation option of a color ramp node. Although sampling strategy similar to Hu al. (003), we develop native im-
15

---

Published as a conference paper at ICLR 2025
plementation for Blender's procedural material system, which applies different sampling algorithms contingent on whether a node parameter is inherently continuous or discrete.
For continuous parameter, i..,, float, vector, color, color ramp, we refer to its defi- nition in the associated node and its value statistics across the procedural material dataset determine feasible range. The parameter is then uniformly sampled within the inter- section the feasible range and +25% interval around the initial value. Vector or array elements sampled independently from another. Considering that the relative inter- val may be insufficient for small initial values, we enforce +:0.5 as an absolute, minimal sampling interval. Specifically, color parameters sampled in the sneesaaurration-aalie (HSV) color space instead of the conventional RGB space. We uniformly sample hue chan- nels from the entire [0, 1] range and sample the other channels within +0.5 interval. This encourages diverse color tones in sampled materials.
We divide discrete parameters into two separate scenarios. An integer parameter is treated as continuous and rounded to the nearest integer after uniform sampling. Categorical and Boolean parameters, in contrast, do not have the notion of absolute values intervals. We thus uniformly sample each parameter from its value range at a probability of 0.25.
An important caveat of sampling all node parameters at once is the additional program length. In Blender Python programs, explicit assignments are unnecessary for node parameters remaining at default values. Sampling all node parameters away from their default values will accumulate dozens of lines of code, creating excessively long code sequences. We prevent this issue by sampling node parameters holding default values with probability of 0.2. If the resulting program exceeds the token length limit (i.e., 2,048 LLaMA tokens), we randomly reset some parameters back to default values reduce the code length. The resulting program must pass the validation (Section 4.1 before being incorporated into training data.
A.2 VLM FINE-TUNING
For supervised fine-tuning, we pair the input image with the following user prompt to the VLM and tokenize the ground-truth Python program using LLaMA 3's tokenizer.
<image> Write a Python function with Blender API to create material node graph for this image.
The trainable modules include the MLP projector and an array of LoRA adapters (Hu et al. 2021 applied to all attention-related linear layers in the LLaMA model (with 8, a 32, and 0.05 dropout probability), amounting to 40M trainable parameters in total.
We use an AdamW optimizer (Loshchilov] 2017) with a le-4 learning rate and a cosine annealing schedule. The initial linear warmup period accounts for 3% of training steps. To effectively utilize GPU memory, we apply FlashAttennion-22 (Dao 2023) and train the model in BF16 precision on 8x NVIDIA H100 80GB GPUs using DeepSpeed ZeRO-3 (Rasley al. 2020; Rajbhandan et al. 2020). Each GPU accommodates a batch size of training samples, resulting in an overall batch size of 32. The fine-tuning process lasts 5 epochs three days.
A.3 POST-OPTIMIZATION
We largely follow the same node parameter sampling method as data augmentation (Appendix A. for MCMC sampling. However, we only randomly sample 10% available node parameters each iteration. Furthermore, +220% (including colors) and the sampling probability of categorical parameters to 0.2. These prac- tices prompt the MCMC algorithm to make smaller steps in each iteration, improving the likelihood of converging to a better solution. We also ignore the program length limit during post-optimization. Therefore, the node parameters
constrain the relative uniform sampling interval for ranged parameters to
longer differentiated based on default values.
16

---

Published as a conference paper at ICLR 2025
A.4 EVALUATION
We apply the same user prompt as training to our fine-tuned VLM for inference. We use the follow- ing prompt evaluate the GPT-4o-mini baseline.
You are familiar with creating procedural materials using Blender's Python API. You will be given image that describes a material appearance. Your task is to write Python function 'shader_materre procedural material to match the appearance of the image when rendered on flat surface. Write your code following the guidelines below.
that creates Blender
Code template:
'python import bpy
def mhader_material (material: soy.types.Material
material. use_noess True
nodes material.node wwweetteeeeeeeeeeeeeeeeeeeeeeeee nodes
links = nnterial.node. www.oooeee_eeeeeeeee roterral
# Create nodes
# YOUR CODE HERE
# Create links to connect. nodes
# YOUR CODE HERE
# Set parameters for each node
# YOUR CODE HERE
Rules:
1. Create no more than 30 nodes.
2. Make sure your code can correctly executed in Blender 3.3. Refer to the Blender Python API documentation for valid node types and parameters.
3. Simply reply with code. Exclude any additional text or explanations.
B ADDITIONAL RESULTS AND DISCUSSIONS
Qualitative comparisons. We provide additional qualitative examples and ablation study results in Figure 5 and Figure 6, respectively. Figure 7 shows some failure cases, a detailed discussion around which is included in Section
Quantative comparison in limited budget. We also compare method and baselines quanti- tatively using a more limited sample budget during inference in Table where the prediction result selected from up to K 4 valid samples out of N 20 trials. Our method is affected by the variance in token predictions, which encourages exploration during inference. Fine-tuning the VLM with RL using an image similarity reward will likely reduce the entropy of token predictions and improve generalization in budget-limited scenarios. Nevertheless, RL fine-tuning should carefully balance exploration and exploitation prevent convergence local minima.
Inference time consumption. We report the average inference time of our method and other base- line methods in Table Specifically, for our method and Conditional MatFormer, the time consump- tion is measured an NVIDIA H100 80GB GPU at the best throughput. The remaining baselines
17

---

Published as a conference paper at ICLR 2025
Conditional MatFormer
Blender Alchemy
Nearest Neighbor
Input
Ours
GPT-4o-mini
Blender (Row 1-3)
Substance (Row 4-6)
Real images (Row 7-9)
Figure 5: More qualitative comparisons with baselines.
18

---

Published as a conference paper at ICLR 2025
Ours + post opt.
Ours
Ours structure aug.
Input
Ours
parameter aug.
Blender
Real images
Figure 6: More examples for ablation study.
Input
Ours
Input
Ours
Input
Ours
Figure 7: Failure cases. The first two examples (Column 1-4) come from the Substance test set. The last example (Column 5-6) is from the real image test set.
timed on AMD Ryzen 5800X 8-core CPU with parallel requests to the OpenAl API. Con- ditional MatFormer has a high inference throughput due to a small network size. However, even after selecting the top-5 graphs from 150 random samples, their predictions still require costly post-optimization to match the input (Hu et al. 2023).
Performance impact of material complexity. We plot the average style loss and program cor- rectness for VL.M-geeeattdd materials with varied complexity in Figure The materials are divided
19

---

Published as a conference paper at ICLR 2025
Table 3: Comparison between our method and baselines under constrained sample budget
(NV 20, K = 4). The results of BlenderAlchemy and Nearest Neighbor are copied from Table
Program correctness
Dataset
Method
Style loss 4
SWD J
CLIP
GPT-4o-mini
0.044
3,618
0.716
0.343
Cond. MatFormer
0.042
3.019
0.682
Blender
BlenderAlchemy
0.027
2.381
0.777
Nearest Neighbor
0.026
2.123
0.778
Ours
0.025
1.978
0.833
0.926
GPT-4o-mini
0.042
3.799
0.678
0.234
Cond. MatFormer
0.040
2.281
0.742
Substance
BlenderAlchent
0.029
2.727
0.690
Nearest Neighbor
0.027
2.546
0.720
Ours
0.031
2,431
0.750
0.926
GPT-44-mini
0.040
4.369
0.636
0.241
Cond. MatFormer
0.037
2.869
0.690
Real
BlenderAlcheme
0.025
2.682
0.700
images
Nearest Neighbor
0.021
2.487
0.700
Ours
0.030
2.566
0.716
0.880
Table 4: The inference time consumption our method and other baselines.
Avg. time per sample (s)
Method
Avg. time (s)
#Samples
GPT-4o-mini
7.5
0.38
Cond. MatFormer
4.1
20
0.21
BlenderAlchemy
252.4
7.89
Ours
47.9
20
2.40
0.06
Program correchness
Programm corecinese
0.8
0.8
0.04
0.04
0.6
0.8
0.4
Style loss
0.4
0.01
0.2
0.01
0.2
Program length (tokens)
Number of nodes
Figure 8: The average style loss and program correctness of VLM-generated materials with varied material complexity. The complexity is measured using program lengths in tokens and node graph sizes, respectively.
into separate bins based on the transpiled program length and the number of nodes. Most materials with moderate complexity show comparable style losses and correctness. Interestingly, both metrics are worse for simple materials with fewer than 4 nodes, largely due to invalid suboptimal choices of custom node groups. Program correctness also declines for materials with more than 22 nodes or 2,000 tokens, implying higher chance of errors in more intricate materials. As mentioned in Sec- tion 6 implementing inference- time syntax checking to mask out invalid tokens can guarantee the correctness of generated programs.
20

---

Published as a conference paper at ICLR 2025
Table 5: Guidelines for interpreting the usability ratings of algorithm-genenan procedural materials in our user study, Intermediate ratings indicate preferences between two adjacent descriptions.
Rating
Description
Barely useful. It is better to create the material manually. Partially reusable as template. Many sections need to be reworked.
4
A good starting point, requiring careful but not too cumbersome post-editing. An ideal solution. Very few edits are necessary.
10
Table 6: User study on the practical usability of procedural materials generated with our fine-tuned VLM and BlenderAlchemy (Huang et al. 2024). We report the average 2AFC preference score and usability rating over different test sets. Numbers after the "=" symbol are standard deviations.
Preference
Blender
Substance
Real images
Average
(Ours over BlenderAlchent
Visual quality
92% 92%
95%
86% 98%
91% 94%
Node graph quality
91%
Usability Rating (1-10)
BlenderAlchemy
4.1 + 1.7
3,9 + 1.9
4.5 + 1.5
4.2 + 1.7
Ours
7.9 + 1.2
5.6 1.7
6.9
1.0
6.8
1.3
C
USER STUDY
To investigate the practical value of VLM-generated procedural materials to an artist's creation pro- cess, hosted user study among professional artists who specialize in Blender material graphs and researchers with sufficient domain expertise. The participants are tasked with comparing the materials generated from our method and BlenderAlchemy (Huang et al.) 2024) for 12 randomly selected test images (4 images from each test set). Each comparison involves the following items:
Preference in visual quality. The participant chooses which method ("A" "B") pro- duces better visual match with the input image. We adopt the two-alternative forced choice (2AFC) approach and randomly shuffle the results to conceal the source algorithms.
Preference in node graph quality. In the same 2AFC setting, we show the participant a snapshot of the generated material node graph from each method and ask them determine which graph requires simpler user post-editing to match the input image.
Usability rating. The participant rates the usability each generated material on a scale from 1 to 10 based on how much it substitutes artist's creation process. The interpreta- tions of usability ratings are provided in T
We collected 16 responses from 8 professional artists and 8 researchers, respectively. Then, we calculated the average preference percentage for our method over BlenderAlchemy and the aver- age usability rating for both methods in Table 6 In more than 90% of the test cases, participants prefer our generated materials over BlenderAlchemy achieves average usability rating of 6.8, indicating that artists generally consider VI. VI.M-generated materials efficient starting points for post-editing. addition, some artists praised our method for the interesting results and noted how our method alleviates the core issues with complex shader graphs compared with image-based textures. They also appreciated the better reusability of simpler graph structures in production environments.
visual and node graph quality. Our method
21


Reviews:
[{"review_id": "63uLo0HGub", "reviewer": "Reviewer_fAs1", "summary": "This paper proposes a framework for finetuning a VLM for generating procedural materials that matches an input image. A new procedural material dataset collected and curated from various sources, and then augmented with an LLM based approach. Output generated from the finetuned VLM also undergoes a MCMC-based gradient free optimization step to bring it closer to the input image. The results are impressive both qualitative and quantitatively, and with better generalzation capacities.", "questions": "I am overall positive on this one. Despite the rather limited novelty and lack of deeper exploration, there is still solid data contribution, a good amount of insights, and solid results. I won't champion this paper in its current state, but more \"insights\" (see weakness section above) will most likely move me towards a more positive stance on this paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Personally, I always enjoy seeing papers that show \"foundation models work on X\". Even if there is minimal contribution architecture and algorithm wise, it still provide me with insight about what kind of tasks can benefit from large models. This paper adds even more evidence about the viability of applying LLM/VLMs to procedural generation (and graphics in general), and I find this message important.\n- Beyond the rather straightforward application of LLM + finetuning, considerable works as also being done here for creating the finetuning data. There are valuable insights on how data is cleaned and augmented, and the LLM based approach for creating new samples from two examples seem generally applicable for anything that involve visual programs. The dataset itself can also be immensely useful, especially when the alternative is not publicly/easily accessible.\n- Solid insights on how to work with procedural material graphs that a not differentiable. The MCMC based approach is reasonable, appears to be working from the ablations and examples. It is also nice (and a bit nolstalgic) to add some exposure to a classic set of optimization methods and show that they are very viable in certain cases.\n- The results look good. I do have a good amount of issues with some results but overall, it does appear to be that there are cases where the proposed method works clearly better than alternatives. Ablation is also solid and effective.", "weaknesses": "- Also I do appreciate the message that \"VLM works for procedural materials\", arguably the novelty is limited. This is one of the lower hanging fruits, and a good amount of work is engineering-centric, making the impact of this work potentially limited. It probably also makes this paper less suitable for a venue like ICLR, since there isn't too much contribution learning-wise. I would definitely love to see this at a graphics venue a bit more.\n- Following the previous point - even for paper that's mostly \"apply VLM to X\", the amount of insights in the paper is still on the relatively low end. Most of the discussion is around the data. They are important, and I do consider the data portion a strength. However, I feel that there's missed opporunities here in further invesgitating how to best apply VLMs to this problem, especially since the supervision is entirely token level, without visual feedback (comparing the generated material to the input). E.g. there are many known limitations with VLM/LLMs that make them not perfectly suitable for directly outputing complex visual programs with lots of parameters that are non-trivial to interpret on a purely language level. How to design something that alleviate such issues? Does fine tuning take care of most of it or do we need to more carefully design/modify the language and the prompts? What part of the output do the model struggle the most with? Is there a more intuitive explanation of why failure cases like those in Figure 7 happen, and what part of the output contribute the most to the discrepancy between the output material and the image (e.g. it does appear that many BSDFs are quite off?)? Discussions along these lines would be very helpful both for people who want to use this approach, or for future researchers that might build upon this.\n- My standard on quality of results is definitely higher on this one, due to the rather limited amount of technical insights. And while the results look good overall, they are still quite far from matching the input image, even among the few qualitative examples provided and after the post-processing step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r6M8oaBgw2", "reviewer": "Reviewer_mxJd", "summary": "VLMATERIAL generates a procedural node based material pipeline from a given input image. The authors create a dataset (will be open sourced) of pairs of material images and their corresponding graphs by doing clever data augmentation for paramters and node structure. The material graphs are also obtained as Python programs. A VLM is fine tuned on this data to generate a python program to create a material graph from an input image. The method is evaluated on in-distribution (Blender) as well as out-of-distribution data(Adobe substance + real images) and an ablation is performed.", "questions": "It would be interesting to see how the complexity of the node setup (number of shader nodes, edges, etc) correlate with the quality of the results and program correctness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is well-written and motivated. All things are explained clearly. \n* The work aims to address a significant issue of directly generating a material graph from an input image as this is highly desriable in CG pipelines. Compared to existing work such as MatFormer and Conditional MatFormer, the results are superior. This is also proven quantitaively in Table 1 against baselines.\n* The method creates a large datasset by performing augmentation using an LLM which picks 2 sample programs and then creates a distinct program. \n* The VLM in the loop to generate a dataset and also synthesize programs after fine tuning is simple but novel in material generation. \n* The ablation results are strong. A test for program correctness is also included.", "weaknesses": "* The method is 90% accurate in terms of program correctness compared to other methods which have guaranteed correctness. Have the authors explored methods to reduce LLM hallucination to improve correctness?\n* It would good to have a reference for the compution time for different methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wFNXqL6M54", "reviewer": "Reviewer_dtAr", "summary": "This paper tackle the problem of procedural material generation with LLM. The authors first generate Python code conditioned on the input image with LLM, then execute the code in Blender, which can match the input image exactly. This method also facilitates the downstream editing and manual creation.", "questions": "+ The paper addresses the challenge of limited material data (~2k) in fine-tuning LLMs through data augmentation. The first question is: what specific techniques were employed in the data augmentation process? Could you summarize these techniques and provide a clearer description?\n\n+ Additionally, the reviewer believes it is quite meaningful to apply LLMs (VLMs) to a broader range of applications, particularly in domains with limited data. Could you provide more evidence regarding the **training/val process** about role of data augmentation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes a straightforward pipeline for procedural material generation. It trains a domain-specific Vision-Language Model (VLM) through meticulous data collection, processing, and fine-tuning, followed by effective post-processing techniques to address the problem.\n+ Both data augmentation and MCMC-based post-processing are validated through qualitative and quantitative results. \n+ Using VLM to tackle graphics problems is a promising and intriguing area for exploration, with potential applications across a wide range of domains. \n+ The results appear robust, surpassing all baselines, including MatFormer and LLM-based approaches, in both quantitative and qualitative evaluations.", "weaknesses": "+ Adapting VLM as a tool for material code generation may not be entirely reasonable, as LLaVa primarily addresses natural images rather than focusing on code generation. It is important for the network design to account for these biases.\n\n+ Additionally, could you provide a detailed user study for artist? Is it possible for this AI tool to substitute certain steps in the artistic creation process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpW6UaXx2q", "reviewer": "Reviewer_bStb", "summary": "The paper presents a novel approach to procedural material generation from input images using a large, fine-tuned vision-language model (VLM). This method transforms material generation into a text-to-program problem, converting material graphs to Python code for Blender's API. The paper introduces an open-source procedural material dataset and proposes a two-tier data augmentation strategy to enhance training, achieving a substantial improvement in program accuracy and visual quality over existing methods.\n\nSeveral contributions:\n1. VLM Fine-Tuning for Procedural Materials: The authors fine-tune a VLM to generate procedural material node graphs in Python code format based on input images, addressing the limitation that existing VLMs lack training data specific to procedural materials.\n\n2. Open-Source Dataset for Procedural Materials: The authors compile an open-source dataset of 550,000 procedural material samples for training and evaluation, combining real artist-created Blender materials with augmented data.\n\n3. Post-Optimization Algorithm: A gradient-free, Markov Chain Monte Carlo (MCMC) approach refines the generated material node graphs to better match the visual appearance of the input image.", "questions": "Question 1: In Figure 7, it seems the model has difficulty reproducing certain intricate textures. Are there particular features (e.g., high-frequency details, irregular patterns) that consistently pose challenges? Could you expand on why these features are difficult to capture with your model? It may help to provide an analysis of failure cases or challenging texture types, as well as insights into potential model improvements or data augmentation strategies to better handle intricate textures. This could guide future work and inspire researchers to address these limitations.\n\nQuestion 2: In Table 3, your model underperforms on out-of-distribution datasets (Substance and real images) with a limited sample budget. Could you elaborate on how the model’s architecture or training process might contribute to this limitation? It might be helpful to explore or discuss possible adjustments, such as adaptive fine-tuning or feature-specific augmentations, to improve generalization on out-of-distribution data under constrained conditions. Additionally, explaining the trade-offs involved in this limitation and suggesting potential remedies would provide a more comprehensive understanding of the model’s practical applicability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Recasting procedural material generation as a vision-language task, effectively combining visual perception with the generation of structured, editable material programs.\n2. Fine-tuning a VLM specifically for procedural material generation, a domain that was previously not widely explored in the vision-language field.\n3. Introducing a dataset of 550,000 procedural material samples, which includes not only real-world data but also creatively synthesized samples generated via program-level and parameter-level augmentations. This contribution provides a foundational dataset for further research in this area.\n4. Evaluation on both synthetic and real-world images shows that VLMaterial outperforms baselines like BlenderAlchemy and Conditional MatFormer in metrics such as style loss, SWD, and program correctness, demonstrating improvements in visual fidelity and program accuracy.", "weaknesses": "1. Limited Novelty in Augmentation Techniques: While the paper presents a large dataset with program-level and parameter-level augmentation, the augmentation techniques themselves rely on GPT-based models and parameter variations, which may not fully capture the variety found in real-world material designs. \n2. Despite its strong performance, the model struggles with highly intricate textures, where certain details are either lost or inaccurately represented, as shown in Figure 7. Furthermore, in Table 3, this method underperforms compared to other approaches on out-of-distribution datasets (Substance and real images) when operating under a more constrained sample budget. This limitation may affect its applicability in scenarios that demand high precision.\n3. The paper’s evaluation relies heavily on quantitative metrics and in-distribution and out-of-distribution tests. However, given the subjective and artistic nature of material design, the paper would benefit from user studies or feedback from professional material artists. Expert insights could help assess aspects like the usability, editability, and practical value of the generated materials in real production workflows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)