Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Authors: Junlin Han
Jianyuan Wang
Andrea Vedaldi
Philip Torr
Filippos Kokkinos
Venue: ICML 2025 poster (2025)
Keywords: 3D Generation
3D Reconstruction
Large 3D Models
Primary area: applications->computer_vision
TL;DR: A two-stage pipeline for generating high-quality 3D assets in a feed-forward manner.

Abstract: Generating high-quality 3D content from text, single images, or sparse view images remains a challenging task with broad applications.
Existing methods typically employ multi-view diffusion models to synthesize multi-view images, followed by a feed-forward process for 3D reconstruction. However, these approaches are often constrained by a small and fixed number of input views, limiting their ability to capture diverse viewpoints and, even worse, leading to suboptimal generation results if the synthesized views are of poor quality.
To address these limitations, we propose Flex3D, a novel two-stage framework capable of leveraging an arbitrary number of high-quality input views. The first stage consists of a candidate view generation and curation pipeline. In the second stage, the curated views are fed into a Flexible Reconstruction Model (FlexRM), built upon a transformer architecture that can effectively process an arbitrary number of inputs. Through extensive exploration of design and training strategies, we optimize FlexRM to achieve superior performance in both reconstruction and generation tasks. Our results demonstrate that Flex3D achieves state-of-the-art performance, with a user study winning rate of over 92% in 3D generation tasks when compared to several of the latest feed-forward 3D generative models.

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Junlin Han Jianyuan Wang" 12 Andrea Vedaldi Philip Torr? Filippos Kokkinos | Abstract
vision and graphics (Mildenhall et al., 2021; Kerbl et al., 2023) and deep learning (Dosovitskiy, 2020; Caron et al., 2021; Oquab et al., 2023), combined with the availability of large datasets of 3D objects (Deitke et al., 2023; 2024; Yu et al., 2023; Chang et al., 2015), have made it possible to learn neural networks that can generate 3D objects from text, single images sparse set of views, and to do so in an feed-forware manner, achieving significantly faster speeds than distillation- based methods (Poole et al., 2022; Qiu et al., 2024; Chen et al., 2023; Wang et al., 2023b).
Generating high-quality 3D content from text, sin- gle images, or sparse view images remains a chal- lenging task with broad applications. Existing methods typically employ multi-view diffusion models to synthesize multi-view images, followed by feed-forward process for 3D reconstruction. However, these approaches are often constrained by small and fixed number of input views, limit- ing their ability to capture diverse viewpoints and, even worse, leading to suboptimal generation re- sults if the synthesized views are of poor quality. To address these limitations, we propose Flex3D, novel two-stage framework capable of lever- aging an arbitrary number of high-quality input views. The first stage consists of a candidate view generation and curation pipeline. In the second stage, the curated views are fed into a Flexible Re- construction Model (FlexRM), built upon trans- former architecture that effectively process an arbitrary number of inputs. Through extensive ex- ploration of design and training strategies, we op- timize FlexRM to achieve superior performance in both reconstruction and generation tasks. Our results demonstrate that Flex3D achieves state-of- the-art performance, with user study winning rate of over 92% in 3D generation tasks when compared several of the latest forwar 3D generative models. See project page for more immersive 3D results.
A particularly successful family of 3D generators are the ones based on sparse-views reconstruction (Xu et al., 2024c; Siddiqui et al., 2024; Li et al., 2024b; Zhang et al., 2024c; Tang et al., 2024a; Xu et al., 2024b; Xie al., 2024a; Wang et al., 2024c; Hunyuan3D, 2024). Compared to single-image reconstructors, multi-view reconstruction models generally produce better 3D assets. This advantage arises because the multi-view images implicitly capture the object geometry much better, substantially simplifying the reconstruction problem. However, to generate 3D object from text or single image, one must first synthesize several views of the objects, for example by means of a multi-view diffusion model. These multi-view diffusion models often generate inaccurate and inconsistent views, which are difficult for the reconstruction network to reconcile, and can thus affect the overall quality of the final 3D output (Tang et al., 2024c).
This paper thus focuses on the problem of generating a high- quality set of different views of an object, with the goal of improving the quality of the final 3D object reconstruction. We build on a simple observation: the quality of the 3D reconstruction improves as the quality and quantity of the input views increases (Han et al., 2024b). Hence, instead of relying on a fixed, limited set of views generated by poten- tially unreliable multi-view diffusion models, we suggest to generate pool of candidate views and then automatically select the best ones to use for reconstruction.
1. Introduction
Fast generation of high-quality 3D contents is becoming increasingly important for video games development (Hao et al., 2021; Sun al., 2023), augmented, virtual, and mixed reality (Li et al., 2022), robotics (Nasiriany et al., 2024) and many other applications. Recent advances in computer
Based on this idea, we introduce a new framework, Flex3D, comprising a new multi-view generation strategy as well as a new flexible feed-forward reconstruction model.
'GenAL, Meta University Oxford. Correspondence to: Jun- lin Han <junlinhan @ mec..comm>
First, we propose mechanism to generate a large and di- verse set of views. We do this by training two diffusion models, one that generates novel views different azimuth angles and the other at different elevation angles. The mod-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
a beantiful
a wedge of cheese: silver platter
4x
Input
Rendered novel views
Gaussians
Figure 1. Results produced by Flex3D. It generates high-quality 3D Gaussians from single image, textual prompt. and performs 3D reconstruction from arbitrary number of input views.
els are designed make the views as consistent as possible. Second, we propose a view selection process that gen- eration quality classifier and feature matching network to measure the consistency of the different views. The result of this selection is good number of high-quality views, which help to improve the quality the final 3D reconstruction.
the tri-plane features into the parameters of 3D Gaussians used to represent the object. We also simplify the process of learning this MLP, thus leading notable performance improvements, by pre-training parts of it, where we ini- tialize the color and opacity parts using an off-the-shelf NeRF (Mildenhall et al., 2021) MLP. For the remaining Gaussian parameters, we learn rotation and scale in con- ventional manner while learning position offsets which are combined with the tri-plane feature sampling locations.
Differently from many prior works, then, we need recon- struct the 3D object from a variable number of views which depends what the selection process returns. Hence, we require a reconstruction model that (1) can ingest varying numbers of input views and different viewing angles; (2) is memory and speed efficiency to handle a large number of input views; and (3) can output a full, high-quality 3D reconstruction the object, regardless the number and pose of input views.
While view selection pipeline identifies the best views for reconstruction, it still does not eliminate all milti-view inconsistencies. To mitigate the impact of the minor in- accuracies that remain, FlexRM employs a novel training strategy. Although our training dataset consists of perfectly rendered images, we simulate imperfections in the input views by leveraging the output of FlexRM itself. Specifi- cally, take FlexRM's reconstructed 3D Gaussians, add noise to them, and generate new noisy views of the object based on these noisy Gaussians, Compared to directly ma- nipulating the views, this approach allows us to inject more expressive and representative types of noise, as shown in fig. 3. The noisy views are then combined with clean ren- dered views and fed as input to the 3D reconstructor, with the goal of producing clean, noise-free free representations of
To this end, introduce Flexible Reconstruction Model (FlexRM). FlexRM starts from the established Instant3D architecture (Li et al., 2024b) and adds stronger camera conditioning mechanism to address the first requirement (1). also introduces simple but effective way of combining Instant3D tri-plane representation with 3D Gaussian Splatting, meeting requirements (2) and (3). Specifically, FlexRM learns Multi-Layer Perceptron (MLP) to decode
2

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
the 3D object. This approach enables the model to learn how to handle imperfect inputs.
feed-forward models offer significant advantages in both reconstruction quality and inference speed.
In summary, this paper makes the following key contribu- tions: (1) We introduce novel pipeline that generates a pool of 2D views of an object and only selects the optimal subset for 3D reconstruction. (2) We propose FlexRM, a 3D reconstruction network that efficiently processes an arbitrary number of input views with varying viewpoints, enabling high-quality feed-forward reconstruction from the selected views. (3) We introduce a novel training strategy to enhance the robustness of the 3D reconstructor by simulating im- perfect input views. This improves FlexRM's resilience to small noise in the input data that may remain.
A representative series of work is LRM (Hong et al., 2024), which learns to generate tri-plane NeRF (Chan et al., 2022) representation using a transformer network. This approach can receive multiple types of inputs, including single im- ages, text (Xu et al., 2024d), posed sparse-view images (Li et al., 2024b), and unposed sparse-view images (Wang et al., 2024a). Further works (Wei et al., 2024; Xu et al., 2024b; Wang et al., 2024c; Boss et al., 2024) focused on improving the geometric quality of generated 3D assets. Some (Zou et al., 2023) proposed to combine the tri-plane representa- tion with 3D Gaussian Splatting for more efficient rendering. They suggest using an additional point cloud network to de- termine the 3D Gaussian position overcome the tendency of 3D Gaussian to get stuck in local optima.
2. Related work
2.1. Multi-view Generation
Another representative series of work (Tang et al., 2024a; Xu et al., 2024c; Zhang et al., 2024c) generates 3D Gaussian points directly through per-pixel aligned Plücker ray embed- ding and predicts the depth for each pixel (Szymanowicz et al., 2023a), which can then be converted to 3D Gaus- sian locations. However, such approaches requires the input views to cover large visible range of the 3D object. Build- ing an intermediate 3D feature representation to regress 3D Gaussian points is also possible (Chen et al., 2024a; Zhang et al., 2024b), but these methods still require sparse-view images as input and typically prefer a fixed number of views with fixed viewing angles. However, in 3D generation tasks where sparse input views are generated through multi-view diffusion model and are not always of high quality, such reconstructors tend to produce suboptimal results.
Generating novel views from single image text with- out learning a 3D representation is a highly ill-posed and challenging task. With the development of image and video diffusion models, this task has become easier to address, as solutions can be built upon these pre-trained models.
Zerol23 (Liu et al., 2023a) first proposed using multi-view data to fine-tune image diffusion model for generating novel views from a single view, conditioned on camera parameters. Following this approach, subsequent works (Li et al., 2024b; Shi et al., 2023b; Tang et al., 2023; Liu et al., 2023b; Long et al., 2024; Wang & Shi, 2023; Woo et al., 2024; Yang et al., 2024b; Ye et al., 2024; Zhao et al., 2024; Tang et al., 2024b) largely focused on generating multiple views simultaneously to ensure 3D consistency.
This paper introduces FlexRM that combines the strengths of the approaches above. Our tri-plane-based based model effi- ciently generates high-quality 3D Gaussian points directly, without needing additional modules. It also accommodates variable number of input views.
With the availability of powerful video diffusion models, recent works (Kwak et al., 2023; Voleti et al., 2024; Melas- Kyriazi et al., 2024; Li et al., 2024a; Han et al., 2024a; Gao et al., 2024; Zuo et al., 2024; Yang et al., 2024a) have adopted them to improve multi-view generation. However, none of these models can reliably generate a large number of perfectly consistent views. Furthermore, even with camera conditioning, models like SV3D (Voleti et al., 2024) perform poorly when the specified elevation angle deviates from zero. This justify our approach of selecting the best views from a pool of generated views.
2.3. Mitigating multi-view inconsistency with feedback.
Methods such as Ouroboros3D (Wen et al., 2024), Carve3D (Xie et al., 2024b), Cycle3D (Tang et al., 2024c), and IM-3D (Melas-Kyriazi et al., 2024) stem from similar motivations to our work, and they share a key idea: the feed- back mechanism. While useful. these methods often require new supervision signals and learnable parameters to imple- ment this feedback, potentially creating complex, mono- lithic pipelines that are difficult to decompose into reusable components for future designs. In contrast, Flex3D's 's com- ponents more easily generalized. Another key difference is Flex3D's 's focus on the input to the reconstructor. This adds negligible computational cost and avoids the significant additional time required for multi-step refinement, preserv- ing the speed advantage often associated with feed-forward
2.2. Feed-forward 3D Reconstruction and Generation
Recent advances in 3D reconstruction and generation have focused on training feed-forward models that directly out- put 3D representations without requiring further optimiza- tion (Yu et al., 2021; Erkoç et al., 2023; Szymanowicz et al., 2023b; Ren al., 2023; Lorraine et al., 2023; Xu et al.,, 2024a; Tochilkin et al., 2024; Zhang et al., 2024a; Jiang et al., 2024; Han et al., 2024a; Siddiqui et al., 2024). These
3

---

Flex3D: Feed-Forwate 3D Generation with Flexible Reconstruction Model and Input View Curation
Candidate view generation and selection
Flexible reconstruction model (FlexRM)
Hli-plane Decoder
Candidate View
View Selection
Image Enccoder
3DGS Decoder
Generation
Inject
Input
Gaussian points
Arbilrary number selected views
& Tokens
Tri-plan
Candidate
Figure Flex3D comprises two stages: (I) candidate view generation and selection, and (2) recoustruction using FlexRM. In first stage, an input image or textual prompt drives the generation of diverse set of candidate views through fine-tuned multi-view and video diffusion models. These views are subsequently filtered based on quality and consistency using a view selection mechanism. The second stage leverages the selected high-quality views, feeding them to FlexRM which reconstruct the 3D object using tri-plane representation decoded into 3D Gaussians.
models. Additionally, the feedback mechanism is orthogo- nal to our work and could be further combined.
view at 6 elevation, which usually results in representative views for the subsequent reconstruction process.
View selection. As the two multi-view diffusion models are focused on different aspects (elevation and azimuth) of novel view generation, there are minimal conflicts in the generated views between them. Even so, and despite ef- forts to improve the quality of the outputs, not all generated views are entirely consistent. Certain views, particularly those from challenging angles like the back, may exhibit suboptimal generation quality, and there can be inconsis- tencies between different generated views. Including such flawed views as input for 3D reconstruction can significantly degrade the quality. Therefore, we introduce a mechanism to filtering poor-quality views. This is done via novel view selection pipeline which consists of two steps:
3. Method
We illustrate our method in fig. 2. We begin by presenting our approach for generating a pool of candidate views and the subsequent selection process in section 3.1. We then describe the design of the FlexRM in section 3.2. Finally, we outline our training strategy that simulates imperfect input views in section 3.3.
3.1. Candidate View Generation and Selection
Here we describe how pool of candidate views is generated from a single image or text and then filtered for quality and consistency before performing 3D reconstruction.
(1) Back View Quality Assessment: We employ multi-view video quality classifier trained to assess the overall quality of generated videos, with particular emphasis on the quality of the back view. This classifier utilized DINO (Oquab et al., 2023) to extract features from the front view and back view of the multi-view video, and subsequently trained a Support Vector Machine (SVM) to classify video quality based on the combined DINO features. The training data consisted of 2000 manually labeled "good" and "bad" tenu-generated video samples. We apply the quality classifier to the multi- view video to determine whether the back view exhibits reasonable generation quality.
varying elevations. Our generator module gener- ates four views of the 3D object from four elevation degrees (-18%,6 6%, 18 and 30%) We utilize the Emu model (Dai et al., 2023), which is pre-trained on massive dataset of billions of text-image pairs, as our base model. Following prior works (Shi et al., 2023b; Li et al., 2024b; Siddiqui et al., 2024), we fine-tune this model on approximately 130,000 rendered multi-view images. This fine-tuning process en- ables the model to predict 2x2 grid of four consistent images, each corresponding one elevation angle.
Multi-view generation wttti/eee-o---uuui-iii-----eeee-
Multi-view generation at varying azimuths. After gen- erating four views at varying elevations, we employ a fine- tuned Emu video model (Girdhar et al., 2023; Melas-Kyriazz et al., 2024; Han et al., 2024a) to generate a video with 16 views spanning full 360 azimuth range. This model is fine-tuned dataset encompassing wide spectrum of elevation angles, enabling it to generate consistent, high- quality views from diverse inputs with varying elevations. We generate the multi-view videos starting from the input
(2) Multi-View Consistency Verification: If the back view quality is deemed acceptable, designate both the back and front views as initial query views. The front view typ- ically possesses the highest visual quality, as it is directly based on the input provided to the fine-tuned EMU video diffusion model. Conversely, if the back view quality is inadequate, only the front view serves as the initial query view. We utilize the Efficient LoFTR (Wang al., 2024b) to match features between all 20 generated views and the
4

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
selected query views. Views with matching point counts exceeding the mean minus 60% of the standard deviation are added to the selected results. This step effectively gath- ers high-quality side views and views at different poses that demonstrate strong consistency with the initial query views.
cause 3DGS is notoriously sensitive the initial Gaussian parameters, we carefully initialize both the MLP predictor and the tri-plane transformer network with an off-the-shelf tri tlanp NeRF network.
A tri tlanp is a compact representation of a volumetric func- tion [-1, 1|s + E mapping points p E [-1, 1] to feature vectors f()) E R Starting from an initial position Po- the model first reads off the corresponding tri-plane feature f(u), and then feeds the latter into MLP to predict the parameters of a corresponding 3D Gaussian, namely, its position, color, opacity, rotation, and scaling. To obtain a mixture of such Gaussians, we simply start from a set of initial positions Pu and apply the MLP at each location. We use a 100 x 100 x 100 grid to sample the initial positions, resulting in the prediction of I million Gaussians.
Candidate view generation typically takes about a minute on a single H100 GPU, and the whole process of view selection can be done in less than a second with a single A100 GPU.
3.2. Flexible Reconstruction Model (FlexRM)
As outlined in the introduction, FlexRM aims to fulfill the following requirements: (1) adaptability to varying numbers of input views and their corresponding viewing angles, (2) memory and speed efficiency, and (3) the ability to infer full 3D reconstruction of the object independently of how many and which views are available. We follow a minimalist design philosophy and strive to minimize modifications to Instant3D, enabling easy reuse of weights from architectures like Instant3D, and simplifying implementation.
The position of the 3D Gaussian is expressed as p opo (1 a'ppp = opo + (1 a) MLP(J(Po)) where óp is a a positional offset output by the MLP and a = 3/4. These positional offsets óp are constrained to the range of [-1, 1] through the application of the tanh activation func- tion. This approach, akin to residual learning, facilitates the optimization process. The multipliers ensure that p remains within the same range as Po, which prevents Gaussian points from moving beyond the visible boundaries, which would result in their projection falling outside the 2D image plane and consequently providing no gradients for optimization. Furthermore, since a < 1, this expression biases Gaussians to shift towards the center of the tri tri-plane grid, where the object is usually located.
Stronger camera conditioning. Handling varying num- bers of input views with viewing angles necessitates pro- viding camera information to the network. In Instant3D, each view's s camera information, including its extrinsic and intrinsic parameters, is represented as a 20-dimensional vector. This vector is then passed through a camera embed- der to produce usually a 1024-dimensional camera feature, which is subsequently injected into the DINO (Oquab et al., 2023) image encoder network (responsible for extracting image features for each view) using an AdaIN (Huang & Belongie. 2017) block. The image encoder's final output comprises a set of pose-aware image tokens, which has 1024 768-dimensional tokens for every 512x512 resolution input view. These per-view tokens are concatenated to form the feature descriptors for input views.
The color and opacity of the Gaussian are output by the same MLP that, in Instant3D, outputs the color and opacity of their NeRF representation. These two parameters need no conversion as color and opacity in NeRF and 3DGS are similar in functionality. Finally, the part of the MLP that predicts the Gaussian rotation and scaling parameters are learned from scratch.
Our aim is to ensure that the DINO-extracted tokens are not only camera- aware, but also explicitly incorporate leamn able camera information into the final feature descriptors. To achieve this, we set the output dimension of the cam- era embedder to 768, enabling it to match the dimension of the pose-aware image tokens and be appended to them, resulting in 1025 (1024 image tokens 1 camera token) 768- dimensional tokens overall. This simple way of attaching explicit camera information enhances the network's 's cam- era awareness, strengthening its performance in complex scenarios where a large number of input views is provided.
Data. Our training dataset comprises multi-view dense ren- ders from an internal dataset analogous to Objaverse. Specif- ically, for each object, we render 512x512 resolution im- ages from 256 viewpoints, uniformly distributed across 16 azimuth and 16 elevation angles. This process yields approx- imately 700,000 rendered objects, with 140,000 classified as high-quality. Furthermore, we leverage the Emu-video synthetic dataset (Han et al., 2024a), which consists of 2.7 million synthetic multi-view videos. Each video comprises 16 frames capturing a 360-degree azimuth range.
Bridging tri-planes and 3D Gaussian Splatting. For ren- dering speed and memory efficiency, we opt to use 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to represent the 3D object. However, Instant3D uses a tri-plane NeRF representation instead. To bridge these two, we predict a set of 3D Gaussian from the tri- tri-p features via an MLP. Be-
Two-stage training, Initially, we pre-train treitrain FlexRM using a NeRF MLP architecture. This stage employs the Emu- video synthetic data, where we randomly select to 16 input images (256 256 resolution) and render 4 views with fixed rendering resolution of 256 x 256 and patch
5

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
set. This replacement probability is based the observa- tion that multi-view diffusion-generatee images typically exhibit inconsistencies and imperfections non-uniformly. Additionally, this approach encourages the reconstructor to focus more on high-quality input views. (3) Finally, we re-run FlexRM with gradients enabled, using the new input set as inputs and supervising it with novel view rendering losses, while utilizing perfectly rendered views as ground truth. This fine-tuning process enables FlexRM to learn to tolerate minor imperfections in input views and still produce high-quality 3D reconstructions.
Posdien
Opaciy
Figure 3. Imperfect Input View Simulation Examples. We simu- late different kinds of imperfect input views by feeding FlexRM's output back as input and manipulating 3D Gaussian parameters.
While FlexRM's outputs naturally contain small imperfec- tions, we also introduce random perturbations to FlexRM's generated 3D Gaussian points during step (1) to simulate wider range of imperfect inputs and promote greater di- versity. We sample randomly sized small cube from the large tri-plane cube and add noise with varying intensities to the 3D Gaussian parameters, excluding rotation. For ex- ample, adding noise to positions can simulate part-level 3D inconsistencies, while adding noise to color parameters can simulate color distortions. Adding noise to opacity results in a speckled or streaky appearance, while adding noise to scale leads blurring effect. Figure 3 illustrates these effects. During training. each effect is randomly used with probability of 0.2. This combines them for greater diversity. Please check Appendix B for more details.
resolution of 128 x 128 for supervision (L2, LPIPS (Zhang al., 2018), and opacity). The pre-training phase aims to provide good initialization for the subsequent GS MLP training and is conducted for 10 epochs, requiring 2 days on 64 A A10 GPUs.
For the second GS training stage, we utilize the 700,000 dense renders. A random number (between and 32) of input images (512x512 12 resolution) are fed into FlexRM, and we render 4 novel views (512x512 resolution) to compute losses. Given that our dense renders encompass images with diverse elevation angles, we implement weighted sampling for both input and rendered images. This assigns higher selection probabilities to images with elevation angles closer to zero. The training process spans 20 epochs and takes 4 days on 128 A100 GPUs.
4. Experiments
FlexRM generates 1M 3DGS points in under 0.5 second and renders in real time with single A100 GPU. Increasing the number of input images has only slight impact on speed and memory usage.
We evaluate Flex3D on the 3D generation (section 4.1) and 3D reconstruction (section 4.2) tasks, comparing it to state- of-the-art methods and ablating various design choices (sec- tion 4.3).
More details on implementation and training of FlexRM are presented in the Appendix
4.1. 3D Generation
3.3. Imperfect input view simulation
Method
|CLIP text sint VideoCLIIP lext simt Flex3D win rale
Even after input view selection, these views may still con- tain minor imperfections. To enhance FlexRM's s suitability for generation tasks, we require it to be robust to such im- perfections while maintaining high-quality 3D outputs. We achieve this robustness by simulating imperfect inputs dur- ing a fine-tuning stage.
OpenLRM
0.243
0.229
100 %
VFusion3D
0.265
0.238
95.0 97.5 %
LGM
0.266
0,240
InstantMesh
0.272
0.236
95.0 %
GRM
0.268
0.253
92.5
IN 33fff
0.252
0.231
95.0 %
3DTopia-X1. XL
0.254
0.231
97.5 %
Flex3D
0.277
0.255
Table I. Comparisons on 3D Generation Task. Flex3D achieves the highest scores for both CL.IP text similarity and VideoCI.IP text similarity, exhibiting better performance in text alignment. For generation quality, we conduct a user study to assess it, and the winning rate of Flex3D is always greater than 92%, demonstrating its strong generation performance.
This necessitates simulating a wide variety of imperfections efficiently. Simply adding noise in image space makes it difficult to simulate imperfections arising from geometric distortions. Instead, we propose a three-step process: (1) First, we perform inference using FlexRM with small ran- dom number of rendered images (between I and 8) as inputs to generate another random number of images (between and 32) for subsequent use as inputs. (2) Next, we use these generated images to replace the rendered images at the same viewing angles with a 50% probability, forming a new input
We leverage 404 deduplicated prompts from DreamFu- sion (Poole et al., 2022) to conduct an experiment on text-to- 3D or single-image- to a few recent feed-forward 3D generation methods in-
generation. We compare Flex3D
6

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
OO4F
bichon frise wearing academic regalia
made out of sushi
frog wearing sweater
OpenLRM
VFusion3D
LGM
InstantMesh
GRM
LN3Diff
3DTopia-XI.
Flex3D
Figure 4. Qualitative Results of Text-to-3D Generation. Flex3D demonstrates higher generation quality with strong 3D consistency, outperforming all other methods.
cluding Op (He & Wang, 2023; Hong et al., 2024),
results are presented in table I, where Flex3D outperforms baselines, showing high alignment in text prompt and generated content.
VFusion3D (Han et al., 2024a), LGM (Tang et al., 2024a), InstantMesh (Xu et al., 2024b), and GRM (Xu al., 2024c).
We also compare FD with two recent direct 3D gener- ation (diffusion) methods: LN3Diff (Lan et al., 2024) and 3DTopia-XL (Chen et al., 2024b). For GRM, we utilize its provided Instant3D's s multi-view diffusion model to generate input multi-vies images, and for InstantMesh, we employ the default Zero123+4 (Shi al., 2023a) for text-to-input multi-view image conversion. For direct 3D diffusion meth- ods, their
To further evaluate the overall quality of the generated con- tent, we conducted user study. Participants were pre- sented with pairs of 360 rendered videos one generated by Flex3D and one by a baseline model and asked to select their preferred video. We randomly selected 40 prompts for evaluation. The corresponding 40 pairs of generated videos were then independently evaluated by five users, with each user assessing all 40 pairs. For each pair, we collect five results, and the majority preference was recorded as a win rate for Flex3D. Results are also shown in table 1, where a significant number of votes goes to our method for its
generation pipeline.
We present qualitative results in fig. 4. Our model demon- strates strong generation capabilities with good global 3D consistency and detailed high-quality textures. Quantitative

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
high quality, regardless of the baselines used for compari- son. This shows that our method clearly generates better 31)
4.3. Ablation study and analysis
FlexRM in 3D Reconstruction. We first ablate various design choices of FlexRM using 3D reconstruction metrics, including: (1) not using the stronger camera conditioning, (2) directly predicting positions (a = 1), (3) not using positional offsets (a = 0), and (4) not using two-stage training. All experiments here are conducted on a weaker version of FlexRM,, trained with 140,000 data points and for 20 epochs only in stage 2. We use the same evaluation setting in section 4.2.
assets.
4.2. 3D Reconstruction
We utilize the Google Scanned Objects (GSO) dataset (Downs et al., 2022) for evaluation. From this dataset, we use 947 objects excluding some shoes that are so similar to be redundant. Each object is rendered at 512x512 resolution from 64 different viewpoints, which are generated using four elevation settings: -30%, 10" 30, and 45 The azimuth angles are uniformly sampled between 0 and 360
Results are shown in table 3, where we report the averaged results across four different settings: 1, 4, 8, and 16 input views. Overall, removing each component leads to a per- formance drop. Notably, directly predicting positions and removing positional offsets result in significant performance decreases. This highlights the importance of accurately mod- eling Gaussian positions for high-quality reconstruction. In- terestingly, removing stronger camera conditioning has relatively smaller impact on performance. This is because the advantages of stronger camera conditioning become more pronounced when a larger number of input views with varying camera poses are used. To validate this, we also test 32-view input experiment, where incorporating stronger camera conditioning improved PSNR by over 0.3 dB.
Method Input views PSNR
SSIM;
L.PPS),
CLIP image simt CDJ. NC1
OpCnLRM VPusion3D
15.83
0.821
0.209
0.602
19:10
0.827
0.158
FlexRM
21.21
0.125
0,832
InstantMesh
21.33 33
0,859
0.133
0.809
1.372 0.841
GRM
55.03
0.899
0. 102
0 0.369
1.496 0,866
FlexRM
25.58
0.894
0.074
0.893
0.878
FlexRM
26.33
0.897
0.069
0.906
1.188 0,881
MeeRM
16
26.51
0,902
0,068
0.911
1.182 0.884
HeRR MexRR
26.65
0.003
0.067
0.915
1.175 0.866
26.77
0.907
0.066
0.919
1.169 0.888
Table 2. Reconstruction Performance on the GSO Dataset.
FlexRM consistently outperforms other baselines, where it achieves the best results across different input view settings. CD (Chamfer Distance) values are multiplied by 10-?, and NC de- notes Normal Correctness. Increasing the number of input views for FMcxRM leads to improved reconstruction quality.
Ablation
PSNR
SSIM
LPIPS] CLIP image sim?
No stronger camera cond
24.31
0.871
0.092
0.865
Directly predict positions
23.41
0.840
0.096
0.831
No positional offscts
22.19
0.798
0.102
0.789
No two-stage training
23.38
0.838
0.098
0.827
In table 2 we report the performance of FlexRM on the GSO reconstruction task, using varying numbers of input views, namely 1, 4, 8, and 16. We compare our results to several baseline methods, including single-view reconstruc- tion models (LRM (He& & Wang. 2023). VFusion3D (Han et al., 2024a)) and vier reconstruction models (In- stantMesh (Xu et al., 2024b), GRM (Xu et al., 2024c)). For the single-view setting, we use the input view at 0 azimuth and 10 elevation as input. For the 4-view setting, we use views at 09, 90', 270", and 360 azimuth degrees, all at 10 elevation. For other numbers of views, we heuristi- cally select more views as input. The remaining views are used to compute the reported novel-view synthesis quality. The CD (Chamfer Distance) and NC (Normal Correctness) calculation protocol follows that of (Siddiqui et al., 2024).
Full model
24.35
0.873
0.090
0.868
Table 3. Ablation Study of FlexRM. We evaluate the impact of removing individual components of our proposed method.
Flex3D in 3D Generation. Here we study the candidate view generation and selection pipeline, utilizing a fully trained FlexRM, fine-tuned with simulated imperfect data, as the reconstructor. The evaluation protocol follows that outlined in section 4.2. We conduct ablation experiments by removing: (1) multi-view generation at varying eleva- tions, (2) consistency verification (resulting in random view selection), and (3) back view generation quality assessment.
Table 4 summarizes the results, demonstrating that the re- moval of any of these components leads to a decrease in both CLIP (Radford et al., 2021) and VideoCLIP (Wang et al., 2023a) text similarity scores. This shows the con- tribution of each component in achieving high-quality 3D generation.
Overall, FlexRM significantly outperforms baselines both 1-view and 4-view settings. This improvement is particu- larly evident in the LPIPS score, key metric reflecting perceptual quality, which demonstrates substantial gains. Beyond fixed input views, FlexRM is also capable of han- dling an arbitrary number of input views. We present re- sults for more view results, both exhibiting progressively stronger performance. Qualitative results are shown in the Appendix D.
Figure 5 demonstrates the impact of view selection on the quality of generated 3D assets. When view selection is not used, some of the generated input views in stage 1 may be less desirable. The blue circles highlight regions where deficiencies in the input result in poor generation quality.
8

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
However, by incorporating view selection, the model is able to select the most high-quality and consistent views input, generally resulting in improved 3D asset generation.
Rendared views wilhout
Imperfect Data Simulation. We analyze our imperfect data simulation strategy using both reconstruction and gen- eration metrics. Evaluation settings mirror those used in the ablation study. As shown in table 5, incorporating im- perfect data simulation yields improvements across both generative and reconstruction tasks. In contrast, simpler augmentation techniques like adding Gaussian or Salt-and- pepper noise with different intensities are found to degrade reconstruction performance. This suggests that our strategy effectively exposes the model to a wider range of data vari- ations, enhancing overall performance and robustness.
Rendered views with selection
Selected inpul views
Ablation
CCLIP text simtt VideoCLIIP text sim IPSNR1 SSIM? LPIPS.
Rendered views without selection
No simulation
Full model 0,277
0,250
24
24.90
0,888
0.889
0,084
0,086
Table 5. Ablation Study on Imperfect Data Simulation. Lever- aging imperfect data simulation strategy leads to a reasonable performance improvement in generative tasks and marginal im- provement in reconstruction tasks.
Rendered views with selection
5. Limitations
Seleted views
While our method can generate high-quality 3D Gaussians, the inherent problems associated with 3DGS are also present. For example, extracting clean meshes is not straightforward and usually requires multi-step post-processing can likely be mitigated in the near future given the fast de- velopment of Gaussians, either through new representations of Gaussians (Huang et al., 2024; Dai et al., 2024) or better ways to convert them to meshes (Wolf et al., 2024). Though our paper focuses on 3D object generation, another poten- tial limitation is that the tri-plane representation is usually limited by resolution size and cannot be easily used for large scene generation.
This issue
Figure 5. Generation Results With and Without View Selection. The top four rows show 3D generation results colorful rain- bow fish. The first row shows the 3D assets generated by Flex3D (displayed as rendered views) without our selection pipeline. The blue circles highlight regions exhibiting generation failures. The second row presents the generated views after applying view se- lection. The third and fourth rows show the sampled input views, where a green checkmark indicates that our method selected the view, and red cross indicates that the view was rejected. The bottom four rows show the same process for a different object.
6. Conclusion
This paper introduces Flex3D, novel feed-forward 3D gen- eration pipeline that produces high-qualit 3D Gaussian rep- resentations from text or single-image inputs. We propose a series of approaches to overcome the limitations of previous two-stage methods, significantly improving final 3D qual- ity. Extensive evaluations on benchmark tasks demonstrate that Flex3D achieves ofate-of-the-ar performance in both 3D reconstruction and generation. These results highlight the effectiveness of our approach in addressing the chal- lenges feed-forward 3D generation, paving the way for more robust, versatile, and accessible 3D content creation workflows for multitude of applications.
Ablation
CLIP text sim? VideoCLIP text sim?
generation at varying elevations
0.273
0.251
0.269 No back view assessment
No consistency verification
0.248
0.272
0.249
Full model
0.277
0.255
Table Ablation study on Candidate View Generation and Selection. We show the results of ablating different components of our proposed candidate view generation and selection pipeline.

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Impact Statement
Dai, P., Xu, J., Xie, W., Liu, X., Wang, H., and Xu, W. High- quality surface reconstruction using gaussian surfels. In ACM SIGGRAPII 2024 Conference Papers. Association for Computing Machinery, 2024.
Our work explores generative AI with a focus on generating 3D Gaussian representations from pre-existing 2D content. While we exclusively utilize ethically sourced and carefully curated training data, our model learns generalized ap- proach to 3D reconstruction. This means that if presented with problematic or misleading 2D image, our model could potentially generate a corresponding 3D object, though with some reduction in quality. However, despite these inherent risks, we believe our work can empower artists and creative tool
Dai, X., Hou, J., Ma, C.-Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al. Emu: Enhancing image generation models us- ing photogenic needles in a haystack. arXiv preprint arXiv:2309.15880 2023.
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: universe of annotated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142- 13153, 2023.
professionals by serving as a productivity-enhanee within their workflow. Furthermore, this technology has the potential to boost 3D content creation by lowering bar- riers to entry and providing access to individuals who lack specialized expertise.
References
Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S. Y., et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
Boss, M., Huang, Z. Vasishta, A., and Jampani, V. Sf3d: Stable fast 3d mesh reconstruction with uv- unwrapping and illumination disentanglement. arXiv preprint arXiv:2408.0066 2024.
Dosovitskiy. A. An image is worth 16x16 words: Trans- formers for image recognition scale. arXiv preprint arXiv:2010.11 11999, 2020.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Downs, L.,, Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T. B., and Vanhoucke, V. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Confer- ence on Robotics and Automation (ICRA), pp. 2553 IEEE, 2022.
Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
Erkoç, Z., Ma. F., Shan, Q.. Nießner, M., and Dai, A. Hyper- diffusion: Generating implicit neural fields with weight- space diffusion. arXiv preprint arXiv:2303.17013 2023,
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang. Q., Li, Z. Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint 112iv:1512.033/2, 2015.
Gao, R., Holynski, A., Henzler, P., Brussee, A.. Martin- Brualla, R., Srinivasan, P., Barron, J. T., and Poole, B. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint XrXiv:2405. 10314 14, 2024.
Chen, A., Xu, H., Esposito, S., Tang, S., and Geiger, A. Lara: Efficient large-baseline radiance fields. In Euro- pean Conference on Computer Vision (ECCV), 2024a.
Girdhar, R., Singh, M., Brown, A., Duval, Q. Azadi, S., Rambhatla, S. S., Shah, A., Yin, X.. Parikh, D., and Misra, I. Emu video: Factorizing text-to-video gener- ation by explicit image conditioning. arXiv preprint arXiv:231 10709, 2023.
Chen, R., Chen, Y., Jiao, N., and Jia, K. Fantasia3d: Disen- tangling geometry and appearance for high-quality text- to-3d content creation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22246 22256, 2023.
Han, J., Kokkinos, F., and Torr, P. Vfusion3d: Learning scalable 3d generative models from video diffusion mod- els. European Conference on Computer Vision (ECCV), 2024a.
Chen, Z., Tang, J., Dong, Y., Cao, Z., Hong, F., Lan, Y., Wang, T., Xie, H., Wu, T., Saito, S., Pan, L., Lin, D., and Liu, Z. 3dtopia-xl: High-quality 3d pbr asset generation via primitive diffusion. arXiv preprint arXiv:2409.12995 2024b.
Han, X., Gao, Z., Kanazawa, A., Goel, S., and Gandelsman, Y. The more you see in 2d, the more you perceive in 3d, 2024b.
10

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Hao, Z.,, Mallya, A., Belongie, S., and Liu, M.-Y. Ganeraft: Unsupervised 3d neural rendering of minecraft worlds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14072-14000 2021.
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., and Vondrick, C. Zero-1-to-3: Zero-shot image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp.
2023a.
He, and Wang, T.
Openlmm: Open-source large reconstruction models. https://gitie: com/ SDTopia/Openti 2023.
Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T. and Wang, W. Syncdreamer: Generating multiview- consistent images from single-view image. arXiv preprint arXiv:2309.0345: 2023b.
Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F. Sunkavalli, K., Bui, T., and Tan, H. Lrm: Large reconstruction model for single image to 3d. ICLR, 2024.
Long, X., Guo, Y.-C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.-H., Habermann, M., Theobalt, C., et al. Wonder3d: Single image to 3d using cross-domain dif- fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9970- 9980, 2024.
Huang, B., Yu, Z., Chen, A., Geiger, A., and Gao, S. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, pp- 1-11, 2024.
Lorraine, J., Xie, K., Zeng. X., Lin, C.-.., Takikawa, T., Sharp, N., Lin, T.-Y., Liu, M.-Y., Fidler, S., and Lucas, J. Att3d: Amortized text-to-3d object synthesis. arXiv preprint arXiv:2306.0733: 2023.
Huang, X. and Belongie, S. Arbitrary style transfer in real- time with adaptive instance normalization. In Proceed- ings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprini arXiv: 17 1,05101, 2017.
Hunyuan3D, T. Hunyuan3d 1.0: A unified framework for text-to-3d and image-to-3d generation. arXiv preprint arXiv:2411.0229; 2024.
Melas- MariazK, L., Laina, I., Rupprecht, C., Neverova, N., Vedaldi, A., Gafni, O., and Kokkinos, F. Im-3d: Iterative multiview diffusion and reconstruction for high-qualin 3d generation. arXiv preprinl arXiv:2402.08862. 2024.
Jiang, H., Huang, Q.. and Pavlakos, G. Real3d: Scaling up large reconstruction models with real-world images. arXiv preprint urXiv:2406.08499 2024.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1):99-106 2021.
Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
Kwak, J..g., Dong, E.,, Jin, Y., Ko, H., Mahajan, S., and Yi,
Nasiriany, S., Maddukuri, A., Zhang, L.,, Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. Robocasa: Large- scale simulation of everyday tasks for generalist robots. arXiv preprint 206.v:240..0255555555 2024.
K. M. Vivid-1-to-3: Novel view synthesis with video dif- fusion models. arXiv preprint arXiv:2312.01300 2023.
Lan, Y., Hong, F., Yang, S., Zhou, S., Meng. X., Dai, B., Pan, X., and Loy, C. C. Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation. In ECCV, 2024.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071993 2023.
Li, B., Zheng, C., Zhu, W., Mai, J., Zhang, B., Wonka, P., and Ghanem, B. Vivid-zoo: Multi-vier video generation with diffusion model. arXiv preprint arXiv:2406.0 08659, 2024a.
Poole, B., Jain, A., Barron, T., and Mildenhall, B. Dream- Fusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Li, C., Li, S., Zhao, Y., Zhu, W., and Lin, Y. Rt-nerf: Real- time on-device neural radiance fields towards immersive ar/vr rendering. In Proceedings of the 41st IEEE/ACM International Conference on Computer-Aidee Design, pp- 1-9, 2022.
Qiu, L., Chen, G., Gu, X., Zuo, Q., Xu, M., Wu, Y., Yuan, W., Dong, Z., Bo, L., and Han, X. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9914-991229 2024.
Li, Tan, H., Zhang, K., Xu, Z., Luan, F., Xu, Y., Hong Y., Sunkavalli, K. Shakhnarovich, G., and Bi, S. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. ICLR, 2024b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, 2021.
Tochilkin, D., Pankratz, D., Liu, Z., Huang, Z.,. Letts, A., Li, Y., Liang, D., Laforte, C., Jampani, V., and Cao, Y.- P. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02154 2024.
Ren, X., Huang, J., Zeng, X., Museth, K., Fidler, S., and Williams, F. Xcube (x3): Large-scale 3d generative modeling using sparse voxel hicrarchies. arXiv preprint arXiv:2312.03800 2023.
Voleti, V., Yao, C.-H., Boss, M., Letts, A., Pankratz, D., Tochilkin, D., Laforte, C., Rombach, R., and Jampani, V. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403. 12008 2024.
Shi, R., Chen, H., Zhang, Z., Liu, M., Xu, C.,, Wei, X., Chen, L., Zeng, C., and Su, H. Zerol23++: single image to consistent multi-view diffusion base model. arXiv preprint uriv:2230..55110, 2023a.
Wang, P. and Shi, Y. Imagedream: Image-promps multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201, 2023.
Shi, Y., Wang, P., Ye, J., Long, M., Li, K., and Yang, X. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023b.
Wang, P., Tan, H., Bi, S., Xu, Y., Luan, F., Sunkavalli, K., Wang, W., Xu, Z., and Zhang, K. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. ICLR, 2024a.
Siddiqui, Y., Monnier, T., Kokkinos, F., Kariya, M., Kleiman, Y., Garreau, E., Gafni, O., Neverova, N., Vedaldi, A., Shapovalov, R., et al. Meta 3d assetgen: Text-to-mesh generation with high-quality geometry, tex- ture, and pbr materials. arXiv preprint arXiv:2407.02445, 2024.
Wang. Y.. He, Y., Li, Y., Li, K., Yu, J., Ma, X., Chen, X., Wang, Y., Luo, P., Liu, Z.,, Wang, Y., Wang, L.,, and Qiao, Internvid: large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942. 2023a.
Sun, C., Han, J., Deng, W., Wang, X., Qin, Z., and Gould, S. 3d-gpt: Procedural 3d modeling with large language models. arXiv preprint 2rXhv:2310.12944 2023.
Wang. Y., He, X., Peng, S., Tan, D., and Zhou, X. Efficient LoFTR: Semi-dense local feature matching with sparse- like speed. In CVPR, 2024b.
Szymanowicz, S., Rupprecht, C., and Vedaldi, A. Splatter image: Ultra-fast single-viee view 3d reconstruction. arXiv preprint arXiv:2312. 13100, 2023a.
Wang. Z. Lu, C., Wang. Y., Bao. F., Li, C., Su, H., and Zhu, J. Prolificdreamer: High-fidelity and diverse text- to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 2325. 2023b.
Szymanowiez, S., Rupprecht, C., and Vedaldi, A. Viewset (Offusion::O)) image-conditioned 3d generative models from 2d data. arXiv preprint arXiv:2306.0788// 2023b.
Wang. Z., Wang, Y., Chen, Y., Xiang. C., Chen, S., Yu, D., Li, C., Su, H., and Zhu, J. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.05033 055344 2024c.
Tang, J., Chen, Z. Chen, X., Wang, T., Zeng, G., and Liu, Z. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024a.
Wei, X., Zhang, K., Bi, S., Tan, H., Luan, F., Deschaintre, V., Sunkavalli, K., Su, H., and Xu, Z. Meshlrm: Large re- construction model for high-quality mesh. arXiv preprint arXivv2404. 2024.
Tang, S. Zhang, F., Chen, J., Wang, P., and Yasutaka, F. Mvdiffusion: Enabling holistic multi-view image genera- tion with correspondence-awalt diffusion. arXiv preprint 3307,01097, 2023.
Wen, H., Huang. Z. Wang, Y., Chen, X., Qiao, Y., and Sheng, L. Ouroboros3d: Image-to-3d genera- tion via 3d-aware recursive diffusion. arXiv preprint arXhv:2406.031844 2024.
Tang. S., Chen, J., Wang, D., Tang, C., Zhang, F., Fan, Y., Chandra, V., Furukawa, Y., and Ranjan, R. Mvdif- fusion++: dense high-resolution multi-view diffusion model for single sparse- view 3d object reconstruction. arXiv preprint 2xX.: 2024b.
Wolf, Y., Bracha, A., and Kimmel, R. Gs2mesh: Surface reconstruction from gaussian splatting via novel stereo views. In ECCV 2024 Workshop on Wild 3D: 3D Model- ing, Reconstruction, and Generation in the Wild, 2024.
Tang, Z., Zhang, J., Cheng, X., Yu, W., Feng, C., Pang, Y., Lin, B., and Yuan, L. Cycle3d: High-quality and consis- tent image-to-3d generation via ceweerattionrronstturette cycle. arXiv preprint arXiv:2407. /9548, 2024c.
Woo, S., Park, B., Go. H., Kim, J.-Y., and Kim, C. Har- monyview: Harmonizing consistency and diversity in
12

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
one-image-to-3d. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10574-10584, 2024.
Yu, X., Xu, M., Zhang, Y., Liu, H., Ye, C., Wu, Y., Yan, Z.., Zhu, C., Xiong, Z., Liang, T., et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9150-9161, 2023.
Xie, D., Bi, S., Shu, Z., Zhang, K., Xu, Z., Zhou, Y., Pirk, S., Kaufman, A., Sun, X., and Tan, H. Lrm-zero: Training large reconstruction models with synthesized data. arXiv preprint arXiv:2406.0937/ 2024a.
Zhang, B., Cheng, Y., Yang. J., Wang. C.,, Zhao, F., Tang. Y., Chen, D., and Guo, B. Gaussiancube: Structuring gaus- sian splatting using optimal transport for 3d generative modeling. arXiv preprint arXiv:2403. 19655, 2024a.
Xie, D., Li, J., Tan, H., Sun, X., Shu, Z., Zhou, Y., Bi, S., Pirk, S., and Kaufman, A. E. Carve3d: Improving multi-view reconstruction consistency for diffusion mod- els with rl finetuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3369-6379 2024b.
Zhang, C., Song, H., Wei, Y., Chen, Y., Lu, J., and Tang, Y. Geolrm: Geometry- aware large reconstruction model for high-quality 3d gaussian generation. arXiv preprint arXiv:2400 15333, 2024b.
Zhang, K., Bi, S., Tan, H., Xiangli, Y., Zhao, N., Sunkavalli, K., and Xu, Z. Gs-Irm: Large reconstruction model for 3d gaussian splatring, arXiv preprint arXiv:2404. 19702, 2024c.
Xu, D., Yuan, Y., Mardani, M., Liu, S., Song, J., Wang, Z.,, and Vahdat, A. Agg: Amortized generative 3d gaussians for single image to 3d. arXiv preprint arXiv:2401.0444 0.4099 2024a.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Xu, J., Cheng, W., Gao, Y., Wang, X., Gao, S., and Shan, Y. Instantmesh: Efficient 3d mesh generation from a sin- gle image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07799) 2024b.
Zhao, H., Dai, X., Wang, J., Tong. S., Zhang. J., Wang, W., Zhang, L., and Ma, Y. Ctrl123: Consistent novel view synthesis via closed-loop transcription. arXiv preprint arXiv:2403.1 10953, 2024.
Xu, Y., Shi, Z., Yifan, W., Chen, H., Yang, C., Peng, S., Shen, Y., and Wetzstein, G. Grm: Large gaussian re- construction model for efficient 3d reconstruction and generation. arXiv preprint ar 2xXiv:2403.146211 2024c.
Zou, Z.-X., Yu, Z., Guo, Y-C.,, Li, Y., Liang, D., Cao, Y.- P., and Zhang, S.-H. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09144 2023.
Xu, Y., Tan, H., Luan, F., Bi, S., Wang, P., Li, J., Shi, Z Sunkavalli, K., Wetzstein, G., Xu, Z., et al. Dmv3d: De- noising multi-view diffusion using 3d large reconstruction model. ICLR, 2024d.
Zuo, Q., Gu, X., Qiu, L., Dong, Y., Zhao, Z., Yuan, W., Peng, R., Zhu, S., Dong, Z., Bo, L., al. Videomv: Consistent multi-view generation based large video generative model. arXiv preprint 2tti:24033122000 2024.
Yang, H., Chen, Y., Pan, Y., Yao, T., Chen, Z., Ngo, C.-W., and Mei, T. Hi3d: Pursuing high-resolution image-to-3d generation with video diffusion models. arXiv preprint arXiv:2409.07452, 2024a.
Yang, J., Cheng, Z., Duan, Y., Ji, P., and Li, H. Consistnet: Enforcing consistency for vieti-view images diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 7079-7088, 2024b.
Ye, J., Wang, P., Li, K., Shi, Y., and Wang, H. Consistent-1- to-3: Consistent image to 3d view synthesis via geometry- aware diffusion models. In 2024 International Confer- ence on 3D Vision (3DV), pp. 664-674. IEEE, 2024.
Yu, A., Ye, V., Tancik, M., and Kanazawa, A. pixelnerf: Neural radiance fields from one or few images. Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 4578 2021.
13

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
A. Implications for future research
Feed-forward Two-stage 3D generation. The key insight is that we introduced a series of methods to handle imperfect multi-view synthesis results in the common two-stage 3D generation pipeline. Our whole Flex3D pipeline introduces little computational cost but yields significant performance and robustness gains, and could serve as a common design pipeline for future research in 3D generation. Additionally, all individual components proposed in this work can be easily adopted by future research in 3D generation to improve performance. Similarly, design ideas analogous to the Flex3D pipeline could be readily adopted for large 3D scene generation.
Feed-forward Two-stage 4D generation. Moreover, our work could be beneficial for 4D generation, which is an even more challenging task that faces similar limitations to two-stage 3D generation pipelines. Our pipeline could be directly extended to handle 4D object generation tasks. One could first generate 64 views (16 time dimensions 4 multi-views) by fine-tuning video-based diffusion models, then slightly modify the view selection pipeline to keep only those views consistent across multiple views and time dimensions. Then, extend FlexRM from a tri tri-plane to a hex- additionally learn time offsets to enable 4D representation. This should yield strong method for 4D asset generation.
Generative 3D/4D reconstruction. Methods like Im-3D, Cat3D, and Cat4D rely on multí-view diffusion models to synthesize a large number of possible views, which are then used to fit a 3D/4D representation. Our work could inspire advancements in this area in two key ways:
View Selection: Our view selection pipeline could directly enhance these methods by filtering out inconsistent synthesized views before fitting a 3D representation, potentially leading to performance improvements. This approach can also be naturally extended to handle synthesized 4D views.
Efficiency: Fitting a 3D representation from scratch is time-consuming. The FlexRM reconstruction model we developed can process up to 32 views and has the potential to scale further. Synthesized views could first be processed through this model, which operates in under a second, to generate a strong initial 3D representation. This approach has the potential to dramatically reduce the time required for fitting 3D representation from the usual half an hour to under minute. This idea could also be extended to 4D, as adapting the current 3D reconstruction pipeline to 4D is straightforward.
Leveraging 3D understanding for generation. Keypoint matching techniques are used in this work to effectively mitigate multi-view inconsistencies. We hope this will also inspire the 3D generation community to incorporate advanced techniques from the rapidly evolving field of 3D understanding. Recent advances in deep learning have led to significant developments in matching, tracking, deep structure from motion, and scene reconstruction. These advancements offer the 3D generation community useful tools (such as pose estimation and keypoint matching), pseudo-supervision signals (e.g.. pse.uuuuooddpth supervision), and new model design ideas.
B. Implementation details
Training details. Our FlexRM is trained in a two-stage manner. In stage 1, we train it with 64 NVIDIA A100 (80GB) GPUs and use a total batch size of 512, where each batch consists of 4 multi-view images at a patch resolution of 128 x 128 for supervision. The input images have a resolution of 256 x 256, and the number of input images varies from I to 16. The model trained for 10 epochs with an initial learning rate of x 10-4, following a cosine annealing schedule. Training begins with a warm-up phase of 3000 iterations, and we use the AdamW optimizer (Loshchilov & Hutter, 2017). We apply gradient clipping at 1.0 and weight decay of 0.05, applied only to weights that are not biases part of normalization layers. Both training and inference are performed using 1 precision. The optimization target is a combination of three different losses: L2, LPIPS, and opacity, with corresponding coefficients of 1, 2, and 1, respectively.
Stage 2 utilizes 128 NVIDIA A 100 (80GB) GPUs. We increase the input image resolution to 512 x 512 and the maximum number of input images to 32. We maintain a total batch size of 512, with each batch consisting of 4 multi-view images at a resolution of 512 x 512 for supervision. The model is trained for 25 epochs. All other training settings including total batch size are identical to Stage 1.
For further fine-tuning using simulated imperfect input views as input, we follow the setting in Stage 2 but only train it with 32 NVIDIA 1100 (80GB) GPUs for 3 epochs. We use total batch size of 128 and an initial learning rate of 2 x 10-5,
3D Gaussian parameterization. For predicted 3DGS parameters with 14 dimensions, we provide implementation details on converting them into position offset, color, opacity, scale, and rotation. We follow the setting used in GS-LRM (Zhang
14

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Figure View Selection Visualizations. show some generated candidate views object. A green check mark indicates that our method selected the view, while red cross indicates that the view was rejected. As the visualization demonstrates, our method can effectively filter out views exhibit poor quality or inconsistent results, such as with artifacts, truncations, awkward perspectives. This allows to focus on reconstruction from high-quality viewpoints, leading to improved overall results.
et al., 2024c) for opacity, scale, and rotation.
Position offset: We activate the predicted offset using tanh function and apply a scaling factor of 0.25. This scaled offset then added the initial positions obtain the final 3DGS positions.
15

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Color: We utilize the same activation function as in Neural Radiance Fields (NeRF). The predicted color values first passed through sigmoid function, then multiplied by 1.002, and finally, 0.001 is subtracted. These processed values serve as zero-order Spherical Harmonics coefficients for the 3DGS.
Opacity: We subtract 2.0 from the predicted opacity before applying a sigmoid function. This approach ensures that the initial opacity values are around 0.1, which stabilizes the training process.
Scale: We subtract 2.3 from the predicted scale and then apply a sigmoid function. Additionally, we clip the scale to a maximum value of 0.3 and a minimum value of 0.0001. This design, similar to the opacity implementation, promotes stability during training.
Rotation:
We predict unnormalized quaternions and apply L2-normalization the activation function to obtain unit quaternions.
3D Gaussian noise injection. For all Gaussian parameters, we sample small cube size within a range of 10 x 10 x 10 to x 40 x 40, assuming whole grid size of 100 x 100 x 100. Each time, the size sampled individually for every parameter achieve greater diversity. The noise levels for position, color, and opacity are 0.1, i.e., random noise between 1 and 0.1 is added. The noise level for scale is 0.02.
C. View selection visualizations
This section provides further visualizations to illustrate the effectiveness of our view curation pipeline. Figure showcases five randomly selected examples where our method successfully identifies and selects high-quality viewpoints while filtering out those with undesirable characteristics. Our method preserves high-quality views from multiple angles for objects, including front, side, and back views.
D. Qualitative Results on 3D Reconstruction
We show qualitative comparison results between FlexRM and reconstruction baselines in 1-view and 4-view settings (fig. 7 and fig. 8). FlexRM demonstrates a stronger ability to perform high-fidelity 3D reconstructions, particularly exceeding other baselines when observed from various elevation angles. This advantage is evident in both single-view and sparse- view scenarios. The results highlight FlexRM's effectiveness capturing fine details and overall object shape, leading to more accurate and visually appealing reconstructions.
E. Failure cases
We present some failure cases of our candidate view generation and selection pipeline in fig. 9. A notable failure occurs when the input image contains floaters or small transparent objects. This leads to incorrect generation results, especially different elevation angles. The view selection pipeline only partially removes these incorrect results. Additionally, our view selection pipeline can sometimes produce incorrect results, particularly with objects containing thin geometries. By enabling the use of an arbitrary number of high-quality views and enhancing robustness input imperfections, Flex3D not only significantly improves the final 3D quality but also offers greater versatility. These thin components contribute less to feature matching, making them more susceptible to incorrect selection.
16

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Input
VFusion3D
FleRR
Ground
Truth
Figure 7. Single-Vies View Reconstruction Results, showcasing FlexRM's ability to achieve reasonable reconstructions from only single-
view observation.
Inputs
InstantMesh
GRM
FlexRM
Ground Truth
Figure 4-view Reconstruction Results. FlexRM able to perform high-fidelity sparse- view reconstructions that closely resemble the ground truth, particularly when viewed from different elevation angles, outperforming baseline reconstructors.
17

---

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Figure 9. Failure Cases. A green check mark indicates that method selected the view, while red cross indicates that the view was rejected, Question marks indicate incorrect selection results. The top two rows show results for a mushroom, highlighting difficulties with floaters or small transparent objects inside the input image. The bottom two rows illustrates the challenges in filtering generated views of objects with thin thin geometries.
18


Reviews:
[{"review_id": "MVtl2s9ISp", "reviewer": "Reviewer_PAZq", "summary": "This paper introduces Flex3D, a novel two-stage framework designed for high-quality 3D generation from text, single images, or sparse views. In the first stage, the framework employs multi-view diffusion models to generate multiple images from diverse viewpoints, coupled with a view selection mechanism to filter out inconsistent or low-quality views. In the second stage, the varying selected views are fed into a transformer-based architecture that leverages a tri-plane representation, which is subsequently decoded into 3D Gaussians for efficient and high-fidelity 3D reconstruction.\n## Update after rebuttal  \nBased on the authors' response, I keep my weak accept rate.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback. We appreciate you acknowledging several strengths, including the paper's clear organization, the effectiveness of our view selection in improving input quality, the practicality of handling varying view numbers in reconstruction, and the potential community value of the proposed modules.We provide point-by-point responses below, addressing each weakness and question raised.\n\n**1: Exploring fundamental challenges or valuable insights would further enhance the paper's impact.**\n\nWe concur with the reviewer's suggestion regarding the value of discussing fundamental challenges and insights. While our primary focus is mitigating suboptimal outputs from the first-stage in common two-stage 3D Gen models, we discussed insights and future directions (e.g., feed-forward 3D/4D generation, generative reconstruction) in the Appendix's section A.\n\n**2.1: Is 2000 labeled samples enough for accurate filtering?**\n\nYes, we found 2,000 labeled samples to be sufficient for training an accurate filter.  This sample size also aligns with practices in related work, such as the data curation pipeline in Instant-3D. Its effectiveness stems from leveraging powerful pre-trained image encoders (DINO).\nTo validate accuracy and robustness, we tested on 100 held-out videos from our method and 100 from SV3D (out-of-distribution). The classifier achieved 93% accuracy on our videos and 90% on SV3D's. These high rates, especially on OOD data, show the 2,000 samples were sufficient and the filter is robust.\n\n**2.2:  Can we trust manual labeling?**\n\nWe conducted a rigorous labeling process, for manual labeling, the authors first carefully labeled 100 sample videos. These were then provided to two labelers, and each labeler was asked to label approximately 1,000 videos, resulting in a total of 2,000 labeled videos. \nThe trustworthiness is corroborated by the strong empirical performance of the classifier trained on these labels (high accuracy detailed in 2.1). Furthermore, results in Table 4, Figures 5-6 show this filter significantly improves our generation pipeline's overall performance, indicating the manual labeling was reliable for its purpose.\n\n**2.3:  Are there better, more systematic ways to evaluate consistency?**\n\nWe conducted a sensitivity analysis that indicates that the final generation results are relatively robust to variations in this threshold within a reasonable range (50% to 70%). For instance, varying the threshold between 50% and 70% yielded comparable final generation quality, as measured by CLIP text similarity (ranging from 27.4 to 27.7) and Video CLIP text similarity (ranging from 25.3 to 25.7).\nWe agree that more sophisticated methods like learned metrics or adaptive thresholds are promising future research directions for potentially more optimal filtering.\n\n**3: A simpler and more efficient training approach might be better—something that still delivers strong results but is easier for others to adopt and build on.**\n\nWe agree the full pipeline (NeRF pre-training, 3D Gaussian training, imperfect view simulation training) is resource-intensive. However, efficiencies can facilitate adoption: Stage 1 (NeRF) can be bypassed by initializing Stage 2 (GS) directly from available pre-trained Instant-3D weights. Stage 3, which enhances robustness to imperfect inputs, is optional if the primary application is high-quality reconstruction from clean views.\nThese optimizations reduce the core training requirement primarily to Stage 2, substantially lowering the resource barrier and complexity compared to training everything end-to-end from scratch.\n\nFor board adapatation, the FlexRM architecture is designed with a minimalist philosophy and can be easily reproduced based on Instant-3D, which can be easily adopted or being implemented by others.\nNevertheless, we recognize that further reducing the computational demands of large-scale 3D generative models remains an active and important research challenge across the field.\n\n**4: Maybe trying other ways to handle noise, like adversarial training or more varied noise types.**\n\nWe tested adding Gaussian noise (σ up to 0.05), salt-and-pepper noise (density up to 0.05), and combining Gaussian noise with our simulation. A comparison using 4-view reconstruction showed these other noise types degraded performance noticeably. In contrast, our proposed simulation pipeline using 3D Gaussians slightly enhanced performance, suggesting it is a more suitable approach in this context.\n\n| Approach          | PSNR↑ | SSIM↑ | LPIPS↓ | CLIP image sim↑ |\n|-------------------|-------|-------|--------|-----------------|\n| No noise          | 25.51 | 0.893 | 0.075  | 0.893           |\n| Proposed          | 25.55 | 0.894 | 0.074  | 0.893           |\n| Gaussian          | 24.93 | 0.871 | 0.083  | 0.874           |\n| Salt and pepper   | 25.18 | 0.882 | 0.078  | 0.880           |\n| Proposed + Gaussian | 25.12 | 0.879 | 0.080  | 0.881           |", "claims_and_evidence": "The main claims made in the paper are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The view selection mechanism seems reasonable and works well in experiments, but it feels a bit too engineered. The quality assessment is trained on just 2,000 manually labeled samples—is that enough for accurate filtering, and can we trust manual labeling? Also, the consistency check uses a fixed threshold (60% matching points), which might not be the most robust approach. Are there better, more systematic ways to evaluate consistency, like learned metrics? Exploring these could make the method more reliable and generalizable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and clearly demonstrates the entire pipeline.\n2. The view selection mechanism effectively filters out low-quality or inconsistent views, providing higher-quality and more consistent inputs for the reconstruction process. \n3. The reconstruction model can handle a varying numbers of input views, making it more practical for real-world applications.\n4. The paper proposes a series of effective modules in the pipeline, like view selection and robust reconstruction. If open-sourced, these could be useful for future work in the community.\n\nWeakness:\n1. This work builds on existing technologies (e.g., multi-view diffusion models, 3D Gaussian splatting, Tri-plane) and introduces a set of engineering tricks (e.g., view selection with SVM and LoFTR). Although these contributions lead to performance gains, exploring fundamental challenges or valuable insights would further enhance the paper's impact.\n2. The view selection mechanism seems reasonable and works well in experiments, but it feels a bit too engineered. The quality assessment is trained on just 2,000 manually labeled samples—is that enough for accurate filtering, and can we trust manual labeling? Also, the consistency check uses a fixed threshold (60% matching points), which might not be the most robust approach. Are there better, more systematic ways to evaluate consistency, like learned metrics? Exploring these could make the method more reliable and generalizable.\n3. The training process is pretty resource-intensive, requiring 32, 64, and even 128 A100 GPUs, and the whole training process is quite complicated, making it hard to reuse or replicate. A simpler and more efficient training approach might be better—something that still delivers strong results but is easier for others to adopt and build on. This could make the method more accessible and practical for the wider research community.\n4. The additional ablation in supplementary material shows only tiny improvements from the imperfect input simulation, so it’s not clear if this really makes the model more robust to noisy inputs. The idea makes sense, but the results don’t show a big impact, it seems unnecessary and redundant. Maybe trying other ways to handle noise, like adversarial training or more varied noise types, could make this part more convincing.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J6bH6W3B1Z", "reviewer": "Reviewer_kNaj", "summary": "This paper introduces two novel modules for achieving high fidelity 3D generation. The first one is candidate view generation and selection module, which generates a pool of novel view images and adopt a SVM scorer to select high fidelity novel views which are then sent to the second module named Flexible Reconstruction Model to reconstruct the final 3D models. The idea of selecting high fidelity novel view images from a large pool is interesting. As there are a lot of novel view synthesis methods, this method allow us to combine them all to achieve better performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful feedback. We are encouraged by your recognition of our core view selection strategy as interesting and novel, and we appreciate you noting its potential to improve reconstruction quality by filtering inconsistent views. We also value your positive comments regarding the paper's clarity, motivation, and its potential to serve as a useful baseline for future 3D generation research alongside related works. We address each specific weakness and question below.\n\n**1: Performance of view selection module, report classification accuracy.**\n\nTo evaluate our view selection module, we first manually labeled 404 videos generated by our multi-view EMU model (from deduplicated prompts from DreamFusion). We manually established a ground truth set (GT_Set) for each video. To mitigate subjective bias in determining the absolute 'best' views among all 20 frames, we focused on selecting approximately 10 clearly high-quality views per video to serve as the GT_Set. The labeling process is similar to that described in our response to Reviewer **PAZq** (**2.2: Can we trust manual labeling?**). The authors first carefully labeled 20 sample videos. The remaining videos were then assigned to two labelers for annotation.\n\nWe used the Intersection over Union (IoU) metric to evaluate the quality of view selection per video. For each video, we compared the ground truth set (GT_Set) with the set of views selected by our model (Selected_Set). The IoU is calculated as:\n\n**IoU = |GT_Set ∩ Selected_Set| / |GT_Set ∪ Selected_Set|**\n\nOur model achieves an average IoU of **0.72**. In terms of accuracy, treating results with an **IoU > 0.6** as accurate, we achieve an **accuracy** of **88.6%**. This result indicates strong performance and demonstrates the effectiveness of our proposed selection module, which also operates in near real-time.\n\n**2: The SVM is only trained with 2000 labeled samples, will such a small number of data enough for the SVM to generalize to different generation cases?**\n \nYes, we found 2,000 labeled samples to be sufficient for training an accurate filter.  This sample size also aligns with practices in related work, such as the data curation pipeline in Instant-3D. Its effectiveness stems from leveraging powerful pre-trained image encoders (DINO).\n\nTo validate accuracy and robustness, we tested on 100 held-out videos from our method and 100 from SV3D (out-of-distribution). The classifier achieved **93%** accuracy on our videos and **90%** on SV3D's. These high rates, especially on OOD data, show the 2,000 samples were sufficient and the filter is robust.\n\n**3: If the back view is not selected, could the model still works well with only side view as input?**\n\nYes, our pipeline functions effectively even without selected back views.\n\nSelection Module: Our filter achieves high accuracy (**>90%**). If it excludes the back view, this typically indicates poor generation quality for the back side. The module correctly removes it and tends to retain other, higher-quality views (often front/side), which is crucial for the final 3D quality.\n\nReconstruction Model: Our FlexRM model is trained to handle a variable number and arbitrary combination of input views. While it can generate a full 3D object even with fewer or missing views (e.g., no back view), the final reconstruction quality definitely benefits from having more, high-quality input views. This is precisely why the selection module is valuable.\n\nTherefore, filtering out poor-quality back views and feeding the reconstruction model with the remaining, better views leads to higher final 3D quality compared to forcing the inclusion of the low-quality one. The CLIP text sim on 404 DreamFusion prompts are **0.277** and **0.270** for such two cases (with back view selection vs always select back view), supporting our claims.\n\n**4: Although the paper proposes a selection module to filter inconsistent inputs, the performance is still bounded by the quality of novel view synthesis models.**\n\nWe acknowledge that final performance is influenced by the novel view synthesis (NVS) model's quality, a characteristic common to two-stage pipelines. However, our selection module specifically mitigates this limitation. By filtering NVS outputs and selecting only the highest-quality views available, our approach makes better use of the NVS model's capabilities and reduces the negative impact of inconsistent views, achieving a tighter performance bound. Consequently, even with the same underlying NVS model, our method yields superior final 3D quality. Furthermore, NVS models themselves are expected to improve, potentially benefiting significantly from advancements in large-scale video generation models which learn implicit 3D consistency from vast video data.", "claims_and_evidence": "The claim that novel views generation is challenging and could not guarantee to be consistent is true and make sense. And the claim that more consistent views lead to better reconstruction results is also demonstrated in Table 2 in the paper.", "methods_and_evaluation_criteria": "The proposed method make sense for me. And the evaluation criteria is consistent with prior works.", "theoretical_claims": "No such claims made in the paper.", "experimental_designs_or_analyses": "The experiments for this method should be divided into two parts. The first part is novel view synthesis, which mainly focuses on the view selection module and the second part is the reconstruction module, focusing on the reconstruction quality. I appreciate that the author conduct thorough experiments (including ablation experiments) for the reconstruction module. However, I think the evaluation of view selection is not enough. I would like to see more discuss about this module:\n* What’s the performance of this module, the author may report the classification accuracy.\n* The SVM is only trained with 2000 labeled samples, will such a small number of data enough for the SVM to generalize to different generation cases?\n* If the back view is not selected, could the model still works well with only side view as input?", "relation_to_broader_scientific_literature": "The paper introduces a novel 3D generation method, which may benefit fields like 3D generation, reconstruction and understanding. The prior related finding like instant3D[1], LRM[2] have been demonstrated to have certain level of influence in the field of 3D vision. \n\n\n\n[1] Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model (ICLR2024)\n\n\n[2] Lrm: Large reconstruction model for single image to 3d (ICLR2024)", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths:\n* The idea of the paper that adopts view selection strategy to filter inconsistent views is interesting and novel to me.\n* Although the idea of involving more views for reconstruction has been proposed in previous work like [1], the paper has introduced several strategies that can help improve performance, which could provide a good baseline for future work.\n* The paper is good-writing and easy to understand.\n\n[1] GeoLRM:Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation (ECCV2024)\n\nWeakness:\n* Please see experiment part for the discuss of evaluation experiments.\n* Although the paper proposes a selection module to filter inconsistent inputs, the performance is still bounded by the quality of novel view synthesis models.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ENSaJ1YTw", "reviewer": "Reviewer_qp7v", "summary": "This paper proposes a two-stage framework called Flex3D for 3D generation and reconstruction. The first stage leverages multi-view and video diffusion models to generate a large candidate set of views, then filters them according to both visual quality and multi-view consistency. The second stage uses a flexible reconstruction module (FlexRM)—a Transformer-based approach—to convert the curated views into 3D Gaussian points, aiming to achieve fast rendering and high-quality 3D outputs. The authors claim that Flex3D is capable of generating consistent and improved 3D objects under various input conditions (e.g., text or single images), and they report advantages over several recent baselines in their experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review and constructive feedback. We are encouraged by your comments on our pipeline's clarity, the tri-plane + Gaussian Splatting design's performance and potential impact, and the solution's coherence for common two-stage 3D generation pipelines. We provide our responses below, addressing the specific weaknesses and questions raised.\n\n**1:Limited novelty.**\n\nTwo-stage 3D generation pipelines, such as Instant3D and many others, represent a popular and effective class of frameworks. However, a significant limitation of all these approaches is that while their reconstructors perform well with sparse-view reconstruction, the final 3D quality remains constrained by the quality of the generated multi-views. Our work directly tackles the challenge of handling suboptimal outputs from this initial staage. To achieve this, we propose three novel key methods including view selection, a flexible view reconstruction architecture, and noise simulation. We believe both the core concept of mitigating first-stage limitations and the specific proposed methods possess some novelty. For instance, the view selection process utilizes 3D geometric priors, and our reconstruction model, combining tri-planes with 3DGS, offers both speed and the flexibility to handle varying numbers of input views.\n\n**2: View selection performance.**\n\nPlease see our response to Reviewer **kNaj** on **point 1**.\n\n**3: Compare view selection pipeline to multimodal LLM-based filtering.**\n\nUsing the IoU metric, we compare our performance with three MLLMs: GPT-4o, GPT-4o mini, and Gemini 1.5 Pro. Their performances are **0.64**, **0.49**, and **0.57**, respectively, all worse than ours (**0.72**).\nFor the \"ramen\" example in Figure 5, our pipeline selected [1, 2, 3, 5, 6, 10, 11, 12, 13, 18, 19, 20] . In comparison, GPT-4o selected [1, 2, 5, 6, 10, 12, 13, 17, 18], GPT-4o mini selected [1, 5, 7, 10, 11, 13, 17, 18], and Gemini selected [1, 5, 7, 13, 17].\nCompared with our pipeline, Gemini rejected frames [2, 3, 6, 10, 11, 12, 18, 19, 20] and selectedbad frames [7, 17] where chopsticks are missing or blurry. GPT-4o mini also selected these bad frames [7, 17] while missing several high-quality frames like [2, 3, 6, 12, 19, 20]. GPT-4o performed better, selecting mostly high-quality frames, but still missed potentially useful views like [11, 19, 20]. \n\nIn conclusion, while MLLMs are not yet as effective and efficient as our proposed pipeline for this specific task, using them for view selection holds strong potential.\n\n**4: More direct analysis of view selection, how errors in filtering affect final 3D quality, how sensitive?**\n\nAlthough our view selection pipeline is generally strong, achieving **93%** accuracy for back view assessment and **0.72** IoU for overall view selection, errors can negatively impact the final 3D quality. Table 4 shows how view selection affects final quality quantitatively. Generally, incorporating bad views degrades the quality of the final 3D output, and the strength of this effect tends to be related to the number of good views. For example, when a larger number of high-quality views are selected as input, the negative impact of incorporating a poor view tends to be less significant. Missing a high-quality view also degrades the final 3D output quality; similarly, this impact is less significant when many other good views are already included.\n\n**5: User study details.**\n\nParticipant: Five computer vision or machine learning researchers participated in the evaluation. Two were from the US, two from Europe, and one from Asia.\n\nMethodology: Participants viewed paired 360° rendered videos—one generated by Flex3D and one by a baseline method—presented via a Google Form. Video pairs were presented in random order and randomized left/right positions. Participants selected the video they preferred based on overall visual quality.\n\nStatistical Significance: We collected **1400** valid results (5 participants * 7 baselines * 40 videos). Flex3D was preferred in at least 92.5% of comparisons across all 7 baselines, strongly suggesting better visual quality.\n\n**6: No 3D metrics for evaluation. How does the view selection process improve geometry consistency?**\n\nIn Table 2, we reported 3D metrics for the reconstruction task, including Chamfer Distance and Normal Correctness, where our FlexRM model clearly outperforms other baselines. Evaluating geometry consistency for generation tasks is challenging due to the absence of GT. We thus employ VGGSfM as a proxy to assess the 3D quality of the generated models. Specifically, we render 16 views covering a 360° azimuth from the generated 3D Gaussians and measure the success rate of VGGSfM in estimating poses (correct poses for at least 8 views). Among 404 results, our full pipeline with view selection achieves a **65.6%** success rate, higher than the **59.6%** rate obtained without it. This confirms that our view selection strategy improves geometric consistency.", "claims_and_evidence": "From the reported experiments, there is some supporting evidence that providing more and better-filtered input views can improve 3D quality. However, direct quantitative demonstrations of how effectively the “view selection” pipeline finds the actual “best angles” or significantly resolves multi-view inconsistencies are somewhat limited. Much of the discussion about “view selection” improvements rests on qualitative results or the authors’ own metrics. Stronger, more thorough experiments—especially on how well their selection strategy consistently selects “optimal” viewpoints—would bolster the paper’s claims.", "methods_and_evaluation_criteria": "Method design:\n\n**Stage 1**: Two specially fine-tuned diffusion models (one focusing on elevation angles, the other on azimuth sweeps) produce candidate images; a quality classifier plus feature matching (LoFTR) filter out inconsistent or low-quality views.\n\n**Stage 2**: A tri-plane + 3D Gaussian Splatting network (FlexRM) renders a 3D object from the selected images, with extra camera encoding and noise simulation strategies to manage varying numbers and qualities of input views.\n\n**Evaluation metrics**: The authors primarily use 2D image-based quality measures (PSNR, SSIM, LPIPS) plus CLIP-based semantic scores, and a user study. While these are relevant, more explicit 3D metrics (e.g., Chamfer Distance, surface normal consistency) or direct benchmarks of the selection accuracy would further strengthen the results.\n\nAdequacy: The chosen metrics address visual quality but fall short of fully capturing 3D geometric consistency. Some additional metrics or more detailed reports on how well the selection step contributes to consistent geometry would be beneficial.", "theoretical_claims": "The paper does not present new theoretical derivations or formal proofs; most contributions lie in the design of a two-stage pipeline and its architectural modifications. As such, there is no major theoretical analysis to be audited.", "experimental_designs_or_analyses": "**Dataset variety and generalization**: The paper mentions a fairly large synthetic dataset and tests with GSO-like scanned objects. While this shows decent coverage, additional experiments on more diverse categories or real-world scenes would clarify the method’s broader applicability.\n\n**User study**: The user study is rather high-level, lacking details on participant demographics, rating procedures, or how statistical significance was assessed.", "relation_to_broader_scientific_literature": "Flex3D follows the increasingly popular “generate multi-view images first, then reconstruct” paradigm. Its main focus is on engineering a pipeline that (1) selects higher-quality and more consistent input views, and (2) learns a robust tri-plane + 3D Gaussian network to handle input imperfection. Compared to direct 3D diffusion approaches, this two-stage system might be simpler to integrate with current 2D diffusion models but does not necessarily provide novel insights beyond an engineering standpoint. The paper might have benefited from a deeper comparison or discussion regarding other advanced multi-view quality-control approaches or large multimodal models for automated viewpoint filtering.", "essential_references_not_discussed": "I believe the related works that are essential to understanding the key contributions of the paper.", "other_strengths_and_weaknesses": "Strengths\n\n1. The overall pipeline is clearly presented and practically oriented.\n2. The tri-plane + Gaussian Splatting design has potential for efficient rendering.\n3. The method provides a coherent solution for integrating multi-view diffusion models with a flexible 3D reconstructor.\n\nWeaknesses\n\n1. Limited novelty: The paper primarily refines existing ideas, an Instant3D with filtered multi-view inputs, with most contributions being incremental engineering or pipeline reorganizations.\n2. View selection performance: The paper lacks rigorous experiments demonstrating that their filter truly finds optimal angles or reliably removes inconsistent samples.\n3. Evaluation scope: The focus on 2D metrics and a relatively small user study might not fully substantiate claims of improved 3D consistency or geometry quality.", "other_comments_or_suggestions": "1. More direct analysis of view selection:  More quantitative metrics would clarify the effectiveness of the filtering and how errors in filtering affect final 3D quality.\n2. Compare to multimodal LLM-based filtering: Attempting view selection via GPT-4V or similar might highlight the pros and cons of simpler geometric feature matching vs. large-model approaches.\n3. Expanding user study details: Clarify participant backgrounds, rating methodology, and any significance testing. This would make user study findings more credible.", "questions_for_authors": "1. Accuracy of filtering: Could you provide quantitative measures (e.g., how often “bad” back views are successfully excluded, or how many “good” side views get missed)? How sensitive is the final 3D quality to mistakes in this step?\n2. Large-model filtering: Have you tested more advanced or larger models (e.g., GPT-4V) for quality checks? Would that significantly improve the system’s overall generation?\n3. 3D metrics: Will you include surface-based measures (e.g., Chamfer Distance)? How does the view selection process specifically improve geometry consistency, if at all?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper initially got mixed feedback before rebuttal, including one weak reject and two weak accepts, with some concerns raised about novelty and other details. However, the authors provided a convincing rebuttal that successfully addressed the main points of criticism, leading all reviewers to converge on recommending acceptance (final scores 3, 3, 3). While initial novelty concerns were noted, the proposed method's ability to handle flexible view inputs and the strong experimental results highlight its contribution and potential impact. 

Therefore, I recommend accepting this paper for ICML, and I strongly encourage the authors to integrate the feedback from all reviewers into their final camera-ready version.
Decision: Accept (poster)