PEERING INTO THE UNKNOWN: ACTIVE VIEW SELECTION WITH NEURAL UNCERTAINTY MAPS FOR 3D RECONSTRUCTION
Authors: Zhengquan Zhang
Feng Xu
Mengmi Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Active viewpoint selection; efficient 3D reconstruction; viewpoint sampling; uncertainty-driven learning
Primary area: applications to robotics, autonomy, planning
TL;DR: An efficient active viewpoint selection algorithm by replacing time costly online reconstruction with neural uncertainty map

Abstract: Imagine trying to understand the shape of a teapot by viewing it from the front—you might see the spout, but completely miss the handle. Some perspectives naturally provide more information than others. How can an AI system determine which viewpoint offers the most valuable insight for accurate and efficient 3D object reconstruction? Active view selection (AVS) for 3D reconstruction remains a fundamental challenge in computer vision. The aim is to identify the minimal set of views that yields the most accurate 3D reconstruction.
Instead of learning radiance fields, like NeRF or 3D Gaussian Splatting, from a current observation and computing uncertainty for each candidate viewpoint, we introduce a novel AVS approach guided by neural uncertainty maps predicted by a lightweight feedforward deep neural network, named UPNet. 
UPNet takes a single input image of a 3D object and outputs a predicted uncertainty map, representing uncertainty values across all possible candidate viewpoints. By leveraging heuristics derived from observing many natural objects and their associated uncertainty patterns, we train UPNet to learn a direct mapping from viewpoint appearance to uncertainty in the underlying volumetric representations. 
Next, our approach aggregates all previously predicted neural uncertainty maps to suppress redundant candidate viewpoints and effectively select the most informative one. Using these selected viewpoints, we train 3D neural rendering models and evaluate the quality of novel view synthesis against other competitive AVS methods. Remarkably, despite using half of the viewpoints than the upper bound, our method achieves comparable reconstruction accuracy. In addition, it significantly reduces computational overhead during AVS, achieving up to a 400 times speedup along with over 50\% reductions in CPU, RAM, and GPU usage compared to baseline methods. Notably, our approach generalizes effectively to AVS tasks involving novel object categories, without requiring any additional training. All code, models, and datasets are available at \url{https://github.com/ZhangLab-DeepNeuroCogLab/PUN}.

Published as a conference paper at ICLR 2026
PEERING INTO THE UNKNOWN: ACTIVE VIEW SELECTION WITH NEURAL UNCERTAINTY MAPS FOR 3D RECONSTRUCTION
Zhengquan Zhang Feng Xu² Mengmi Zhang
1 Deep NeuroCognition Lab, Nanyang Technological University, Singapore
Fudan University, China
f Address correspondence to mengmi. zhangentu.edu
ABSTRACT
Imagine trying to understand the shape of a teapot by viewing it from the front-you might see the spout, but completely miss the handle. Some perspectives naturally provide more information than others. How can an Al system determine which viewpoint offers the most valuable insight for accurate and efficient 3D object reconstruction? Active view selection (AVS) for 3D reconstruction remains a fundamental challenge in computer vision. The aim to identify the minimal set of views that yields the most accurate 3D reconstruction. Instead of learning radiance fields, like NeRF or 3D Gaussian Splatting, from current observation and computing uncertainty for each candidate viewpoint, we introduce a novel AVS approach guided by neural uncertainty maps predicted by a lightweight feedforward deep neural network, named UPNet. UPNet takes a single input image of 3D object and outputs a predicted uncertainty map, representing uncertainty values across all possible candidate viewpoints. By leveraging heuristics derived from observing many natural objects and their associated uncertainty patterns, we train UPNet to learn a direct mapping from viewpoint appearance to uncertainty in the underlying volumetric representations. Next, our approach aggregates all previously predicted neural uncertainty maps to suppress redundant candidate viewpoints and effectively select the most informative one. Using these selected viewpoints, we train 3D neural rendering models and evaluate the quality of novel view synthesis against other competitive AVS methods. Remarkably, despite using half the viewpoints than the upper bound, our method achieves comparable reconstruction accuracy. In addition, it significantly reduces computational overhead during AVS, achieving up to a 400 times speedup along with over 50% reductions in CPU, RAM, and GPU usage compared to baseline methods. Notably, our approach generalizes effectively AVS tasks involving novel object categories, without requiring any additional training. All code, models, and datasets are available at bbttps://qitht.. com/ ccon:/2aannnaaa--eeeeeerroooooa
INTRODUCTION
Actively interacting with the environment reduce uncertainty and minimize prediction errors is fundamental capability of embodied intelligent systems HastemsHaan & Zhang (204));)002)))))) (Fistoon (2010); Zhang Xu (2024). Some viewpoints naturally offer more informative observations than others for example, front view of teapot may only reveal the spout, providing limited information, whereas a side view can expose both the handle, the spout, and detailed surface textures of the body. Active View Selection (AVS) Sequeiia et al. (1996); Jia et al. (2009); Connolly (1985); Pito (1999) addresses this problem by selecting a minimal set of viewpoints that collectively maximize information gain. See Fig. Taa for an illustration of the AVS task in the context of 3D object reconstruction. AVS is critical in a range of real-world applications, including robotic control
Khandelwal al. (2023): Zhhag et al. (2018h) Lv al. (2023), search and rescue Zhang et al.

---

Published as a conference paper at ICLR 2026
2022; Jia et al. (2025): Ding et al. (2022); Gupta et al. (2021); Wang al. (2025); Niroui et al. 2019 Zhang et al. (2018a), and cultural heritage digitization Serafin et al. (2016).
Classical 3D reconstruction methods represent objects using explicit volumetric formats such as point clouds Dornhege & Kleiner (2013), voxel grids Dai et al. (2020); Dang et al. (2018), or occupancy maps Georgakis al. (2022); Ramakrishnan et al. (222)). AVS methods built on these representations typically use depth images to estimate uncertainty in the 3D reconstruction, enabling efficient and real-time viewpoint selection. However, these approaches typically produce low-fidelity 3D reconstructions, due to their strong dependence on the quality of depth images.
Recently, neural rendering methods such as Neural Radiance Fields (NeRF) Yu et al. (202T): Wang itt al.] (2021); Trevithick & Yang (202 I): Mildenhall et al. (2021) and 3D Gaussian Splatting (3DGS)
Kerbl et al. (2023): Szymanowiez et al. (2024); CCarataan et al. (2024); Chen et al. (2024) have gained significant attention for their strong performance in 3D object reconstruction. However, their effectiveness comes with the high computational cost, largely due to the large number of viewpoints required for training. To mitigate this issue, recent research has explored acceleration strategies such as few-shot training Jain et al. (2021), optimized ray sampling [i et al. (2023), and hybrid scene representations that combine continuous volumetric functions with voxel grids Cao et al. (2024).
Another promising direction is Active View Selection (AVS), which aims to identify and use only a
small subset of the most informative viewpoints. Most existing AVS methods Pan et al. (2022)
Ran et al. (2023); Lee et al. (2022); Yan et al. (2023); Zhan et al. (2022); Xue et al. (2024);
Sünderbauf et al. (2023); Hoffman et al. (2023); ;; Jin et al.] (2023) follow two-stage pipeline: they first train a neural rendering model using a limited set of views, then estimate the utility of
candidate viewpoints using uncertainty heuristics derived from the trained model. The next view is selected based on these heuristics. For instance, several AVS approaches Pan et al. (2022); Ran al. )2023); Lee et al. (2022): Yan et al. (2023); Zhan et al. (2022): : Xue et al. (2024); :) Yu et al. (2021) train a NeRF model on the current set of views. and then evaluate the uncertainty of each
candidate view. Some methods Lee al. (2022): : Zhan et al. (2022); Yan et al. (2023) estimate uncertainty by computing the entropy of opacity distributions along rays. Later approaches Pann jet al.] (2022): |Ran et al.] (2023); Xue et al.) (2024) incorporate pixel-level reconstruction quality by averaging color variance across sampled points along each ray. Xue et al. (2024) further enhances this by factoring in visibility, assigning higher uncertainty to occluded or previously unobserved regions. However, these methods require retraining the NeRF model after each new view is added, resulting in significant computational overhead and slow iterative training cycles. As a result, current NeRF-based AVS pipelines are impractical for applications with limited time computational resources. A group of existing approaches Jin et al. (2023); Smith et al. (022) utilize pre-trained multi-view reconstruction models avoid the costly retraining process. Specifically, (Smith et al.) (002)) employs a pre-trained reconstruction model at inference to predict occupancy from multiple input views, and then derives geometry- based uncertainty from the predicted occupancy. However, this is not an end-to-end pipeline: the model makes indirect predictions by first estimating occupancy and then converting it into uncertainty. This two-step procedure can be time-consuming, and errors in occupancy prediction may propagate and compound during uncertainty estimation. Recently, AVS methods have leveraged properties of 3D Gaussians from 3DGS models to estimate uncertainty. ActiveSplat [i et al. (2024) uses Gaussian variance to model scene density and selects viewpoints that maximize spatial coverage. ActiveGS Jin et al. (2025) and ActiveGAMER Chen et al. (2025) employ depth supervision and Gaussian density to estimate the number of currently occluded voxels that would be revealed from each candidate view. However, these methods primarily focus on geometry and overlook rich pixel-level cues such as color in the volumetric representations.
To address these limitations, another class of AVS approaches aims to improve computational efficiency by learning direct mapping from the current viewpoint to the next optimal one. This is typically achieved via supervised learning Mendoza et al.] (020). using ground-truth next-best viewpoints that maximize point cloud coverage. or through reinforcement learning Wang et al. (2024), where the reward is based on the increase in voxel or point cloud coverage after next view selection. However, these methods rely on fixed, discrete set of candidate viewpoints, which makes them rigid and limits their generalizability to novel viewpoints in new environments.
Here, we propose a novel AVS method called PUN (Peering into the UnkNowN), which consists of two components: (1) neural uncertainty map prediction and (2) next-best-view selection. First, we introduce UPNet (Uncertainty Prediction Network), a lightweight feedforward neural network that
2

---

Published as a conference paper at ICLR 2026
Which viewoiint most informative?
Contesian cocrdinate relative to the Viewporn
Uncertainty Map
90'
180
1.0
90
Selected siewpoinss
Minimal reconstraction error
180
180
180
viewpoint
0.8
270'
3D soane rendering (NoRF/ / 30BS)
Polar
Resonstruction
Uncertainty value 270
coordinate
relectre the
Neural uncertainty map Figure/ 1: (a) Illustration of Active Viewpoint Selection (AVS). The goal of AVS is to select the most informative viewpoints (red cones) from candidate views (gray cones) to minimize reconstruction enror between the ground-truth view and the novel view synthesized by rendering methods such as NeRF Mildenhall et al. (202I) or 3DGS Kerbl et al.) (2023). (b) Neural Uncertainty Map (UMap). For each selected viewpoint, coordinates, assigning uncertainty values to candidate viewpoints on a spherical surface. A viewpoint parameterized by azimuth & (0,360") (red) and elevation 0 [0 180") (blue), at fixed radius from the current view. The coordinate system is oriented with the z-axis pointing from the object center to the current view, the y-axis upward, and the x-axis to the right. A sample viewpoint is shown as a black dot, with uncertainty visualized by the colorbar. See Sec. for details.
(a) Active selection
viowpoint
method predicts UMap in polar
takes the current view as input and predicts neural uncertainty map. This map highlights regions of high ambiguity, enabling selection of informative views based solely on the current observation. UPNet is trained via supervised learning on ground-truth neural uncertainty maps, obtained by comparing single-view 3DGGS-synthesized images with ground-truth views using heuristic-based metrics. Compared to prior approaches, uncertainty maps offer higher interpretability by explicitly visualizing uncertainty across all candidate viewpoints. Since UPNet is lightweight and independent of past viewpoints, it eliminates the need for costly iterative neural rendering model retraining at test time, resulting in 400 times speedup and over 50% reduction in compute usage,
Next, PUN aggregates all the past predicted uncertainty maps, suppresses redundant views with low uncertainty, and selects the next viewpoint with highest uncertainty. Unlike prior methods Mendoza al. (2020); Wang et al. (2024) that rely on mappings over a fixed, discrete candidate set, PUN generalizes to arbitrary viewpoints.
We evaluate PUN on novel object instances from the same categories as the training set by training neural rendering model on the selected views and comparing the synthesized outputs to ground truth across pixel-level, semantic, and geometric metrics. PUN achieves reconstruction quality comparable to the all-views upper bound using only half as many views. This performance holds even when using neural rendering models different from those used to generate the neural uncertainty maps. Furthermore, PUN generalizes effectively to AVS for novel object categories and realistic scenes not seen during training. We summarize our key contributions:
1. We introduce Peering into the UnkNowN (PUN), novel AVS method comprising neural uncertainty map prediction and next next-best-view selection. Remarkably, PUN achieves reconstruction accuracy comparable to the all-views upper bound using only half many views.
2. We contribute large-scale Neural Uncertainty Map (NUM) dataset comprising 48 viewpoints and their corresponding neural uncertainty maps across 13 object categories, each with 100 3D object instances. The uncertainty maps are derived using heuristic-based metrics to highlight uncertain regions from single-view view synthesis.
3. Compared to competitive baselines, PUN is compute-efficient, highly interpretable, and generalizes well arbitrary viewpoints and novel object categories. Notably, the viewpoints selected by PUN consistently improve 3D reconstruction performance across different neural rendering models.
2 NEURAL UNCERTAINTY MAP (NUM) DATASET
We randomly select 13 object categories, such sofa, car, and airplane, each with 100 object instances, from the ShapeNet dataset Chang et al. (2013) to construct our Neural Uncertainty Map (NUM) dataset. See Fig. a for the NUM dataset generation pipeline and Fig. ST for the list of
3

---

Published as a conference paper at ICLR 2026
Uniformly sampled anchor viewpoints (py}
Current Vt
UPNet
UMap U/
Transtormer Encoder
Filter and Aggregate
UMaps
PSNR/SSII//PIIS/MMEE
Next viewpoint
Single-view view synthesis (NeRF/ / 3DGS)
Pt+1
t=T
(a) Creation of our NUM dataset
(b) Overview of our PUN method
Figure 2: (a) Pipeline for Creating the Neural Uncertainty Map (NUM) Dataset. Given a current viewpoint of 3D object, its fixed anchor viewpoints Py (red dots) defined around 3D object, with ground-truth images rendered in Blender. A single-view view synthesis method, Splatterlmage Szymanowicz et al. (2024) synthesizes novel views these anchors, and reconstruction errors (PSNR, SSIM, LPIPS, or MSE) are projected into polar coordinates to form the UMaps. (b) Overview of our proposed Peering into the Unknown (PUN) method for AVS. From the current view Ut, our transformer- based UPNet takes image patches as input and predicts an uncertainty map U. PUN integrates past UMaps (t = O to L, red triangles), filters redundant viewpoints, and selects the next best view wt+1 (blue triangles). This process repeats iteratively to guide exploration.
selected categories. For each object instance, uniformly sample 48 viewpoints and compute their corresponding neural uncertainty maps (UMaps) as described below. In total, our NUM dataset comprises 62400 pairs of viewpoints and their associated UMaps.
Neural Uncertainty Maps (UMaps). Analogous to a depth map estimation in computer vision, given an input viewpoint of 3D object, one can estimate pixel-wise uncertainty and predict UMap, as detailed below. In our NUM dataset, exhaustively generating UMaps for all possible input viewpoints is infeasible due to the vast combinatorial space. Therefore. for each 3D object instance, we generate UMaps for 48 selected viewpoints. Specifically, the ShapeNet objects, which are normalized and center-aligned, are directly imported into the Blender simulation (208) and the Hierarchical Equal Area isol Latitude Pixelization (HEALPix) algorithm Gorski et al. 2005) with resolution parameter Niidd = 2 is used to uniformly sample 48 viewpoints on a spherical surface centered at the object's origin, starting from the coordinate (0, 0, 2.73) meters in Blender.
Each of these viewpoints is used to train a neural rendering model for single-view view synthesis. The model can then synthesize novel views from unseen camera poses. We define each camera pose by a fixed radius r = 2.73 meters, an elevation angle 9 E [0 180%). and an azimuth angle [09, 160%]. The camera is always oriented toward the object center, and r 2.73 meters ensures the object is fully visible but not intersected in Blender. To evaluate the quality of synthesized views, reuse the same HEALPix method above generate 48 camera poses centered at the object's 's origin, starting from the location of the input viewpoint. In other words, the 48 camera poses for view synthesis transformed depending the pose of the input viewpoint, such that their relative positions on the spherical surface remain constant across all input viewpoints.
spherical surface
We name these 48 camera poses as our anchor set P and render ground-truth views in Blender. For each anchor camera pose Pj P where j E {1, 2.., 48), we also synthesize corresponding view using the trained neural rendering model. The reconstruction eиrоr between the ground-truth, and synthesized image is computed using various uncertainty metrics (described below), and this value becomes the uncertainty anchor point p, in the UMap. Finally, these anchor points are mapped to polar coordinates to form the UMap. See Fig. [b) for illustration of the camera pose setup.
2.....44,,
Single-view view synthesis backbones. In single-view view synthesis, neural rendering models can be broadly categorized into four types: NeRF-based methods Yu et al. (2021) l); Wang et al. Trevithick & Yang (022I, diffusion- based approaches Anciukevičius et al. (2023); Liu et al.
2021
2023 Watson et al. (2022) end-to-end Transformer-bseee models that take current view input
and synthesize novel views Hong et al. (2023): Tochilkin et al. 2024). and 3D Gaussian Splatting
4

---

Published as a conference paper at ICLR 2026
(3DGS)-based techniques Szymanowicz al. (2024): Charatan al. (2024); Chen al. (2024) In principle, any of these models could be used as backbone to synthesize views and compute UMaps. However, due to the high computational cost of view synthesis, training separate neural rendering model for each of the 48 viewpoints per object instance is impractical. To address this, adopt Splatter-Image Szymanowicz as our synthesis backbone for its high compute efficiency and accurate 3D object reconstruction. Built on 3D Gaussian Splatting, Splatter-Image uses U-Net Ronneberger et al. (2015) map an input image into one 3D Gaussian per pixel. In ablation studies (see Sec.5..1 we demonstrate that our PUN method, which trained on UMaps generated by Splatter-Iman Image. generalizes well and can effectively select next-best viewpoints for other neural rendering models.
al. (2024), pretrained on ShapeNet Chang al.] (2015),
Uncertainty measures. We use four uncertainty measures to quantify differences between synthesized and ground truth views: PSNR, SSIM, LPIPS, and MSE. PSNR (Peak Signal-to-Noise Ratio) measures pixel-wise fidelity and penalizes large intensity deviations. SSIM (Structural Similarity Index) captures structural similarity by accounting for luminance, contrast, and texture. LPIPS (Learned Perceptual Image Patch Similarity) is a perceptual metric based on deep feature representations from pretrained networks Krizhevsky al. (2017), reflecting semantic similarity. MSE (Mean Squared Error) computes average squared pixel differences and emphasizes raw intensity errors, though it may not align well with human perception.
Training, validation, and test data split. To train UPNet, we use data from 11 object categories in the NUM dataset, with 100 instances per category. Each category is divided into training, validation, and test sets with an 8:1:1 split by instance. In addition, 2 extra object categories from the NUM dataset are held out entirely from training and used solely as test data for evaluating UPNet.
3 OUR PEERING INTO THE UNKNOWN (PUN) METHOD
An overview of our PUN method is shown in Fig. 2 (b). At each timestep E f0, (0.1....,T) ,), given an input view I, from the current viewpoint us. PUN first predicts a UMap U, using a feedforward deep neural network, named the Uncertainty Map Prediction Network (UPNet). We then randomly sample a set of 512 candidate viewpoints Cj. where E 21.2.... 512]. Similar to the anchored camera poses Pt. each candidate viewpoint lies on spherical surface and parameterized by a fixed radius/ r = 2.73 meters, an elevation angle 0 E [0 180"). and an azimuth angle e E [0 360%). The aggregated uncertainty values for these candidates Cj are computed based on the sequence of UMaps observed so far, (Ui. U2..... ........, U.}. PUN then filters out candidate viewpoints with consistently low uncertainty and selects the one with the highest remaining uncertainty as the next best viewpoint.
3.1 UNCERTAINTY MAP PREDICTION NETWORK (UPNET)
Given the input view I at the current viewpoint it the UPNet of PUN predicts a UMap Ug E R which is 48- 18-dimensional vector representing uncertainty values corresponding to the 48 predefined camera poses in the anchor set Pt (see Sec. for the definition of UMaps in our NUM dataset). To train UPNet, we fine-tune all weights of a Vision Transformer (ViT) Dosovitskiy al. (020) pre-trained on ImageNet [eng et al. (2009) for image classification. Á fully connected layer is appended
the classification token output of the ViT to generate the predicted UMap U.. The network is supervised using mean squared error (MSE) loss between the predicted and ground-truth UMaps. In our NUM dataset, UMaps are defined using four different uncertainty measures (Sec.2) 2). For PUN, we use PSNR as the uncertainty measure to construct the ground-truth UMaps. An ablation study in Sec. 5.3 further investigates the impact of alternative uncertainty measures. See Sec[C for additional training and implementation details.
3.2 UNCERTAINTY VALUE INTERPOLATION AT THE CANDIDATE VIEWPOINTS
Given the uncertainty values of a UMap U, set of 48 anchor points P. on a spherical surface, apply the following interpolation method to estimate the uncertainty at a candidate viewpoint Cj- For simplicity, we omit the subscript / in the following paragraph.
First, we define the set of neighboring anchor points P P as those whose angular distance to C³ is within 30 degrees. The uncertainty value at C³ is then computed as a weighted sum of the uncertainty values at the neighboring anchors P, where the weights are determined by applying a softmax function to the negative angular distances, thus assigning higher weights to closer anchors:
5

---

Published as a conference paper at ICLR 2026
S5 UC. u and wi se Pgy where U refers to the uncertainty value at the
neighboring anchor point P3 on the UMap, and Oz is the angular distance between P3 and C.. For all past and current UMaps up timestep 4, we apply the interpolation method described above to estimate the uncertainty values each candidate viewpoint C",. For example, we denote by U the interpolated uncertainty from the UMap - at candidate viewpoint C
3.3 NEXT VIEWPOINT SELECTION
Given the uncertainty values up to timestep at each candidate viewpoint, we exclude redundant viewpoints whose uncertainty value falls below threshold of 0.1 any previous timestep up to This design is motivated by the observation that candidate viewpoints near previously selected viewpoints are more likely to have already been observed, resulting in lower uncertainty values; thus, they need not be explored again. See the ablation study in Sec. 5.3 for the effect of excluding redundant viewpoints.
Next, for the remaining candidate viewpoints, since each UMap only reflects the information from single input image timestep aggregate the uncertainty values each candidate viewpoint across timesteps to t by multiplying (lD all the interpolated uncertainty values at C.: 9++1 = arg maxe, I11..... Uf"". The candidate with the highest accumulated uncertainty is then selected as the next viewpoint 1/-1- In Sec. 5.3, we also explore alternative aggregation strategies.
4 EXPERIMENTS
For fair comparisons, we evaluate all AVS methods using the same experimental configurations. Each AVS method is allowed to select up 20 viewpoints. These selected viewpoints are passed to Blender Community (2018) to render 20 corresponding views at a resolution of 512 x 512, which then fed into the same backbone for 3D reconstruction. To avoid any confounding factors during evaluation, all 3D reconstruction backbones are trained from scratch. At each timestep, 512 candidate viewpoints are randomly sampled on a spherical surface of radius 2.73, and all the methods select a viewpoint from this sampled set.
Datasets. We evaluate AVS performance across six datasets using object instances from NUM, NeRFAssets Mildenhall et al. (202I). and MIP360 Barron et al. (2022). NUM-inst comprises unseen instances per category from the same 11 categories used to tram UPNet PUN, while NUM-cat contains 10 instances each from 2 novel categories not seen during training. NUM-light evaluates the same instances under varied lighting, retaining original illumination and adding a 1000W white point light at (2, -2, 3) m in Blender. NUM-cam-dist reduces the camera radius (r= 2,73/ 1.5 m) for NUM-cat instances, bringing objects closer and enlarging their projection (see select 4 of 8 instances (Lego, Drums, Hotdog, Materials) with complex geometry and occlusions. MIP360 contains 9 real-world scenes; we select 4 (Bicycle, Bonsai, Garden, Stump) and retain only views with the target object centered, yielding 97, 146, 93, and 63 views, respectively.
Fig. S2). From NeRFAssets,
Baselines. We include four competitive AVS baselines: WD al. (2022) estimates volumetric uncertainty by computing the entropy of the weight distribution of sampled points along each ray cast from the candidate viewpoint. A-NeRF Pan et al. (2022) models NeRF's color outputs as Gaussian distributions and estimates uncertainty based on their variances. NVF N et al. (2024) incorporates visibility into uncertainty estimation, assigning higher uncertainty to regions that invisible in the training views. Uniform employs Fibonacci sphere sampling to generate a fixed set of uniformly distributed candidate viewpoints over the viewing sphere. Upper Bound (Upper-bnd) trains the neural rendering models directly on the ground-truth rendered views corresponding to the evaluation set of 40 camera poses. These same 40 views are then used for evaluating novel view synthesis. Since the training and testing views are identical in this setting, the resulting performance represents an idcalized upper bound for novel view synthesis quality.
Evaluation of 3D reconstruction quality, We evaluate all AVS methods using the standard neural rendering model NeRF Xue et al. (2024). trained for 2,000 iterations on the 20 views selected by each method. In Sec. 55.1 we further vary the reconstruction backbone on NUM-3DGS to test whether AVS methods generalize across different backbones. For novel view synthesis evaluation on all datasets except the MIP360 dataset, randomly sample 40 camera poses uniformly distributed

---

Published as a conference paper at ICLR 2026
on the spherical surface surrounding the object. The same set of 40 poses is used for all test object instances and AVS methods to ensure consistent evaluation. For the MIP360 dataset, where 3D object instances are unavailable, we follow the protocol of Barron al. (2022) and evaluate using every 8th image in each object's s camera view sequence.
Given test object instance, a 3D reconstruction backbone trained on all the views selected by each AVS method synthesizes novel views. Following prior work Xue et al.] (2024), we evaluate the quality of these novel views using three sets of metrics from a fixed set of evaluation camera poses above or a fixed set of evaluation poses from every 8th view in all available views for the MIP360 dataset: Image Quality. We report Peak Signal-to- -o-isee Ratio (PSNR), Structural Similarity Index Measure (SSIM). Learned Perceptual Image Patch Similarity (LPIPS), and Mean Squared Error (MSE). which respectively measure image fidelity, structural similarity, perceptual differences, and pixel-wise reconstruction error. See Sec. 2 for more details. Mesh Quality. We evaluate aceuracy (Acc) and completion ratio (CR) as proposed in /uuaar al.] (202T) by comparing the reconstructed mesh-obtained from high-opacity points sampled from the trained NeRF Xue et al. the ground-truth mesh from ShapeNet Chang et al. (213)). Specifically, Acc (cm) the average distance from points on the reconstructed mesh to their nearest neighbors ground-truth mesh, while CR denotes the percentage of points on the reconstructed mesh whose nearest ground-truth points lie within em. These metrics not applicable to 3DGS [Kerbl et al.. (222)) as it does not explicitly produce mesh representations. Visual Coverage. We compute Visibility (Vis) and Visible Area (Vis. A.) for evaluation on visibility coverage. See Sec(C the detailed definition of all evaluation metrics.
the
5
RESULTS
5.1 OUR PUN ACHIEVES MOST ACCURATE 3D RECON. WITH HIGH COMPUTE EFFICIENCY
Our PUN outperforms all baselines and matches the upper bound. Tab. Ta) presents the 31> reconstruction results on NUM-inst. Our PUN consistently outperforms all baselines across all eight metrics, including image quality, mesh quality, and visual coverage. Notably, even NVF, which explicitly models visual visibility, performs worse than PUN. Although PUN is trained using UMaps derived solely from PSNR, it generalizes well beyond image fidelity, achieving strong performance on geometry- aware and visual coverage metrics. Remarkably, PÚN performs on par with the Upper-bnd, despite using only half the number of selected viewpoints.
Our PUN generalizes to novel object categories. To assess generalization, we evaluate all AVS methods on NUM-cat, as shown in Tab. Th). Consistent with the results on NUM-inst, our PUN achieves the best performance across all eight metrics, surpassing all baselines. As reported in Tab. S1 Uniform attains strong visual coverage owing to its even viewpoint distribution, but its performance
other metrics lags behind PUN. Notably, PUN matches Upper-bound while requiring only half the training views. Reconstruction quality of all AVS methods as a function of the number of input viewpoints is shown in Fig. S3 PUN consistently surpasses all baselines regardless of the number of input viewpoints in most cases.
The viewpoints selected by PUN are agnostic to the choice of reconstruction backhones. To test generalization across reconstruction backbones, we introduce the NUM-3DGS-recon dataset, which uses the same instances as NUM-cat but evaluates AVS methods with the 3D reconstruction backbone 3DGS Tancik et al.] (2023). Using the same selected views and default parameters for all AVS methods, 3DGS is trained for 30,000 iterations. As shown in Tab. (c), PUN consistently outperforms all baselines across all metrics. In addition, we compare our method with the SOTA baseline NVF using Binocular3DGSS Han et al. (2024). a few-shot reconstruction method, as the reconstruction backbone. As shown in Tab S2) our method achieves the best performance. Together with the NUM-cat results (see Tab. Tb)). these findings confirm that the viewpoints selected by PUN are broadly informative and remain agnostic to the reconstruction backbone.
PUN generalizes to various lighting conditions, camera distances, and realistic scenes with complex geometry and backgrounds. PUN demonstrates strong robustness and generalization in 3D reconstruction. Without retraining or fine-tuning, it outperforms baselines on NUM-light and NUM-cam-dist (Tab. [dd), showing robustness to changes in lighting and camera distance. also achieves the best performance on realistic scenes from NeRFAssets and MIP360 (Tab. 2 highlighting its ability to select informative views in complex environments.

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation on the AVS performance. We evaluate the performance of all AVS methods on NUM-inst, NUM-cat, NUM-3DGS-recom NUM-light and NUM-cam-dist. si and s2 refer to experiments on NUM-light and NUM-cam-diss Tab. [SI for the full results. Best is in bold and second best is underlined. For readability, MSE
Average results over runs are reported. See
values are scaled by 10s on NUM-3DGS- recon and NUM-cam-dist, and by 10 in other datasets.
Method
PSNRT
SSIM1
LPIPS]
MSEL|
Method
PSNRT
SSIMT LPIPS,
MSE.
na)novel instance
A-NeRF
32.71
0.982
0.031
8.19
(d)eny, (b)novel
category
A-NeRF
33.16
0.984
0.024
7.04
NVF
33.08
0.984
0.028
6.98
NVF
33.15
0.985
0.021
6.65
Ours
33.19
0.984
0.025
6.96
Ours
34.74
0.985
0.019
5.03
Upper-bnd
36.47
0.989
0.017
4.11
Upper-bnd
36,91
0.990
0.013
3.33
(G)3DGS recon.
WD
20.59
0.906
0.21
15.59
setting
NVF-s1
31.57
0.985
0.022
11.64
A-NeRF
26.22
0.948
0.12
7.50
Ours-sl
32.84
0.987
0.018
8.32
NVF
30.67
0.977
0.07
2.28
NVF-s2
27.34
0.961
0.050
3.38
Ours
36.71
0.990
0.03
0.40
Ours-s2
31.19
0.969
0.042
1.39
Input
Ground truut
Prediction
Input
Ground -rruh
Prediction
90'
180"
180
Cabinet
Car
uncertainty
high
low
(a) NUM-novel-instance
(b) NUM-novel-category
Figure 3: Visualization of ground-truth and predicted uncertainty maps by our PUN method.
We present two examples from: (a) NUM-inst and (b) NUM-cat. In each case, the input view I at the current viewpoint 2, is fed into UPNet to produce the predicted UMap. The corresponding ground-truth UMap is shown alongside for comparison. Both maps are min-max normalized to [0, 1] for visualization only. See the colorbar for uncertainty values and polar coordinates for the UMaps.
Our PUN is robust to training on UMaps generated by different novel view synthesis backbones. We use Smlatter-Image Szymanowicz et al. (2024) as the default synthesis backbone for generating UMaps, which are then used to train UPNet in PUN. To test sensitivity to alternative backbones, we replace Splatter-Image with NVF Xue et al.] (2024). Since NeRF synthesis is computationally intensive, we train UPNet in PUN on a subset of 50 airplane instances from the NUM dataset, denoting this variant as PUN-NeRF and the corresponding dataset as NeRF-NUM. We then evaluate PUN-NeRF and other AVS methods on 5 airplane instances, using 3DGS as the reconstruction backbone. As shown in Tab. SS3] PUN-NeRF achieves superior performance the NeRF-NUM dataset. These results confirm that our method remains robust for AVS, even when trained on UMaps generated by different synthesis backbone.
Our PUN predicts reasonable neural uncertainty maps. To quantitatively evaluate the UMap predictions of our UPNet in PUN, we present the scatter plot of the predicted uncertainty and the ground-truth uncertainty in Fig. S4 We compute the Pearson correlation between the ground-truth and predicted uncertainty maps on the test set. The strong correlation coefficient of 0.82 indicates that PUN can reliably predict accurate UMaps and approximate the underlying reconstruction landscape. Next, we present two visualization examples of predicted uncertainty maps by PUN on both NUM-inst and NUM-cat (one instance from each dataset) in Fig. 3 In (a), given side view of cabinet as input, the predicted uncertainty increases with elevation angle 0, represented radially in the polar coordinate. One would expect the uncertainty to also increase along the azimuth angle o as the camera rotates 180" counter-clockwise toward the cabinet's 's opposite side. Surprisingly, the uncertainty remains low in these regions likely due the cabinet's structural symmetry. UPNet may leverage this symmetry to confidently infer the appearance of the unseen side from the visible one. In (b), given a side view of car, we observe similar trend in the predicted uncertainty along @ and á as seen on NUM-inst. Notably, the UMap retains a degree of symmetry despite the object class being unseen during training. This consistency may stem from the inherent symmetry bias present in many household objects within the training dataset, which appears to generalize to novel object categories.
PUN is highly compute-efficient in next-view selection. Unlike competitive AVS baselines that retrain neural rendering models after each new view incurring heavy overhead and slow iterative
8

---

Published as a conference paper at ICLR 2026
More realistic and complex
Dataset
Method PSNR1 SSIM1
LPIPS. MSEJ
NVF
26.31
0.928
0.115
0.005
NeRFAssets
Ours
26.73
0.944
0.093
0.003
NVF
15.41
0.203
0.653
32.03
MTP360
Ours
17.49
0.294
0.545
19.98
ShapeNet NeRFAssets MIP360
Table 2:
Evaluation on NeRFAssets and MIP360. We report the average results over four instances from each of the two datasets. We refer the readers to Tab. SS for the full results and the detailed caption. Best is in bold.
Figure 4: Visualization of the datasets used for AVS evaluation. From left to right, the object instances exhibit increasing geometric complexity, occlusion, and background clutter.
Table 3: Ablation Analysis of Key Components in Our PUN Method. From left to right, we analyze: (a) different uncertainty metrics used to generate ground truth UMaps and train UPNet, (b) the effect of different next-viewpoint selection policies as illustrated in Sec. [..] and (c) the trade-off between instance diversity and viewpoint density in training samples. In (c), instance diversity refers to the number of object instances per category, while viewpoint density refers the number of viewpoint-UMap pairs per instance. Best is in bold and second best is underlined. MSE
values are scaled by 10 See Tab. S6 for full results in all metrics.
(a) Uncertainty metrics
(b) Viewpoint selection policy
(c) Divensty
(d) Anchor number
small-all
disableeali
Log. 32+al)
singletall
small-bs Hooe
small wiif
Cinll--ad udda
PSNR
(ours)
SSN
MSSE
(ours)
8W112
40
20:48
PSNR
37.4
36.4
36
35.0
37,4
36.9
36,8
36.9
36.9
36.3
35.7
37.0
36.8
36.8
35.4
36.0
36.1
training- uses UPNet, pretrained on our NUM dataset, to predict UMaps directly, avoiding on-the-fly retraining during inference. We benchmark all methods on identical hardware (4x NVIDIA RTX 3090 GPUs, 2x Intel(R) Xeon(R) Silver 4210 CPUs). In Tab. S4 PUN achieves up to 400x faster viewpoint selection while drastically reducing resource use: CPU (903% to 74%), RAM (4292 MB to 1870 MB), GPU utilization (30.6% to 0.3%), and GPU memory (8098 MB to 655 MB). Even with the additional minutes for reconstruction after selection, PUN cuts the total runtime from 175 to just 5.5 minutes.
5.2 UPNET CAPTURES VIEWPOINT COMPLEXITY
To better understand why UPNet can generalize to previously unseen object categories and remain effective when coupled with different reconstruction backbones, analyze what cues are implicitly captured by its learned representation. In particular, we investigate whether UPNet's reconstruction-errro-oooe based uncertainty measures correlate with basic geometric and texture complexity characteristics of the observed views, which are known to influence the difficulty of single-view reconstruction.
For each anchor viewpoint, we compute five standard view-complexity measures derived from the depth map and the RGB image: (1) depth gradient variance, (2) edge density, (3) color gradient, (4) color entropy, and (5) Laplacian energy. The first two geometric complexity measures computed from the depth map, while the remaining three are appearance complexity measures computed from the RGB image. See Sec. E for detailed deseriptions of these measures.
As shown in Tab.ST all metrics show moderate correlation, indicating that viewpoints with greater geometric variance, higher edge density, richer texture tend to exhibit higher reconstruction emror for the single-view synthesis backbone. We also visualize the scatter plot between the complexity metries and reconstruction етrоr metrics in Fig. S5 to provide better understanding of this correlation. By learning such patterns over a large number of diverse objects in ShapeNet, UPNet acquires generic notion of viewpoint difficulty that is largely domain-agnostic.
5.3 ABLATIONS ON OUR PUN REVEAL CRITICAL DESIGN INSIGHTS
Uncertainty metric ablation. In NUM, we define four uncertainty metrics PSNR, SSIM, LPIPS, and MSE- to quantify the difference between synthesized and ground truth views. By default, PUN is trained using ground truth UMaps based on PSNR. To evaluate the effect of different uncertainty measures, conducted ablation study by training UPNet with UMaps computed from each
9

---

Published as a conference paper at ICLR 2026
metric. From Tab. B (a), the choice of uncertainty metric has limited impact 3D reconstruction performance using NeRF at inference.
Ablations next viewpoint selection. In PUN, we adopt two key policies: removing redundant viewpoints with low uncertainty (small) and aggregating uncertainty across all past and current timesteps (all). To evaluate their impact, explore three variations for redundancy removal and two for uncertainty aggregation. Disabling redundancy removal entirely (disable) leads to a noticeable PSNR drop in 3D reconstruction (Tab. 3 b)), highlighting the importance of filtering redundant viewpoints. We also test two alternative strategies: (I) excluding candidates ranked among the 32 lowest uncertainty values across previous timesteps (top-32), and (2) removing candidates within 5 angular distance of previously selected viewpoints (single). Both underperform compared to the default small + all in PUN. For uncertainty aggregation, we compare last, which selects candidates with the highest uncertainty in the current UMap U, and diff, which favors candidates with the largest uncertainty difference relative to 5 neighbors U.. Both ignore prior information and yield lower performance. Our default all aggregation achieves the best results by leveraging temporal consistency across timesteps.
Ablation on the dataset size. We perform ablation study to examine the trade-off between instance diversity and viewpoint density when training UPNet in PUN. To control for total training data, keep the product of the number of categories, instances per category, and viewpoints per instance constant. As shown Tah. 3 (c), increasing instance diversity yields greater performance gains than increasing viewpoint density. This suggests that sampling more views from a single object often introduces redundant information, especially in symmetric objects, whereas incorporating a wider range of object instances offers more valuable data variation.
Ablation on the anchor number. We further investigate how the number of anchor viewpoints affects UMap quality and downstream view-selection performance. Increasing the anchor count improves angular coverage but also significantly raises the computational cost of NUM generation. As shown in Tab. 3 (d), increasing the anchor number from 12 48 provides a noticeable improvement across most metrics, indicating that denser angular sampling leads to more accurate UMaps. However, further increasing to 108 anchors brings only marginal gains, suggesting diminishing returns. Overall, 48 anchors provide good balance between reconstruction performance and dataset-generation cost, and we adopt this setting as our default configuration.
Ablation on the redundancy threshold. The redundancy threshold determines how many candidate viewpoints are removed due to redundancy. large threshold may discard too many viewpoints, while small threshold may preserve excessive redundant viewpoints. We evaluate thresholds from 0.0 to 0.3, and the results Tab. S66 (e) show that a threshold of 0.1 provides the best overall performance. Notably, the final reconstruction quality is not highly sensitive this threshold.
DISCUSSION
We propose Peering into the UnkNowN (PUN), an efficient AVS method guided by neural uncertainty maps predicted by lightweight feedforward network, UPNet. Unlike traditional approaches that derive uncertainty from radiance fields, UPNet directly maps a single input image to an uncertainty map over candidate viewpoints. By aggregating these maps over time and applying selection heuristics, PUN identifies informative views with minimal redundancy. Using only half the views of the upper bound, it achieves comparable 3D reconstruction quality while offering up to 400x speedup and 50% resource savings over strong AVS basclines. PUN also generalizes to unseen object categories, varied lighting, camera distances, and realistic scenes, and supports diverse neural rendering models without re-training. Despite its effectiveness, PUN relies on image reconstruction metries, which may not fully capture geometric fidelity. Incorporating geometry-awan metrics (e.g., mesh quality, visual coverage) into uncertainty prediction could further improve performance. Moreover, PUN currently assumes a spherical viewpoint distribution around a single object; future work will extend UPNet to handle arbitrary distributions in unconstrained 3D environments.
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its NRFF award NRF-NRFF15-20233 and Mengmi Zhang's Startup Grant from Nanyang Technological University, Singapore.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiftusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 12608-12618, 2023.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5470-5479, 2022.
Junyi Cao. Zhichao Li, Naiyan Wang, and Chao Ma. Lightningg nerf: Efficient hybrid scene representation for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. (6803-16809 IEEE, 2024.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 2015.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19457-19467, 2024.
Liyan Chen, Huangying Zhan, Kevin Chen, Xiangyu Xu, Qingan Yan, Changjiang Cai, and Yi Xu. Activegamer: Active gaussian mapping through efficient rendering. arXiv preprint arXiv:2501.0689) 06897, 2025.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pp. 370-386. Springer, 2024.
Blender Online Community. Blender a 3D modelling and rendering package Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL
blender. org
CI Connolly. The determination of next best views. In Proceedings. 1985 IEEE international conference on robotics and automation, volume 2, pp. 432-435. IEEE, 1985.
Anna Dai, Sotiris Papatheodorou, Nils Funk, Dimos Tzoumanikas, and Stefan Leutenegger.
Fast frontier-based information-driven autonomous exploration with mav. In 2020 IEEE international conference on robotics and automation (ICRA), pp. 9570-9576. IEEE, 2020.
Tung Dang, Christos Papachristos. and Kostas Alexis. Visual saliency-aware receding horizon 2018 IEEE international
autonomous exploration with application to aerial roboties. conference on robotics and automation (ICRA), pp. 2526-2533. IEEE, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce. 2009.
Zhiwei Ding, Xuezhe Ren, Erwan David, Melissa Vo. Gabriel Kreiman, and Mengmi Zhang. Efficient zero-shot visual search via target and context-aware transformer. arXiv preprint 2rXiv:2211.13470, 2022.
Christian Dornhege and Alexander Kleiner. A frontier-void-based approach for autonomous exploration in 3d. Advanced Robotics, 27(6) 2013.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 11 (2):177-138, 2010.
11

---

Published as a conference paper at ICLR 2026
Georgios Georgakis, Bernadette Bucher, Anton Arapin, Karl Schmeckpeper, Nikolai Matni, and Kostas Danilidis. Uinertainty-driive planner for exploration and navigation. In 2022 International Conference Robotics and Automation (ICRA). pp. 1tp..111100010 11295-11302. IEEE, 2022.
Krzysztof M Gorski, Eric Hivon, Anthony J Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. Healpix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622(2):759, 2005.
Shashi Kant Gupta, Mengmi Zhang, Chia-Chien Wu, Jeremy Wolfe, and Gabriel Kreiman. Visual search asymmetry: Deep nets and humans share similar inherent biases. Advances in neural information processing systems, 34:6946 6959, 2021.
Liang Han, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han, Binocular-guided 3d gaussian splatting with view consistency for sparse view synthesis. Advances in Neural Information Processing Systems, (1168995-686211 2024.
Shuangpeng Han and Mengmi Zhang- Unveiling ai's blind spots: An oracle for in-domain, out-of-domain, and adversarial errors. arXiv preprint arXiv:2410.02386 2024.
Matthew D Hoffman, Tuan Anh Le, Pavel Sountsov, Christopher Suter, Ben Lee, Vikash K Mansinghka, and Rif Saurous. Probnerf: Uncertainty- -aware inference of 3d shapes from 2d images. In International Conference Artificial Intelligence and Statistics, pp. 10425-10444. PMLR, 2023.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprini SKGi:23311.04000 2023.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5885-5894 58945,942222 2021.
Yanhao Jia, Ji Xie, S Jivaganesh, Hao Li, Xu Wu, and Mengmi Zhang. Seeing sound, hearing sight: Uncovering modality bias and conflict of ai models in sound localization. arXiv preprint arXiv:2505.11217. 2025.
Zhaoyin Jia, Yao-Jen Chang, and Tsuhan Chen. Active view selection for object and pose recognition. In 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 641-648. IEEE, 2009.
Liren Jin, Xieyuanli Chen, Julius Rückin, and Marija Popović. Neu-nbv: Next best view planning using uncertainty estimation in image-based neural rendering. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11305-11313 IEEE, 2023.
Liren Jin, Xingguang Zhong, Yue Pan, Jens Behley, Cyrill Stachniss, and Marija Popović. Activegs: Active scene reconstruction using gaussian splatting. IEEE Robotics and Automation Letters, 2025.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
Naitik Khandelwal, Xiao Liu, and Mengmi Zhang. Adaptive visual scene understanding: incremental scene graph generation. arXiv preprint arXiv:2110.01636, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Soomin Lee, Le Chen, Jiahao Wang, Alexander Liniger, Suryansh Kumar, and Fisher Yu. Uncertainty guided policy for active robotic reconstruction using neural radiance fields. IEEE Robotics and Automation Letters, 7(4): 12070-177772 2022.
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. Nerface: Efficient sampling accelerates nerfs. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 18537-18546, 2023.
12

---

Published as a conference paper at ICLR 2026
Yuetao Li, Zijia Kuang, Ting Li, Guyue Zhou, Shaohui Zhang, and Zike Yan. Activesplat: High-fidelity scene reconstruction through active gaussian splatting. arXiv preprint arXiv:2410.21955, 2024.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview conssssttnnnn images from a single-view image. arXiv preprint M3Ki::2309.0355,, 2023.
lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXi0:1711.05101, 2017.
Jun Lv, Yunhai Feng, Cheng Zhang, Shuang Zhao, Lin Shao, and Cewu Lu. Sam-rl: Sensing-ave aware model-based reinforcement learning via differentiable physics-based simulation and rendering. The International Journal of Robotics Research, pp. 027836492412846533 2023.
Miguel Mendoza, Irving Hisquez-Gomez Hind Taud, L Enrique Sucar, and Carolina Reta. Supervised learning of the next-best-view for 3d object reconstruction. Pultern Recognition Letters, 133:224-231, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robotics and Automation Letters, 4(2):610-617, 2019.
Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. uncertainty estimation. In European Conference on Computer Vision, pp. 230-246. Springer, 2022.
Activenerf: Learning where to see with
Richard Pito. A solution to the next best view problem for automated surface acquisition. 1EEE Transactions on pattern analysis and machine intelligence, 211100:1016-1030, 1999.
Santhosh K Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for efficient exploration and navigation. In Computer EisionEECCy 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part V 16. pp. 400-418. Springer, 2020.
Yunlong Ran, Jing Zeng, Shibo He. Jiming Chen, Lincheng Li, Yingfeng Chen, Gimhee Lee, and Qi Ye. Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations. IEEE Robotics and Automation Letters, 8(2):1 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted MnterVention-MICCA/ 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Vítor Sequeira, JoaoG M Goncalves, and M Isabel Ribeiro. Active view selection for efficient 3d scene reconstruction. In Proceedings of 13th International Conference on Pattern Recognition, volume 1, pp. 815-819. IEEE, 1996.
Jacopo Serafin, Maurilio CICCO, TAIGO Bonanni, Giorgio Grisetti, Luca Tocchi, Daniele Nardi, Cyril Stachniss, Vittorio Amos Ziparo, al. Robots for exploration, digital preservation and visualization archeological sites. Artificial intelligence for cultural heritage, pp. 121-140. Cambridge Scholars Publishing, 2016.
Edward J Smith, Michal Drozdzal, Derek Nowrouzezahrai, David Meger, and Adriana Cameero-Sorino.. Uncertainty-driven active vision for implicit scene reconstruction. arXiv preprint ar
2022.
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6229-6238, 2021.
13

---

Published as a conference paper at ICLR 2026
Niko Sünderhauf, Jad Chou-Chakra, and Dimity Miller. Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2370-9376, IEEE, 2023.
Stanislaw Szymanowicz, Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10208-10217, 2024.
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, al. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 conference proceedings, pp. 1-12, 2023.
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from single image. arXiv preprint vrXiv:240300215/2 2024.
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 15182-15192, 2021.
Bo Wang, Dingwei Tan, Yen-Ling Kuo, Zhaowei Sun, Jeremy M Wolfe, Tat-Jen Cham, and Mengmi Zhang. Gazing at rewards: Eye movements as a lens into human and ai decision-making in hybrid visual foraging. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp-
14810-14823, 2025.
Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2021.
Tao Wang, Weibin Xi, Yong Cheng, Hao Han, and Yang Yang. RI-nbv: deep reinforcement learning based next-best-view method for unknown object reconstruction. Pattern Recognition Letters, 184:1-6, 2024.
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho. Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628, 2022.
Shangjie Xue, Jesse Dill, Pranay Mathur, Frank Dellaert, Panagiotis Tsiotra, and Danfei Xu. Neural visibility field for uncertainty-driven active mapping. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18122-18132. 2024.
Dongyu Yan, Jianheng Liu, Fengyu Quan, Haoyao Chen, and Mengmeng Fu. Active implicit object reconstruction using umcertainty-guice next-best-view optimization. IEEE Robotics and Automation Letters, 8( 8(10):6355-644) 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4578 4587, 2021.
Huangying Zhan, Jiyang Zheng, Yi Xu, Ian Reid, and Hamid Rezatofighi. Activermap: Radiance field for active mapping and planning, arXiv preprint urXiv:2211.12653. 2022.
Mengmi Zhang. Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Zhao, and Gabriel Kreiman. Finding any waldo with zero-shot invariant and efficient visual search. Nature communications, 911):373,
2018a.
Mengmi Zhang, Keng Teck Ma, Shih-Chema Yen, Joo Hwee Lim, Qi Zhao. and Jiashi Feng. Egocentric spatial memory. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 137-144. IEEE, 2018b.
14

---

Published as a conference paper at ICLR 2026
Mengmi Zhang, Marcelo Armendariz, Will Xiao, Olivia Rose, Katarina Bendtz, Margaret Livingstone, Carlos Ponce, and Gabriel Kreiman. Look twice: A generalist computational model predicts return fixations across tasks and species. PLoS computational biology, 18( :11110c1,,, 2022.
Zhengquan Zhang and Feng Xu. An overview of the free energy principle and related research. Neural Computation, 36(5):963-101,, 2024.
15

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models were used to assist in polishing the writing of this manuscript.
B REPRODUCIBILITY STATEMENT
All code, models, and datasets are
available at https://github.com
ZhangLab DewpNeeroCoogabb//.......
C ADDITIONAL TRAINING AND IMPLEMENTATION DETAILS
Training details. UPNet is trained using the Adam W optimizer Loshchilov & Hutter (2117) with learning rate of 1 x 10-4, batch size of 32, for 100 epochs on single NVIDIA RTX A6000 GPU. The original input images of size 248 248 are center-croppet to 224 x 224 and normalized using mean and standard deviation of 0.5 per channel. We conduct three runs per experiment with different random seeds, following the data splits introduced in Sec.
Evaluation Metrics. Given a test object instance, 3D reconstruction backbone trained all the views selected by each AVS method synthesizes novel views. Following prior work Xue et al.) (0224). we evaluate the quality of these novel views using three sets of metrics: Image Quality, We report Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM). Learned Perceptual Image Patch Similarity (LPIPS), and Mean Squared Error (MSE), which respectively measure image fidelity, structural similarity, perceptual differences, and pixel-wise reconstruction error. See Sec. / for more details. Mesh Quality. We evaluate accuracy (Ace) and completion ratio (CR) as proposed in Sucar et al. (202T) by comparing the reconstructed mesh-obtained from high-opacity points sampled from the trained NeRF Xue et al. (2024)agaiin) the ground-truth mesh from ShapeNet [Chagg al. (2015). Specifically, Acc (cm) is the average distance from points on the reconstructed mesh to their nearest neighbors on the ground-truth mesh while CR denotes the percentage of points the reconstructed mesh whose nearest ground-truth points lic within 5 cm. These metrics are not applicable to 3DGS Kerbl et al. (2023) as it does not explicitly produce mesh representations. Visual Coverage. We compute Visibility (Vis) the proportion of ground-truth mesh faces that are directly visible by intersecting a ray from at least one selected viewpoint without occlusion. Visible Area (Vis. A.) measures the total surface area of these visible faces relative the entire surface area of the ground-truth mesh. Notably, both metrics are independent of the neural rendering models and solely depend on the selected viewpoints.
Train dataset of UP-Net
sofa
airplane
NUM-inst
lamp
telephone
NUM-cat
vessel
loudspeaker
cabinet
categories
table
display
bench
rifle
chair
car
90 instances
10 instances
Figure S1: The data distribution of our NUM dataset. In total, we select 13 categories from the ShapeNet dataset Chang et al.) (2015), with 100 object instances per category used to generate our NUM dataset. The blue regions indicate the training dataset for our UPNet, consisting of 90 instances from each of 11 categories. The yellow regions represent the instances in NUM-inst. The green regions show the instances in NUM-cat.
16

---

Published as a conference paper at ICLR 2026
D DETAIL EXPERIMENT RESULTS
Table S1: Evaluation on the view selection performance of PUN. We evaluate the performance of PUN on NUM-inst, NUM-cat, NUM-3DGS-recon, NUM-light and NUM-cam-dist, sl and s2 separately refers to experiments on NUM-light and NUM-cam-dit. Average results over runs are reported, with standard deviations shown in the brackets. For the experiment on NUM-3DGS-recon. since we use 3DGS as the reconstruction backbone, it is not possible to extract predicted mesh to compute the mesh quality metric, which is therefore denoted by "_" Best is bold and second best is underlined. For readability, MSE values are scaled by 103 in the experiment NUM-3DGGS-rcon and NUM-cam-dist, and by 10 in other experiments.
Evaluation setting
Method
PSNRT SSIM LPIPS., MSEL Acc.t CR
Vis.1
Vis. A.t
(a) Evaluation NUM-inst
31.92
0.979 0.035
8.55
0.36
0.14
0.33
0.72
WD
(0.34)
(0.001)
(0.002)
(0.71)
(0.01)
(0.01)
(0.010)
(0.011)
32.71
0.982
0.031
8.19
0.36
0.16
0.37
0.77
A-NeRF
(0.41)
(0.001) (0.003)
(0.54)
(0.01)
(0.01) (0.014) (0.006)
33.08
0.984
0.028
6.98
0.36
0.18
0.39
0.79
NVF
(0.44) (0.001) (0.002)
(0.72)
(0.01)
(0.01) (0.013) (0.009)
novel instance
31.14
0.968
0.049
14.93
0.35
0.17
0.41
0.81
Uniform
(0.66)
(0.002) (0.004)
(1.85)
(0.01)
(0.01) (0.016) (0.014)
33.19
0.984
0.025
6.96
0.35
0.18
0.41
0.82
Ours
(1.74)
(0.005) (0.004)
(0.24)
(0.01)
(0.01) (0.018) (0.010)
36.47
0.989
0.017
4.11
0.35
0.17
0.49
0.85
Upper-bnd
(0.67)
(0.001) (0.001)
(0.05)
(0.01)
(0.01)
(0.018)
(0.009)
(b) Evaluation on NUM-cat
31.93
0.980
0.029
8.34
0.35
0.14
0.30
0.67
WD
(2.17)
(0.006)
(0.013)
(0.44)
(0.09)
(0.05)
(0.066)
(0.023)
33.16
0.984
0.024
7.04
0.35
0.15
0.34
0.72
A-NeRF
(2.28)
(0.006) (0.013)
(0.32)
(0.09)
(0.04) (0.088) (0.007)
33.15
0.985
0.021
6.65
0.35
0.19
0.35
0.75
NVF
(2.42)
(0.005) (0.010)
(0.40)
(0.09)
(0.06) (0.084) (0.024)
novel category
31.62
0.972
0.037
11.59
0.34
0.18
0.37
0.78
Uniform
(2.91)
(0.010) (0.020)
(8.94)
(0.07)
(0.06) (0.100) (0.040)
34.74
0.985
0.019
5,03
0.34
0.18
0.37
0.78
Ours
(2.81)
(0.005)
(0.006)
(0.36)
(0.07) (0.05) (0.097)
(0.037)
36.91
0.990
0.013
3.33
0.34
0.17
0.44
0.81
Upper-bnd
(2.56)
(0.003)
(0.006)
(0.21)
(0.07) (0.04) (0.111)
(0.026)
(c) Evaluation on NUM-3DGS-recon
20.59
0.906
0.21
15.59
WD
(1.47)
(0.013)
(0.03)
(1.88)
26.22
0.948
0.12
7.50
A-NeRF
(2.47)
(0.018)
(0.03)
(3,56)
3DGS
30.67
0.977
0.07
2.28
NVF
(2.11)
(0.007)
(0.02)
(1.63)
36.71
0.990
0.03
0.40
Ours
(1.17)
(0.002)
(0.01)
(0.15)
(d) Evaluation on NUM-light and NUM-cam-dist
31.57
0.985
0.02
11.64
0.34
0.19
0.44
0.80
NVF
Point lighting
(3.13)
(0.007)
(0.013)
(10.08)
(0.08)
(0.07)
(0.173)
(0.04)
32.84
0.987
0.02
8.32
0.31
0.18
0.43
0.79
Ours
(3.23)
(0.007)
(0.010)
(7.11)
(0.08)
(0.07) (0.169)
(0.064)
27.34
0.961
0.05
3.38
0.38
0.18
0.42
0.77
NVF
(3.47)
(0.020)
(0.027)
(3.35)
(0.10)
(0.08) (0.094)
(0.003)
Camera distance
31.19
0.969
0.04
1.39
0.36
0.19
0.43
0.81
Ours
(3.06)
(0.013)
(0.023)
(1.07)
(0.07)
(0.05) (0.108)
(0.032)
17

---

Published as a conference paper at ICLR 2026
Table S2: Evaluation of AVS Methods using Binocular3DGS the reconstruction backbone. Best is in bold. MSE values are scaled by 10* for readability.
Method
PSNRT
SSIMI
LPIPSL
MSR
NVF
41.17
0.992
0.012
12.72
ours
43.78
0.996
0.007
6.15
NUM-cat
NUM-light
NUM-cam-dist
Figure S2: Visualization of NUM-light and NUM-cam-dist. We select chair instance to illustrate how different environment settings affect the input view. In NUM-light, the added point light source results in more pronounced shadows caused by occlusion from the chair back. NUM-cam-dist, the reduced camera distance makes the object appear larger in the input view.
E FURTHER DISCUSSION
UPNet learns a generic notion of viewpoint diffficulty. To better understand why UPNet can generalize previously unseen object categories and remain effective when coupled with different reconstruction backbones, we analyze what cues are implicitly captured by its learned representation. particular, we investigate whether UPNet's reconstruction-error -based supervision correlates with basic geometric and texture complexity characteristies of the observed views, which are known influence the difficulty of single-view reconstruction.
For each anchor viewpoint, we compute five standard view-complexity measures derived from the depth map and RGB image:(I)depp)) gradient variance - the variance of the depth-gradient magnitude, where the gradient is computed using Sobel filters. This captures geometric relief, surface curvature changes, and depth discontinuities; (2)edge density - the ratio of edge pixels in the depth map, obtained by applying Canny edge detector. It reflects the amount of geometric structure visible from the viewpoint; (3)color gradient the mean magnitude of image-space color gradients (Sobel-based), measuring the strength and density of texture variations; (4)color entropy - the entropy of the RGB color histogram, quantifying the diversity and complexity of the appearance distribution; (5)laplacian energy the average absolute response of the Laplacian operator on the RGB image, capturing high-frequency texture, fine details, and sharp edges.
This also explains that predicting the reconstruction error of single-view 3DGS model is valid and effective proxy for guiding multi-view NeRF reconstruction. Reconstruction difficulty largely determined by the geometry and visibility structure of the object, rather than the specific reconstruction backbone. Viewpoints that are occluded or geometrically complex tend to produce higher errors for any reconstruction model midel--snggleview or multi-view--beaue ultimately optimize the same objective of minimizing view-dependent reconstruction error as shown in S7 As a result, the error patterns produced by a vingle-iiww 3DGS predictor reflect general notion of "ieewpoitt difficulty" that is shared across model classes.
such models
By training UPNet to approximate this error landscape across many objects, the network learns to identify viewpoints that are inherently hard or easy to reconstruct, independent of the downstream backbone.
18

---

Published as a conference paper at ICLR 2026
140
unform
WVF
014
Number Views
Number Views
Number of Siarer
Figure S3: Visualization of reconstruction performance function of the number of views. We present the trends of PSNR, accuracy, and visibility with respect to the number of selected views on NUM-cat, serving representative metries from each of the three category of metrics. The horizontal axis indicates the number of selected views, while the vertical axis shows the corresponding metric values.
Table S3: Evaluation of AVS Methods NeRF-NUM. Best is in bold and second best is underlined. MSE values are scaled by 10x for readability.
Method
PSNRT
SSIM
LPIPS
MSEJ
WD
18.92
0.895
0.270
19.09
A-NeRF
21.96
0.921
0.198
13.94
NVF
26.40
0.958
0.122
6.25
PUN-NeRF
35.69
0.991
0.049
0.51
27.5
25.0
22.5
Urcettinng
20.0
17.5
15.0
GT Unceriainty
Figure S4: Scatter plot comparing predicted uncertainty with the ground-truth uncertainty. Uncertainty points are sampled from the test set. The red line denotes the fitted linear trend.
19

---

Published as a conference paper at ICLR 2026
Table S4: Processing Time and Resource Comparison. We report the average time, and computing resources (from left to right: CPU usage, RAM, GPU usage, and GPU memory) required to select 20 viewpoints for each AVS method during inference. Average results over 3 runs are reported, with standard deviations shown in brackets. Best is bold and the second best underlined.
Time (min)
CPU
RAM (MB)
GPU
GPU
Method
(%)
(%)
(MB)
198
630
4354
28.52
8144
WD
(19)
(68)
(52)
(1.82)
(271)
184
678
4308
29.60
8043
A-NeRF
(13)
(104)
(114)
(1.95)
(97)
175
903
4292
30.59
8098
NVF
(19)
(86)
(54)
(2.09)
(38)
0.5
80
1870
0.28
655
Ours
(0.13)
(14)
(250)
(0.06)
(70)
Table S5: Evaluation NeRFAssets and MTP360 dataset. We report the average results four instances from each of the two datasets. For the MIP360 dataset, since ground-truth geometry available, the mesh quality and visual coverage metrics cannot be computed and are therefore denoted by "_". Best is in bold. MSE values are scaled by 103 for the experiment on MIP360 dataset and by 10* for the experiments on NeRFAssets.
NeRFAssets NVF
PSNRT
SSIM
LPIPS]
MSEI
Acc.L
CR
Vis.1
Vis. A.f
26.31
0.928
0.115
0.005
0.294
0,51
0.457
0.64
Ours
26.73
0.944
0.093
0.003
0.291
0.47
0.424
0.62
MIP360 NVF
PSNRT
SSIM
LPIPS
MSEJ
Acc.L
CRT
Vis.1
Vis. A.t
15.41
0,203
0.653
32.03
Ours
17.49
0.294
0.545
19.98
PSNR
SSIM
MSE
LPIPS
Depth gradd var
Edge density
Color glad
Color eolro_entropy
Laplacian energy
Figure S5: Scatter plots illustrating the relationship between reconstruction error and view-complexity indicators. Each subplot shows one reconstruction-errrr metric (PSNR, SSIM, MSE, or LPIPS) plotted against one of the five complexity measures (depth gradient variance, edge density, color gradient, color entropy, laplacian energy). Blue dots represent sampled viewpoints from the test set, and the red line denotes the fitted linear correlation trend for that metric pair.
20

---

Published as a conference paper at ICLR 2026
Table S6: Ablation experiment on key components in PUN, including the different viewpoint selection policies, different uncertainty metrics used for guiding the selection, and instance diversity and viewpoint density in the UPNet Training dataset. Instance diversity refers to the number of instances per category the training set, while viewpoint density denotes samples (i.e., input image-UMap pairs) per instance. Bold indicates the best performance for each metric, while underline indicates second best. (MSE values are scaled by 10a for readability.)
number data
(a) Ablation on different uncertainly metries
metric
PSNR
SSTM t
LPIPS
MSE
Acc. 4
CR
Vis.
Vis. A.
PSNR (ours)
37.41
0.990
0.015
2.50
0.37
0.16
0.37
0.74
SSIM
36.37
0.989
0.016
2.90
0.37
0.18
0.36
0.73
LPIPS
36.09
0.989
0.015
3.23
0.38
0.17
0.35
0.72
MSE
34.95
0,870
0.017
4.11
0.36
0.14
0.33
0.68
(b)
Ablation on different viewpoint selection policy
policy
PSNR
SSIM f
LPIPS
MSE +
Acc. 4
CR 1
Vis.
Vis. A.
small+all (ours)
37.41
0.990
0.015
2.50
0.37
0.16
0.37
0.74
disablc+all
36.85
0.989
0.014
2.38
0.37
0.16
0.37
0.73
top-32+all
36.80
0.990
0.013
3.40
0.37
0.16
0.37
0.73
single+all
36.88
0.989
0.014
2.59
0.37
0.17
0.37
0.74
diable+last
35.82
0.989
0.016
3.44
0.36
0.16
0.36
0.70
small+last
36.94
0.990
0.014
2,51
0.36
0.16
0.36
0.72
lup-32+last
37.00
0,990
0.013
2.55
0.36
0.17
0.37
0.72
single+last
36.16
0.989
0.015
3.12
0.35
0.17
0.36
0.72
diable+diff
31.22
0,980
0.027
0.35
0.14
0.30
0.64
small+diff
36.27
0.989
0.015
3.36
0.35
0.16
0.36
0.72
top-32-diff
33.08
0.983
0.023
7.50
0.33
0.17
0.32
0.67
single+diff
34.71
0,987
0.017
4.00
0.35
0.17
0.34
0.71
small+add
35.68
0.985
0.021
3.74
0.37
0.15
0.36
0.72
(c) Ablation on different instance diversity and viewpoint sample density
diversity/
PSNR
SSIM t LPIPS
MSE
Acc.
CR
Vis. 1
Vis. A. 1
density
80/12
36.96
0.990
0.014
2.97
0.37
0.17
0.37
0.73
40/24
36.76
0.989
0.014
2.69
0.37
0.19
0.36
0.73
20/48
36.82
0.990
0.014
2.68
0.37
0.17
0.37
0.74
(d) Ablation on different anchor numbers
anchor number
PSNR
SSIM f
LPIPS
MSE +
Acc. 4
CR
Vis. 1
Vis. A.
12
35.37
0.992
0.014
3.01
0.33
0.19
0.18
0.66
48
35.97
0.992
0.014
2.78
0.34
0.17
0.17
0.64
108
36.13
0.993
0.013
2.60
0.34
0.17
0.18
0.68
(e) Ablation on different redundancy threshold
redundancy threshold
PSNR
SSIM f
LPIPS
MSE 4
Acc. 4
CR
Vis. 1
Vis. A. 1
0.0
35.30
0.986
0.019
4.51
0.37
0.15
0.37
0.72
0.1
36.04
0.986
0.020
4.75
0.37
0.15
0.36
0.73
0.2
35.70
0.986
0.020
5.19
0.37
0.15
0.37
0.73
0.3
35.81
0.985
0.021
4.13
0.36
0.14
0.36
0.73
Table S7: Correlation between basic geometric/texture complexity indicators and reconstruction enror metrics. Negative values ({) indicate that higher complexity correlates with lower PSNR/SSIM (worse reconstruction), while positive values (f) indicate correlation with higher MSE/LPIPS (higher error).
PSNR
SSIM
MSE
LPIPS
depth.grad_va
0.36 4
-0.40
0.28
0.28
edge_density
-0.35 1
0..1
0.26 1
0.29
color grad
-0.60 4
-0.60
0.56
0.43
color entropy
-0.50 1
-0.99
1
0.46 T
0.48 1
laplacian _nneryy
0.56
0.38
0.53
0.25
21


Reviews:
[{"review_id": "zRY9kzCkBC", "reviewer": "Reviewer_Wvnw", "summary": "The paper tackles AVS for 3D reconstruction by predicting a neural uncertainty map from a single current view. A lightweight feed-forward network, UPNet, takes one image and outputs uncertainty values over all candidate viewpoints. Using the selected views, standard neural rendering models (NeRF/3DGS) are trained and evaluated against competing AVS methods. The approach attains comparable reconstruction accuracy while dramatically reducing computational cost with up to 400× speedup and >50% lower CPU/RAM/GPU usage. It can also generalize to novel object categories without extra training.", "questions": "1. In the experimental setup, the authors removed real-scene views that are not centered on the object. In practical applications, what happens if the initial view is not oriented toward the object center?\n\n2. The UPNet gives sampling quality at 48 uniformly distributed positions on the sphere. However, in real scenes, a full 360-degree range of view may not be available, for example, some of the views may be under the ground (MipNeRF360 dataset). If the next best view is predicted in such a region, how should it be handled?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "This idea is very interesting: training a feed-forward uncertainty prediction network on a self-constructed dataset with ground truth supervision for AVS task. This can greatly reduce the computational resources and time required for next-best-view selection of AVS task. Experiments show that, compared with baseline AVS methods, the proposed approach exhibits clear improvements.", "weaknesses": "This paper proposes an interesting and effective method for the AVS task. However, my main concerns are the experimental evaluation and the value of this task. I find it hard to imagine a scenario where, given a single input view, the system should output where the next view should be obtained. For robotic applications, if it has the ability to move from one viewpoint to another, then it can capture dense views at arbitrary positions. For multi-view reconstruction, all ground truth captured views are already provided, and there is no need to select a subset for training.\n\nIf the authors claim that the goal of this task is to reduce the number of training views so as to reduce training time while maintaining quality, then I believe the evaluation in this work is incomplete. At a minimum, it should include comparisons of rendering quality and training resources with the following: (1) 3DGS trained on all available views. (2) pipelines of sparse-view NVS 3DGS (e.g. FSGS, Binocular3DGS) based on the selected small set of views.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gd8pPL1J8l", "reviewer": "Reviewer_3VjJ", "summary": "The paper tackles next-best-view selection for 3D reconstruction. The authors fine-tune a ViT to predict, from a single input image, a reconstruction uncertainty/score map. Supervision for fine-tuning is derived from PSNR, SSIM, LPIPS, or MSE computed on outputs of Splatter-Image, a monocular 3D reconstruction model; training data is curated from ShapeNet. At inference, the next best view is selected sequentially: uncertainties at each candidate view are interpolated from a fixed set of spherical anchors and aggregated across timesteps. Experiments on synthetic and real scenes show the method achieves comparable reconstruction quality with fewer views and lower computation, and the selection generalizes to different reconstruction backbones (e.g., 3DGS).", "questions": "- First, please refer to the weakness.\n- Second, are there cases where high uncertainty ≠ reconstruction gain?\n- Third, does the model has explicit pose conditioning and why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: Next-best-view selection is a classic problem in 3D vision; the paper’s backbone-agnostic approach—predicting a feed-forward uncertainty/score map for view selection—is a novel angle within this space.\n\n* Quality:\n\n  * The solution is simple and effective.\n  * The evaluation is comprehensive, covering both synthetic and real scenes, with robustness tests (e.g., lighting and distance). Generalization to multiple reconstruction backbones is also tested and supports the claim.\n\n* Clarity:\n\n  * The paper is well organized and clearly written, especially in framing the problem and motivation.\n  * The proposed method is introduced clearly, with the selection pipeline easy to follow.\n\n* Significance:\n\n  * The approach improves efficiency of existing 3D reconstruction pipelines while maintaining comparable performance.\n  * Backbone agnosticism enables plug-and-play use in arbitrary pipelines.\n  * The method’s simplicity makes it easy to extend for follow-up research.", "weaknesses": "* Uncertainty definition: The paper uses PSNR/SSIM/LPIPS/MSE as “uncertainty” labels; all are photometric and ignore geometry quality. This limits relevance for downstream tasks that depend on accurate shape.\n\n* Anchor design lacks justification: The anchor layout (48 HEALPix points) is fixed without analysis; it is unclear whether performance is bottlenecked by anchor density or discretization choice.\n\n* Candidate sampling is unclear: “Randomly sample 512 candidates” is under-specified (sphere-uniform vs. azimuth/elevation uniform vs. more sophisticated sampling). The chosen distribution can materially affect results.\n\n* Scope beyond object-centric canonical sphere: Training/evaluation assume canonical camera shells around single objects; performance and assumptions may not transfer to indoor/outdoor scenes with general camera placement, depth variation, and occlusions.\n\n* Single-image conditioning vs. relative view dependency: The ViT takes only one image, yet the output implicitly depends on the relative pose between the input view and the 48 anchors; the current design does not encode this dependency explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TjodcmIMqK", "reviewer": "Reviewer_yuxC", "summary": "The paper proposes a method to address the high computational cost of Active View Selection (AVS) for neural rendering methods. Traditional AVS methods for these models are slow because they require retraining the rendering model at each step to estimate uncertainty.\nThe paper proposes Peering into the UnkNowN (PUN), a method that uses a lightweight feedforward network, UPNet, to directly predict a neural uncertainty map (UMap) from a single input image. This decouples the view selection process from the costly model retraining.\n\nTo train UPNet, the paper proposes generating a Neural Uncertainty Map (NUM) dataset from ShapeNet objects. It takes a single view of an object, trains a single-view synthesis model (Splatter-Image), and then calculates the reconstruction error (e.g., PSNR) between its synthesized images and ground-truth images at 48 fixed \"anchor\" viewpoints. This 48-dimensional error vector serves as the ground-truth UMap for the initial input view.\n\nAt inference time, PUN takes the current view, passes it through the trained UPNet to predict a UMap, and aggregates these maps over time to select the next-best-view with the highest uncertainty, while filtering out redundant views. The paper claims this method is up to 400x faster than baselines, achieves comparable reconstruction accuracy to an upper bound with half the views, and generalizes to novel objects and complex, realistic scenes.", "questions": "Can the paper provide a stronger conceptual justification for why predicting the reconstruction error of a single-view 3DGS model is a valid and effective proxy for guiding a multi-view NeRF reconstruction?\n\nGiven the massive domain gap between the ShapeNet training data (isolated objects, white background) and the MIP360 test data (complex, cluttered 360 scenes), what features does UPNet actually learn that allow it to generalize so effectively? \n\nHow are the ground truth views chosen? Are they always the same 48 poses, or are they variable? Why do we choose 48 views? Why not 24, 64, or some other number? Is this ablated somewhere?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper is very well-written and easy to follow. The overview figures (e.g., Figs. 1 and 2) are high-quality and provide a good overview of the proposed method and task. The structure is logical, and the claims are stated unambiguously.\n\n2. The paper's primary strength is its computational efficiency at inference time. By replacing a full retraining loop with a single forward pass of the lightweight UPNet, the authors achieve a reported 400x speedup in selection time and massive reductions in resource usage.\n\n3. The method empirically outperforms the selected baselines (A-NeRF, NVF, WD)  in reconstruction quality across most metrics on the test datasets (e.g., Table 1).\n\n4. The paper shows surprisingly good zero-shot generalization results, where UPNet, trained only on synthetic ShapeNet objects, is used to select views for complex realistic scenes from NeRFAssets and MIP360, outperforming baselines", "weaknesses": "1. The papers premise rests on the proposed ground truth uncertainty maps. However, they do not actually measure uncertainty (as do entropy or variance), but it is a map of the reconstruction error for a specific single-view reconstruction model. I would like to see a more formal analysis of why the model should be an effective policy for guiding a multi-view reconstruction, especially for a completely different model class like NeRF.\n\n2. I also believe that the main claim regarding efficiency is a bit misleading. The stated gains only refer to inference time. It completely ignores the massive offline computational cost required to generate the NUM dataset. The paper states it's impractical to train a model for each viewpoint. However, the solution still requires taking 1300 objects (13 categories x 100 instances), sampling 48 viewpoints for each, running a synthesis model (Splatter-Image) from each view, and generating 48 novel views from each of those. This process, repeated to create 62,400 UMap pairs, represents a colossal, one-time computational burden. The paper does not solve the computational problem; it merely amortizes it.\n\n3. In terms of novelty, I am questioning what we learn from this paper. The UPNet is trained entirely on the ShapeNet dataset. This dataset consists of single, isolated objects on white backgrounds, with views sampled on a sphere. This setup has a significant domain gap compared to any real-world application. It is highly questionable how a model trained on this data can learn a meaningful uncertainty prior for complex, 360-degree, cluttered real-world scenes beyond single objects.\n\n4. I am wondering how this method would compare against an active heuristic-based method, such as, for example, a greedy farthest point sampling instead of simply uniform sampling. \n\n5. The UMap is not a continuous map. It is a fixed 48-dimensional vector corresponding to 48 predefined, fixed relative anchor poses. This is a strong, discretizing assumption that limits the granularity of the uncertainty prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qRoVu3rn48", "reviewer": "Reviewer_bRcV", "summary": "This paper introduces PUN, a novel and highly efficient method for Active View Selection (AVS) in 3D reconstruction. Instead of the conventional, computationally expensive approach of retraining a neural rendering model at each step, PUN employs a lightweight network, UPNet, to directly predict a \"Neural Uncertainty Map\" from a single input view. This map guides the selection of the next most informative viewpoint. To enable this, the authors created a large-scale dataset (NUM) of images paired with pre-computed uncertainty maps. The method demonstrates significant improvements in speed (up to 400x) and computational efficiency while achieving reconstruction quality comparable to using a full set of views, and impressively generalizes to novel objects and real-world scenes without retraining.", "questions": "The \"ground truth\" uncertainty is dependent on the Splatter-Image model. How sensitive is UPNet's performance if the NUM dataset were generated with a different backbone model, for example, a NeRF-based one? Does the model learn general geometric uncertainty or just the artifacts of the proxy?\n\nCould you elaborate on the choice of the aggregation for historical uncertainty values? What are the theoretical or empirical advantages of this strategy over additive aggregation or simply taking the maximum uncertainty?\n\nThe paper speculates that generalization stems from learning priors like symmetry. Have you considered experiments on highly asymmetric or topologically complex objects (e.g., a tangled rope) to test the limits of this learned prior?\n\nHow were the hyperparameters, such as the 48 anchor points for the UMap and the 0.1 redundancy filter threshold, determined? How sensitive is the final reconstruction performance to these choices?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The core contribution is reformulating the AVS problem from an iterative optimization into a direct prediction task. This is a significant conceptual shift that effectively breaks the efficiency bottleneck plaguing prior AVS methods for neural rendering.\n- The experimental validation is extensive and convincing. The method is tested across multiple datasets ranging from synthetic objects (ShapeNet) to complex real-world scenes (NeRFAssets, MIP360), demonstrating robust generalization. The ablation studies are thorough and provide clear insights into the method's design.", "weaknesses": "- The \"ground truth\" for the uncertainty maps is generated using the reconstruction error from a single synthesis model (Splatter-Image). UPNet may be learning the specific failure modes of this proxy model rather than a universal notion of 3D uncertainty, potentially limiting its theoretical soundness.\n- Several key design choices lack clear justification. For instance, the paper does not provide a strong rationale for using multiplicative aggregation of uncertainty over time, which is highly sensitive to low values, compared to simpler alternatives like addition.\n- The paper demonstrates impressive generalization from synthetic to real-world data but offers limited insight into why this works so well. The discussion on learning priors like object symmetry is speculative and would be strengthened by targeted experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)