CLiFT: Compressive Light-Field Tokens for Compute Efficient and Adaptive Neural Rendering
Authors: Zhengqing Wang
Yuefan Wu
Jiacheng Chen
Fuyang Zhang
Yasutaka Furukawa
Venue: NeurIPS 2025 spotlight ()
Keywords: Novel View Synthesis
Scene Representation
Compression
Primary area: applications
TL;DR: This paper proposes a novel neural rendering approach that represents a scene as a compact set of "compressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene.

Abstract: This paper proposes a neural rendering approach that represents a scene as "compressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene. CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent a scene or render a novel view with one trained network. Concretely, given a set of images, multi-view encoder tokenizes the images with the camera poses. Latent-space K-means selects a reduced set of rays as cluster centroids using the tokens. The multi-view ``condenser'' compresses the information of all the tokens into the centroid tokens to construct CLiFTs.  At test time, given a target view and a compute budget (i.e., the number of CLiFTs), the system collects the specified number of nearby tokens and synthesizes a novel view using a compute-adaptive renderer. trained to handle a variable number of tokens.  Extensive experiments on RealEstate10K and DL3DV datasets quantitatively and qualitatively validate our approach, achieving significant data reduction with comparable rendering quality and the highest overall rendering score, while providing trade-offs of data size, rendering quality, and rendering speed.

CLiFT: Compressive Light-Field Tokens for Compute Efficient and Adaptive Neural Rendering
Zhengqing Wang' Fuyang Zhang!
Yuefan Wu
Jiacheng Chen'
Yasutaka Furukawa
Simon Fraser University
Wayve
Abstract
This paper proposes a neural rendering approach that represents a scene as "com- pressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene. CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent or render a novel view with one trained network. Concretely, given a set of images, multi-view encoder tokenizes the images with the camera poses. Latent-space K-means selects a reduced set of rays as cluster centroids using the tokens. The multi-view "condenser" compresses the information of all the tokens into the cen- troid tokens to construct CLiFTs. At test time, given target view and a compute budget (i.e., the number CLiFTs), the system collects the specified number of nearby tokens and synthesizes a novel view using a compute- compute-adaptive renderer. Extensive experiments RealEstate 10K and DL3DV datasets quantitatively and qualitatively validate our approach, achieving significant data reduction with com- parable rendering quality and the highest overall rendering score, while providing trade-offs of data size, rendering quality, and rendering speed. Check demos and code on the project page: setts://cciffe nvs. gi thub.
Introduction
Global consumption of visual media skyrocketing, driven by platforms like Instagram, YouTube, and TikTok. Billions of photos and videos are captured, shared, and streamed daily, placing enormous demands on storage and bandwidth. The success of these platforms owes much advances in visual data compression, allowing high-resolution content to be delivered efficiently across range of devices and network conditions. From images high-definition videos, modern compression algorithms have enabled rich visual experiences to become ubiquitous part of everyday life.
Beyond passive viewing of recorded media, interactive novel view synthesis (NVS) is gaining momentum, allowing users to freely navigate virtual environments. Neural rendering techniques such Neural Radiance Fields (NeRF) and 31) Gaussian Splatting (3DGS) the driving forces. Recent research has explored efficiency of neural rendering pipelines, including compression of radiance
fields 24 35 sparse representations [1 22 and adaptive quality rendering 36
Along the line of interactive visual media, reconstruction-fii free novel view synthesis is emerging a promising direction. Models such as Large View Synthesis Models (LVSM) [12] and Scene Representation Transformers (SRT) [25] synthesize novel views directly, without defining bottleneck geometric and photometric representations by heuristics and reconstructing them. By avoiding explicit reconstruction, these methods would better handle scene dynamics and capture fine- aieeggraieed visual details directly from the data. Compute-efficient representations and compute-adaptio rendering within these systems would unlock the full potential of NVS technology, driving new applications
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

across real estate (virtual property tours), entertainment (immersive media and games), online shopping (interactive product displays), and autonomous driving (simulation and model validation).
Towards realizing this potential, this paper introduces a new representation and rendering framework centered on the concept of Compressive Light-Field Tokens (CLiFT). CLiFT is compact set of light field rays with compressed learned embeddings. Concretely, given set of images with camera poses input, multi-view encoder tokenizes the images with camera poses. A latent-space K-means algorithm selects a reduced set of rays as cluster centers. Intuitively, the resulting cluster centers preserve coverage due to geometric diversity among rays, while becoming denser in texture-ried regions. Multi-view "condenser" compresses the information of all the embeddings into the centroid tokens to construct CLiFTs At test time, given a target camera pose and a compute budget (i.e., the number of CLiFTs to use), the system collects the corresponding nearby tokens and synthesizes a novel view using a compute-adaptive renderer that trained to handle a variable number of tokens.
We have evaluated the approach on RealEstate 1OK 37 and DL3DV [66 datasets, and compared with three state-of-the-art methods, LVSM [12] from the reconstruction-fre free approach, and MVS- plat [3] and DepthSplat (34) from the reconstruction-basee approach. Extensive quantitatively and qualitatively validate our approach, achieving significant data reduction with comparable rendering quality and the highest overall rendering score, while providing trade-offs of data size, rendering quality, and rendering speed.
2
Related Work
Light Field Imaging. Computational light field imaging techniques date back to the 1990s with notable work by Levoy et al. on Light Field Rendering [15] and Gortler et al. on Lumigraph [10]. Both introduced the idea of capturing dense array of rays from multiple viewpoints, enabling novel view generation without geometry reconstruction. Subsequent efforts, such as multi-camera arrays (32) and hand-held plenoptic cameras [23], brought these ideas into practical systems. For example, commercial light field cameras like the Lytro (circa 2011 1) demonstrated post-capture refocusing and viewpoint adjustment. Recent works combine classical light field concepts with neural rendering to
enable efficient and accurate novel view synthesis with view-dependent effects (3] 26 28 29]. The
idea of using light fields (or rays) as scene representation forms the core of our approach, where we combine this classical concept with modern neural rendering techniques.
Compressive Sensing. In the mid-2000s, compressive sensing emerged paradigm for capturing signals with far fewer samples than traditional Nyquist sampling theory would require, given the signal is sparse in some basis 4 In computational imaging, compressive sensing inspired novel camera designs that sample and project measurements into lower-dimensional spaces. Notable examples include the single-pixel camera (6] and coded aperture systems (20] that optically encode high- gigh-dimensional scene information into compressed sensor readings. Similar to compressive sensing, our approach selects which rays to store and how to represent them in a compact form for successful view synthesis. A key distinction is that our method learns to compress all the input rays into a compact set of representative rays using neural networks, whereas compressive sensing relies on predefined heuristics to drop information and project to a lower dimension.
Reconstruction-Basee Novel View Synthesis. Early image-based rendering (IBR) systems relied on reconstructed scene geometry for novel view generation. Photo Tourism is a seminal example, using a planar geometric proxy for rendering-simple yet producing compelling visual experiences 27 Sub-
sequent work extended this idea by more accurate depth maps to enhance rendering quality g More recently, Neural Radiance Fields (NeRF) introduced continuous volumetric scene representation as a 5D function that maps spatial location and viewing direction to color and density using an MLP (21). 3D Gaussian Splatting (3DGS) eliminates neural networks in favor of explicit point-based primitives, representing the scene as millions of 3D Gaussians and rendering images by projecting ("splatting") them with rieeddependent shading [3]. These methods typically require per-scene optimization, lack generalization across scenes, and assume dense input coverage. To overcome the limitations, feed-forward Gaussian splatting methods 2 4 eliminate per-scene optimization by predicting Gaussian parameters in single forward pass. In parallel, image and video generative models have been incorporated to refine NVS outputs, enhancing realism and temporal consistency [1]. Recent methods like Reconfusion
3 and NerfDiff [1] take step further by using generative refinement.

---

CLiFT Construction
Plucker Coord.
LiFT
2
Centroid LiFT
CLiFT
2
Transformer Encoder
CLiFT Rendering
2
2
Training
2
2
SA
Decodaa
3
Multi-view Encoding
Latent K-means
Neural Condensation
CLiFT Rendering
CLiFT Construction
Inference
Query
CLiFT Rendering
view
The training consists of three steps: 1) Multi-view encoder, tokenizing the input images; 2) Latent K-means, selecting representative of tokens; and 3) Neural condensation, compressing the information of all the tokens into the representative set to produce Compressed Light-Field Tokens (CLiFTs), Bottom: At inference time, multi-view images are encoded into CLiFTs following the same process in training. Given target view, we collect relevant set of CLiFTs with simple heuristics and render a novel view.
Figure 1: The training and the inference system overview. Top:
Reconstruction- Free Novel View Synthesis. Reconstruction novel view synthesis is gaining attention end-to-end solution. The Scene Representation Transformer (SRT) encodes set of input images into a latent scene representation- collection of tokens-and renders novel views from this latent in single feed-forward pass 25 Another representative work is LVSM (Large View Synthesis Model), fully transformer-basse pipeline that achieves state-of-the-art results from sparse inputs 12 LVSM includes both an encoder decoder transformer, which compresses input images into lexed-lentth latent code before decoding novel views, and decoder-onl transformer that maps input views directly to output pixels. Multi-view generative models share similar characteristics, producing consistent images across views without reconstructing or generating explicit geometry
decooerr
This paper advances the frontier of reconstruction-fres NVS by introducing Compressive Light-Field Tokens: compact, variable-size scene representation paired with renderer that adaptively balances quality and resource usage on demand.
18 30
Compressive Light-Field Tokens (CLiFT)
light field collection rays, each associated with the radiance information. Compressive light-field tokens (CLiFTs) are a collection of rays, each associated with latent vector to which neural encoder compresses the geometry and radiance information target camera pose, traditional light field rendering synthesizes a color for each target ray, typically by linear interpolation of nearby input rays. CLiFT rendering synthesizes an image (not necessarily per ray) by using nearby CLiFTs via neural renderer. The framework allows to control data size,
scene (See Figure T). Given

---

rendering quality, and rendering speed by two hyper parameters 1) Storage CLiFT count (N,), the number of tokens represent scene; and Render CLiFT count (N,,), the number of tokens to use in rendering frame. The section explains the problem, CLiFT construction, and CLiFF rendering.
3.1 Problem Definition
This paper tackles the problem of taking Ne images with the camera poses, and constructing N, CLiFTs scene representation. Given target view, the output is its synthesized image, where N, CLiFTs tokens are to be selected and used for rendering. The evaluation is based on standard image reconstruction metrics, PSNR, SSIM, and LPIPS. A token (CLiFT) is D dimensional embedding, associated with a ray. Ray geometries are represented a global coordinate frame.
3.2 CLiFT Construction
Construction of compressed light field tokens (CLiFTs)) involves three key steps: encoding multi-view images and camera poses, selecting representative rays, and condensing information into the selected set. The section explains the steps.
Multi-view Encoding. Given images with camera poses, we use a standard Transformer encoder to extract our Light Field Tokens (LiFT), which capture both geometry and appearance information. Specifically, for each pixel in each image, we concatenate the 6D Plücker coordinates of the corre- sponding ray with its normalized 3D color vector. We then patchify non-overlapping 8x8 regions, resulting in 576 = (3 + 6) x dimensional vectors. Each vector is linearly projected to token of dimension D= 768. This process converts N. input images of resolution 256 x 256 into 1024 NN tokens. The Transformer encoder performs self-attention across all 1,024 -Ne tokens, producing total of 4,096 LiFTs for scene of input views. Specifically, our encoder has six self-attention blocks, each comprising Self-Attention (8 heads), Add & LayerNorm, Feedforward Network, and another Add & LayerNorm. Unlike the meoder-decoder architecture LVSM, which uses learnable token to aggregate scene information, our method directly uses the outputs of Transformer encoder, which retains the geometry and appearance of a specific region.
Latent-space K-means for Ray Selection. Effective ray selection is crucial, as uniform ray selection across input images leads two redundancies: Appearance redundancy at texture homogencous regions; and 2) Geometric redundancy at visual overlaps among different views. Our approach performs latent-space clustering to select representative rays that compactly represent an entire scene. K-means clustering algorithm finds the clusters from LiFT of all the images. The nearest neighbor sample from each center is retained as the cluster centroid. Centroid LiFT will be the storage CLiFTs, and K is set to N₂ (i.e, the number of storage CLiFTs).
Neural Condensation. A lightweight transformer condenses information from all LiFTs into set of centroid LiFTs, producing CLiFTs. Self-attention operates over the centroid LiFTs to enable information exchange across clusters. Within each cluster, cross-attention the query and the remaining LiFT's as keys and values. To preserve the pretrained latent space, a irooiinitiaizze linear layer aggregates features back into the centroid LiFTs.
the centroid LiFT as
Concretely, the condensation network consists of two transformer decoder blocks. Each block includes inter-cluster self-attention (8 heads), intra-cluster tross-attenting (8 heads), and standard Add & LayerNorm operations. Let Tg E IR" denote the embedding of the k-th centroid LiFT, and {Tk, i} represent the remaining LiFTss assigned to the k-th cluster. SA(-), CA(-,). ), and FFN(-) denote the self-attention, cross-attention, and feed-forward network layers, respectively. W. is pero-initialized linear projection operator W, DDD, Each decoder block updates the centroid
I As explained in $4 we use RealEstate10K (37] and DL3DV [16] datasets. Image resolutions are 256x256 and 2488 respectively. The section uses RealEstate 10K as example to explain the feature dimensions. The patch size is the same for DL3DV, resulting in 1,792 NN tokens instead.

---

embeddings as follows, producing CLiFTs after the second block:
{T&} - SA ([LN(T),, -. }), Vk Th CA(TR (LN(,,))
(1)
Vk T₁ + FEN(LN((T)),
(3)
Vk T& + W.(k)...
3.3 CLiFT Rendering
CLiFTs enable a flexible and efficient rendering mechanism. Given a target view and the render CLiFT count (i.e., the number of tokens to use), we collect set of relevant CLiFTs, which are then used by a neural renderer to synthesize the view.
The renderer is a simple Transformer decoder where the target view serves as the query and the selected CLiFTs serve keys and values. Specifically, we initialize 2D grid of Plücker coordinates at each pixel and patchify them. Each patch is represented as a 3844-dimeniional vector derived from the 6x 8 8 Plücker coordinates, which projected to 768 dimensions via a linear layer. The rendering network has six decoder blocks, each comprising Self-Attention (among query tokens), Add & LayerNorm, Cross-Attention (from CLiFTs to query tokens), Add & LayerNorm, Feedforward Network, and Add & LayerNorm. The number of heads is eight in SA and CA layers. The decoder output is passed through a linear projection, followed by a sigmoid activation to map to the RGB space, and then an unpatchify operation to reconstruct the full-resolution image. We use combination of L2 loss and perceptual loss (with 0.5 weight) following LVSM [12. applied the normalized RGB intensities. During training, we randomly vary the number of CLiFTss passed to the decoder, allowing to learn how to handle different token counts and enabling dynamic trade-offs between rendering quality and computational cost.
Neural Renderer.
Token Selection. The token selection algorithm employs a simple heuristic. Let N, denote the number of CLiFTs used for rendering. To ensure spatial coverage of the target view, we divide the view into 16x 16 grid of patches and pad it with a 4-patch margin on all sides, resulting in a x4 grid. For each patch, we cast a ray through its center and retrieve the N2,((2 x 24) closest CLiFTs from pool of N₂ storage CLiFTs. The distance between patch and a CLiFT computed using a heuristic based on their associated rays (ray origins and directions); details are provided in the supplementary material. Since CLiFTs may be selected multiple times, we greedily accumulate CLiFT' in order of their minimum distance to any patch ray until N, unique tokens are selected.
4 Experiments
4.1 Experimental Setup
Datasets. We use two scene datasets RealFsttate 33 and DL.3DV 16,
following the recent
literature 3 12 34
We preprocess videos and create training/testing video clips in exactly the
same as PixelSplat 2 the number of images to use from each clip for training and testing. Concretely, LVSM [12, MVSplat 3 and DepthSplat [34] used images for training and testing for RealEstatel MVSplat and DepthSplat used 2-6 images for training and testing for DL3DV. We use 4-6 images for training and 4-8 images for testing in both datasets to handle larger scenes.
for RealFsstate 10K and DepthSplat [4 for DI.3DV. The only difference
Evaluation Metrics. We evaluate the rendered image quality by PSNR, LPIPS, and SSIM. To assess computational efficiency, we report data size, rendering FPS, and rendering FLOPs, where FLOPs are theoretical numbers instead of measured ones.
Training Details. We adopt a two-stage training strategy. The first stage is to train the multi-view encoder where the neural renderer is directly connected to mack-propagate gradients without the latent K-means or the condensation modules. In this case, all the tokens from the encoder are passed to the renderer through cross-attention. The second stage uses all the modules while freezing the multi-view encoder. Since K-means is not efficient for online sampling, we pre-compute the cluster assignments
"RealEstate10K is under Creative Commons Attribution 4.0 International License. DL3DV is under DL3DV. 10K Term of use and Creative Commons Attrbution-NonCCommecce NonCommercial 4,0 International License.
5

---

offline and use them in the second stage. For experiments on the 256x256 RealEstate the first and the second stages train for 90.000 steps with batch-size 64 and 50,000 steps with a batch-size 80, respectively. For DL3DV (256x448), we finetune the RealEstate 100 p0r-prerraieed model for 100,000 steps with batch size of 24 in the first stage and 32 the second stage. Both datasets use the same cosine learning rate scheduler with 2500-step warmup. The peak learning rate is 4 x 10-1 for RealEstate 10K and 2 x 10 4 for DL3DV, with the learning rate of the renderer scaled by 0.1 in the second stage. We use four NVIDIA RTX A100 GPUs to train our model. Training takes approximately 3 days on RealEstate IOK and days on DL3DV,
Baselines. We compare with three state-of of-the-art methods, one from the reconstruction-fit approach and the other two from the reconstruction- based approach.
LVSM [12] is a tfate-of-the-art reconstruction-fret method. While a publicly available checkpoint exists for RealEstate 10K, it corresponds to a significantly larger model, featuring twice as many blocks in both the encoder and decoder, and was trained using 64 A100 GPUs with only 2 input views. To enable fair comparison, we trained the LVSM-ED system under our setting: using 6 blocks in both the encoder and decoder, and input views. Given the substantial compute requirements of LVSM (i.e., 64 A100 GPUs as reported by the authors), we chose to compare on RealEstaae As the LVSM code was not available during our initial experiments, we first reproduced their system independently, and later refined our implementation based on the official code after its release.
MVSplat [3] and DepthSplat [34] are state-of-the-arr feedforward splatting- -ased novel view syn- thesis systems. They achieve strong performance on benchmarks like RealEstate 10K and DL3DV by leveraging explicit 3D scene representations and depth-guidel 3D Gaussian splatting for ren- dering. We use publicly available checkpoints for MVSplat and DepthSplat (i.e., 2-view case for RealEstate 10K and 2, 4, or v-view cases for DL3DV).
4.2 Main Results
Figure 2 and Figure 4 show the main evaluation results on the RealEstate and DL3DV datasets. In the plots, the x-axis represents the data size of the scene representation: Storage CLiFTs for our method, decoder input tokens for LVSM, and splats for MVSplat and DepthSplat. The y-axis reports PSNR, SSIM and LPIPS.
None of the baselines support controlling the data size and require a separately trained model for each data point. Many baseline points are missing from the plots because we either use publicly available checkpoints (for MVSplat and DepthSplat) or train a new model ($4.1). In contrast, our method trains single model per dataset and supports fine-grained control over both the storage data size and the render data size via the numbers of Storage (N₂) and Render (N₂, CLiFT tokens.
The plots show that CLiFT achieves comparable PSNR with approximately 5-7x less data size than MVSplat and DepthSplat, and about 1.8x less than LVSM, highlighting the effectiveness of our compressed tokens and overall scene representation. CLiFT also attains the highest overall PSNR with significantly lower data usage. Qualitative results in Figure 4 support these findings: our method preserves sharp appearance details that are closer to the ground truth and maintains high visual fidelity even under strong compression, with only minor loss in high-frequency content.
4.3 Ablation Studies
To demonstrate the effectiveness of our individual components, Figure 3 compares three variants of our system: 1) The full system (blue): 2) The system without the condenser (green); and 3) The system without both the condenser and latent K-means (red), where tokens are selected randomly instead. We evaluate all variants on 92 randomly selected scenes to ensure consistent and fair comparisons, using an NVIDIA RTX A6000 GPU.
CLiFT Construction.
Figure shows how rendering quality (PSNR, LPIPS, SSIM), rendering speed (FPS), and rendering cost (FLOPs: theoretical number instead of measured) change with different compression rates. Specifically, we fix the number of CLiFT tokens for storage (N₂) and rendering (N2,) to the same, and vary them together. When the compression rate is low (<2x), all variants perform similarly since even randomly selected tokens provide reasonable coverage. However, at high compression rates, the gap between random selection and K-means selection becomes significant: random selection fails to capture representative tokens, while K-means selects tokens that better summarize the scene. Figure visualizes the selected centroids and their associated cluster members. The clustering is performed

---

0.10
Caee L/S3EED DepdirSplat
0.58
0.10
PSNR
SSIM
0.20
LPIPS
0.14
0.84
0.12
85
12
13
Storage/Render CLiFTs [MB]
Ours 16 Varaal
Viows
Ourr (4 Vanani
24-
0.35
Deptttiyaa SIVippat
PSNN
22-
Vieews
SSIM
LPIPS
0.330
20-
Viewss
0.20
Views
2
Views
18-
0.15
10 20 30
40
Storage/Render CLiFTs [MH]
Figure 2: Main evaluation results on the RealEstate 10K dataset (top) and the DL3DV dataset (bottom), comparing our approach with three baseline methods (LVSM-ED [12], DepthSplat [34], and MVSplat (3). The x-axis is the data size of the scene representation, and the y-axis is the rendering quality (PSNR, SSIM and LPIPS). Our approach CLiFT can flexibly change the data size (i.e., number of tokens) with trained model, achieving significant data size reduction with comparable rendering quality and the highest overall PSNR, while providing trade-offs among data size, rendering quality, and rendering speed.
Table We fix the number of storage CLiFTs to represent scene (N,,=4066) then vary the number of render CLiFTS (how many tokens to use for rendering) on-the-fly and measure rendering quality (PSNR), rendering speed (PSNR), and rendering cost (theoretical number as GFI.OPs).
Render CLiFTs
Metrics
4096
3072
2048
1024
512
PSNR
26.72
26.71 (-0.01)
26.56 (-0.16)
25.75 (-0.97)
23.89 (-2.83)
GFLOPs
70.6
63.3 (-10%)
56.1 (-21%)
48.9 (-31%)
45.23 (-36%)
FPS
54.3
53.89 (-1%)
66.44 (+22%)
80.77 (+49%)
90.15 (+66%)
jointly across multiple views, encouraging tokens from different images to form non-redundant clusters. Furthermore, clusters tend to grow larger in hotuureeoomogeeeuuu regions.
Effectiveness of K-means clustering. We also evaluate the effectiveness of latent K-means clus- tering by comparing it against a simple patch-wise grouping baseline. The baseline divides the im- age into non-overlapping local patches, assigning tokens uniformly across regions, whereas latent K-means adaptively clusters tokens in latent space. As shown in Table latent K-means consistently outperforms patch-wise clustering across both to- ken budgets, with especially large gains under stronger compression (e.g., +2.24 PSNR at 256 to- kens with condensation). This improvement arises because patch-wise clustering distributes capacity evenly, while latent K-means allocates more tokens to informative regions. illustrated in |Figure 11 []d FFiuure 12, this adaptive allocation leads to better visual quality when budgets are tight.
Table 1: Ablation study comparing latent K-
means clustering with simple patch-wise group- ing baseline. We report PSNR under two token budget settings (256 and 1024), with and without the condenser.
Method
256 tokens
1024 tokens
Patch-wise
22.64
27.48
Patch-wise + Condenser
22.97
27.55
K-means
24.46
28. 17
K-means + Condenser 25.21
28.41
7

---

0.90
0.30
0.85
PSNR
LPIPS
SSIM
0.80
24-
0.75
0.15-
22
0.10
1024
Storage/Render CLiFTs
Storage/ Renigee/Rnnerr CLiPTs
Storago'Rende CLiFTs
Ours
Rendering FPS
100
70-
GFLOPS
Ours w/o Condenser
Ours w/o Condenser w/o K-meons
50-
20
LVSM-ED
128
256
812
1024
2048
4096
120
256
512
1024
2048
4096
Storage/Render CLiFTs
Storage/Rende CLiFTs
Figure 3: Ablation studies on our individual components, in particular, latent K-means and neural condensation. The plots compare three variants of system dropping latent K-means and neural condensation one by one from the system, while varying the data size. Specifically, the x-axis the size of the representation. The y-axis rendering quality (PSNR, LPIPS, and SSIM), rendering speed (FPS), or rendering cost (FLOPs), measured on an NVIDIA RTX A6000 GPU.
CLiFT Rendering, A CLiFT token is associated with specific ray, encoding localized geometry and appearance. This ray-based design enables intuitive and efficient token selection heuristics at inference time (3.3), allowing the renderer to control the trade-off between quality and speed the fly. For example, in large scenes with multiple rooms, rendering a view within one room can be done without using tokens from unrelated areas. contrast, methods like LVSM represent the entire scene using a fixed set of global tokens, which limits their ability to selectively render parts of the scene or adjust computation dynamically.
We validate this ability on large scenes from the RealEstate I0K dataset. Specifically, we construct a large-scene test set by filtering for scenes with more than 200 frames, higher frame counts typically indicate broader camera motion. We use such scenes for evaluation, with input views per scene to ensure sufficient coverage. As shown in Table 2 our method supports on- the fly adjustment of rendering cost by varying the number of tokens used, achieving lower FLOPs and higher FPS when needed, allocating more tokens for higher quality.
5 Conclusions and Future Challenges
We introduced CLiFT, compressive light-field token representation paired with compute-adaplln transformer renderer. Experiments on RealEstate and DL3DV demonstrate that CLiFT achieves higher data reduction rate with comparable rendering quality and the highest overall rendering score, while capable of controlling the trade-offs between data size, rendering quality, and rendering speed. Our current system exhibits two typical failure modes. The first occurs when camera motions deviate significantly from the training distribution. As shown on the left of Figure 6) RealEstate 10K training data primarily consists of smooth translations with minor rotations, making it difficult to generalize to more complex motions. The second failure mode arises large scenes where target views are not adequately covered by the input views, resulting in blurry renderings. This is illustrated on the right of Figure 6 with examples from DL3DV. A promising future direction is to extend our framework by incorporating generative priors improve rendering quality in unseen occluded areas.
Broader Impacts. On positive side, our method has potential applications in digital media content consumption, enabling more efficient and flexible rendering for immersive experiences. However, like generative models, it raises concerns around cessuse-prticclartt in the creation of deep-fakes. We emphasize the importance of responsible deployment and clear content provenance.

---

LVSM-ED 9.44 MB
Ours
Ground Truth
0.79 MB
1.57 MB
3.15 MB
6.29 MB
12.58 MB
23.16
22.37 (-0.79)
23.57 (+0.41)
23.98 (+0.82)
24.50 (-1.34)
24.99 (+1.3)
27.59
23.17 (-4.42)
25.38 ( 0.21)
27.28 0.31)
28.57 ( 0.98)
29.46 (+1.87)
25.68
22,18 (-3.50)
24.31 (-1.37)
25.41 (-0.27)
26.56 (-0.88)
26.89 (+1,21)
31.68
28.24 ( (3.44)
30.61 L.0))
31.90 (10.22)
34.07 (12.39)
34.90 (13.22)
DepthSplat
Ours
Ground Truth
4.13 MB
16.51 MB
33.02 MB
24.18
26.24 (12.96)
27.16 (12.98)
27.8 (13.62)
19.17
23.41 (+4.24)
24.99 (+5.82)
25.61 (+6.44)
23.01
20.77 (-2.24)
22.86 (-0.15)
23.44 (+0.43)
24.34
24.11 (-0.26)
25.51 (+1.14)
25.89 (+1.52)
Figure 4: Qualitative rendering results of the baselines and ours with different data size (i.e., the number CLiFT tokens for ours). Top: Ours vs. LVSM 12 on RealEstate 10K. Bottom: Ours vs. DepthSplat 334 DL3DV, The PSNR value is recorded under each rendering.

---

Figure 5: Visualization of the latent K-means clustering, where K=N,=128. Each color represents a cluster, and the yellow ring indicates the centroid token. Note that clustering is performed across multiple views, so a single cluster can span multiple images. As a result, some clusters may not have visible centroid in given image.
Estate 10K
DL3DV
Figure 6: Typical failure cases (Left) Camera motions deviating too much from the training distributions in RealEstate100K and (Right) Target views not covered by the input images in DL3DV, The top shows results, and the bottom row shows the ground truth.
Acknowledgments
We thank Haian Jin for helpful discussions on reproducing LVSM and training on the DL3DV dataset. We thank Jiaqi Tan for assistance with the project page and demo design. This research partially supported by NSERC Discovery Grants, NSERC Alliance Grants, and John R. Evans Leaders Fund (JELF). We thank the Digital Research Alliance of Canada and BC DRI Group for providing computational resources.
10

---

References
[1] Emmanuel J Candès, Justin Romberg, and Terence Tao. 2006. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory (2006).
[2] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. 2024. pixclsplat: 3d gaussian splats from image pairs for scalable generalizable reconstruction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 19457-19467.
[3] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. 2024. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision. Springer, 370-386.
[4] David Donoho. 2006. Compressed sensing. IEEE Transactions on Information Theory (2006).
[5] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. 2023, Learning to render novel views from wide-baseline stereo pairs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 4970-4980.
[6] Marco F Duarte, Mark A Davenport, D) Takhar, Jason N Laska, Ting Sun, Keith F Kelly, and Richard G Baraniuk. 2008. Single-pixel imaging via compressive sampling. IEEE Signal Processing Magazine (2008).
[7] Lue Fan, Hao Zhang, Qitai Wang, Hongsheng Li, and Zhaoxiang Zhang. 2024. FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes. arXiv preprint arXiv:2412.03566 (2024).
[8] Yasutaka Furukawa and Jean Ponce. 2009. Accurate, dense, and robust multiview stereopsis. IEEE transactions pattern analysis and machine intelligence (2009).
[9] Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEF/CVF international conference on computer vision. 14346-14355.
[10] Steven J Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael F Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 43-54.
[11] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. 2023. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In International Conference on Machine Learning. PMLR, 11808-11826.
[12] Haian Jin, Hanwen Jiang. Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. 2024, Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint arXXi:2410.17244 (2024).
[13] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. (2023).
[14] Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J Davison. 2024. Eschernet: A generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9503-9513.
[15] Mare Levoy and Pat Hanrahan, 1996. Light field rendering. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques. 31-42.
[16] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. 2024. D13dy-10k: A large-scale scene dataset for deep learning-basee 3d vision. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. 22160-22169

---

[17] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020. Neural sparse voxel fields. Advances in Neural Information Processing Systems 33 (2020), 15651- 15663.
[18] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. 2023. Synedreamer: Generating multiview-consister images from a single-view image. arXiv preprint arXiv:2309.03553 (2023).
[19] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. 2024. Wonder3d: Single image to 3d using cross-domain diffusion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9970-9980.
[20] Kshitij Marwah, Gordon Wetzstein, Yosuke Bando, and Ramesh Raskar. 2013. Compressive light field photography using overcomplete dictionaries and optimized projections. In ACM Transactions on Graphics (TOG). ACM.
[21] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM (2021).
[22] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions on graphics (TOG) 41, 4 (2022), 1-15.
[23] Ren Ng. Mare Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan. 2005. Light field photography with a hand-held plenoptic camera. Ph. D. Dissertation. Stanford university.
[24] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF international conference on computer vision. 14335-14345.
[25] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, al. 2022. Scene represen- tation transformer. Geometry- free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition.
[26] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand, 2021. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems 34 (2021),
[27] Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006. Photo tourism: exploring photo collections in 3D. In ACM Transactions on Graphics (TOG). ACM.
[28] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. 2022. Generalizable patch-based neural rendering. In European Conference on Computer Vision. Springer, 156-174.
[29] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. 2022. Light field neural rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. 2279.
[30] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. 2024. Mydiffusion+++ A dense high- resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. In European Conference Computer Vision. Springer.
[31] Qitai Wang, Lue Fan, Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. 2024. Freevs: Generative view synthesis on free driving trajectory. arXiv preprint arXiv:2410. 18079 (2024).
[32] Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Mare Levoy. 2005. High performance imaging using large camera arrays. In ACM Transactions on Graphics (TOG).
12

---

[33] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin. Jonathan Barron, Ben Poole, al. 2024. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. 21551-21561.
[34] Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Mare Pollefeys. 2024. Depthsplat: Connecting gaussian splatting and depth. arXiv preprint arXiv:2410.13862 (2024).
[35] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021. Plenoc- trees for real-time rendering of neural radiance fields. In Proceedings the IEEE/CVF interna- lional conference on computer vision. 5752-5761.
[36] Shiran Yuan and Hao Zhao. 2024. Slimmerf: Slimmable radiance fields. In 2024 International Conference on 3D Vision (3DV). IEEE, 64-74.
[37] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo mag- nification: Learning view synthesis using multiplane images, arXiv preprint (2018).
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our main contributions are summarized in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the Isection 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All necessary details to reproduce the main experimental results are provided in section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification:
We will release the code and model in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee 7/nips.com htte://alllllgiiiee////eeeuaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We include the training and test details
Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the official benchmark conduct the evaluation to be consistent with prior works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resource information the section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper completely conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: he paper does not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite every asset and we follow the license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our method does not involve LLMs in any part of the core methodology, model design, or experimentation.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Appendix
The appendix provides:
A Additional more rendering results and intermediate visualizations (i.e., clusteirng and token-
se selectonn results).
Implementation details of the latent K-means algorithm (during CLiFT construction) and the token selection algorithm (during CLiFF rendering).
Please check our project page htttps:/cclitt nvs. github........ io which shows rendering results at different compression rates and compares with the baseline methods.
A Additional Rendered Images
Figure 4 of the main paper presents rendering results under varying data size constraints. We provide additional qualitative examples on RealEstate 10K (see 7 and Figure 8) and DL3DV (Figure Figure 10). These examples cover finer range of compression rates, extending up to 32x, and highlight the robustness of our method across varying levels of compression. We also show more K-means clustering results in Figure 11 and Figure 12
B
Additional implementation details
B.1
Details of K-means algorithm
During training, we precompute cluster assignments after the multi-view encoder training and before the condensation training, using faiss. Kmeans which supports GPU acceleration. At test time, use sklearn.cluste KMeans for better accuracy.
B.2 Details of Token Selection
Rendering specific region within large scene does not require all tokens. Using only the tokens essential to the target view improves FPS and reduces FLOPs, with minimal impact on PSNR. Algorithm is our token selection algorithm.
21

---

Ours
Ground Truth
0.39 MB
0.79 MB
1.57 MB
3.15 MB
6.29 MB
12.58 MB
Figure 7: Additional qualitative results on the RealEstate 10K dataset.
22

---

Ours
Ground Truth
0.39 MB
0.79 MB
1.57 MB
3.15 MB
6.29 MB
12.58 MB
Figure 8: Additional qualitative results on the RealEstate 10K dataset.
23

---

Ours
Ground Truth
4.13 MB
8.26 MB
16,51 MB
33.02 MB
Figure Additional qualitative results on the DL3DV dataset.
24

---

Ours
Ground Truth
4.13 MB
8.26 MB
16.51 MB
33.02 MB
Figure 10: Additional qualitative results on the DL3DV dataset.
25

---

Algorithm Token Selection Algorithm
1: Definitions:
2:
8: Ray-angle distance between patches
3: 5: Camera center distance, = | 0k
4: my. Last frame token mask
5: P: Number of rays per patch (e.g.. P = 16 x 16)
6: N: number of condition images
7: Wangle 1.0, Waist 0.02, Wmask -0.03
8: momentum factor 0.5
9: Input:
Target view ray (or, de) with or R3 de Rxx3,
11:
Condition view rays {(o", d with ot R3, RR
12: Last frame token mask {m* L
Tk=1
13:
Total number of tokens to select
14: Output: Selected token indices I
15: Downsample target and condition view rays into 16 x 16 patches
16: Expand each target patch
24 x 24 region, including out-of-image rays, to incorporate edge-adjacent context and avoid boundary under-coverage
Set number condition rays per target patch + / (24 24)
18: Initialize selected patches Ipatch <-
// Patch-wise selection
20: for each patch P. in expanded target patches do for each condition view from 1 to do for each condition patch p in condition view do
22:
Compute ray-angle distance between rays in and pk.j
24:
Compute camera center distance between target and condition view ö = l|0: 0¹)
25:
Retrieve last frame mask m
Compute frame distance: D Wangle + Wdist 8 Wmask m
Apply momentum: D. + (1 11) D. + D..5
27:
Store previous step objective: D. D. kgj end for
28:
29:
30:
end for
Let D: = (D. vk,j}
31:
32: Ti +- Top,, (argsort (D:))
Ipuich +- Ipatch Ii
33: end for
34: Compute unique set: Tunipatch +- unique
35: Let Trest f- [unipatch
36: // Global f'alback selection
Initialize set of fallback distances: D Dgiobal (
38: for each condition view & from 1 to N do
for each condition patch j in view i do
40:
Compute D min; D..
best match to any target patch
41: Detohal &- D Pllchal U BBbd.
42: end for
43: end for
Iglobal <- Top,, (argsort (e (Delobal)
45: return = Iparch Iglobal
26

---

Figure 11: Additional visualization of the latent K- memans algorithm for the RealEstate 10k dataset.
27

---

Figure 12: Additional visualization of the latent K-means algorithm for the DL3DV dataset.
28

---

Figure 13: Additional visualization of the latent K-means algorithm with different values of K, which
is 512, 1024, 2048, and 3072 from the top.
29


Reviews:
[{"review_id": "dhqfgWWW7q", "reviewer": "Reviewer_gUfU", "summary": "This paper introduces CLiFT, a novel approach for neural rendering that represents a scene using a set of \"compressive light-field tokens.\" The primary contribution is a framework that is both data-efficient and adaptive, allowing a single trained model to trade-off between storage size, rendering quality, and computational speed. The method consists of a three-step construction process: 1) a multi-view Transformer encoder converts input images and poses into initial \"Light Field Tokens\" (LiFTs); 2) a latent-space K-means algorithm clusters these tokens to select a compact set of representative \"centroid\" rays, effectively reducing geometric and appearance redundancy ; and 3) a \"neural condenser\" further compresses information from all tokens in a cluster into these centroids, creating the final CLiFTs. At inference time, a novel view is synthesized by a Transformer decoder that uses a subset of these CLiFTs, selected based on proximity to the target view's rays. Experiments on the RealEstate 10K and DL3DV datasets show that CLiFT achieves comparable or better rendering quality than state-of-the-art methods like LVSM and DepthSplat, but with significantly smaller data sizes.", "questions": "Regarding the Two-Stage Training: The paper describes a two-stage training process where the encoder is trained first and then frozen.  What is the motivation behind this design instead of end-to-end training? \n\nOn Failure Cases: The limitations section mentions that the method struggles with camera poses that are very different from the training data.  Is there any mechanism within the model to estimate the uncertainty or expected quality of a rendering for a given target pose? This would be very useful for practical applications.", "limitations": "Yes, the authors discuss limitations in Section 5, pointing out failure cases related to out-of-distribution camera motions and insufficient view coverage in the input images.  This discussion is fair and provides a clear picture of the method's boundaries.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "- Strengths:\n1. Adaptive Representation: The most compelling strength of CLiFT is its ability to operate across a spectrum of data sizes and compute budgets with a single trained model. Users can dynamically adjust the number of tokens for storage or rendering to balance quality and speed. This is a significant practical advantage over existing methods, which typically require retraining a separate model for each desired data size or quality level. \n\n2. High Compression Efficiency: The paper demonstrates impressive data compression. CLiFT achieves comparable PSNR to MVSplat and DepthSplat with 5-7x less data and to LVSM with 1.8x less data.  This efficiency is achieved through the intelligent reduction of redundant information, making the representation highly compact.\n\n3. Principled Token Selection and Condensation: The use of latent-space K-means to identify and select representative rays is a more principled approach to compression than uniform or random sampling. The ablation studies in Figure 3 clearly show that this selection strategy is critical for maintaining quality at high compression rates. The subsequent neural condensation step further refines these tokens, leading to consistent performance gains.\n\n- Weakness:\n1. Generalization Limitations: As acknowledged by the authors, the method struggles when faced with camera motions that deviate significantly from the training distribution or when target views are not well-covered by the input images.  This highlights a potential weakness in generalization, which is a common challenge for learning-based novel view synthesis methods.\n\n2. Multi-Stage Training Complexity: The training pipeline is quite complex, requiring a two-stage process where the multi-view encoder is trained first, then frozen, followed by a second stage to train the condenser using pre-computed K-means assignments.  This separation may introduce optimization challenges and could be a barrier to reproducibility and future extensions compared to a single, end-to-end trainable system.", "was_revised": true, "final_justification": "The author addressed my concerns by providing the reference and justification, as well as discussing the generalization limitations and multi-stage training complexity. I believe the author is aware of these limitations and has a grasp on addressing them. Therefore, I uphold my initial decision to Accept.", "rebuttal": "Thank you for the constructive comments and questions.\n\n**W1: Generalization limitations on camera poses.**\n\nWe agree that generalization to novel camera poses is a key challenge for learning-based novel view synthesis (NVS), particularly for reconstruction-free approaches that do not rely on hand-crafted geometric or photometric representations. Figure 6 shows our typical failure cases. We observe that this generalization issue often occurs for RE10K, which has limited camera motion variations and results in poor generalization (L267-270). DL3DV has more complex camera trajectories, and we observe far fewer such failures, indicating that the limitation is mainly due to training data coverage. We will clarify the point in the paper.\n\n**W2&Q1: Multi-Stage training complexity.**\n\nWe acknowledge that single-stage training presents challenges, as latent K-means is not readily differentiable. This naturally leads to a decoupled training process, where the encoder is trained separately from the condensation and rendering modules. While end-to-end training using a form of “soft clustering” is an exciting direction for future work, we emphasize that multi-stage training is a practical and effective strategy, often leading to more stable optimization and easier modular development. For example, latent diffusion models typically train the autoencoder and the denoising network in two separate stages. Importantly, at inference time, our method only requires running the encoding (latent K-means, neural condensation) for each scene once, and the rendering module is very fast.\n\n**Q2:  Mechanism to estimate the quality of rendering given a target pose**\n\nWe thank the reviewer for the valuable suggestions. As in many novel view synthesis works, rendering quality is typically higher when the target camera pose is closer to the input camera pose and degrades as the distance increases. A simple approach is to compute the translation and rotation differences between the target pose and the closest input pose, which can serve as a rough estimate of rendering quality. We will include related results in the paper."}, {"review_id": "83ETLytqBH", "reviewer": "Reviewer_g7Ys", "summary": "The authors present CLiFTs (Compressed Light-Field Tokens), a novel and data-efficient neural scene representation that addresses the high memory demands of existing neural rendering methods. Built on a Transformer encoder-decoder architecture, the framework introduces an encoding stage that constructs CLiFTs by applying K-means clustering in latent space to identify feature centroids from multi-view inputs. A proposed condensation network is then used to compress neighboring features around each centroid.\n\nThis approach enables significantly reduced memory storage compared to traditional feed-forward neural rendering pipelines, while still producing high-quality rendered images. The method is evaluated on two benchmark datasets, RealEstate10K and DL3DV, against three representative baselines, LVSM, MVSplat, and DepthSplat. CLiFTs demonstrate a good trade-off between rendering quality and data efficiency, particularly outperforming prior methods from the perspective of storage and computational cost.", "questions": "- It would be helpful to include an experiment that evaluates how the method performs across scenes with varying complexity. Since both datasets used in the paper may contain scenes of similar visual complexity, it’s hard to tell how well CLiFT generalizes to more diverse scenarios, and its range of scenes' complexity that CLiFT can cover.\n\n- Including an ablation study that replaces the K-means clustering with simple alternative, such as grouping locally in patch style, and uses only the neural condensation component would be valuable to understand the proposed methods' contribution.\n\n- The paper could benefit from more explanation around the architecture and design choices of the condensation network. A brief motivation for the layer/module design and why certain decisions were made would help readers better understand how this component contributes to the overall method.\n\n- Does the model maintain good performance when more input views are provided? It would be good to include an experiment that tests the model’s generalization ability with a larger number of input views, especially if the number of CLiFT tokens is reduced at the same time. Showing the upper-bound of models' performance within the trade-off of the number of input views and tokens could further highlight its strength as a data-efficient approach.\n\nBased on the authors' rebuttal, I'm willing to revise my initial rating.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "> Strengths\n- The paper is clearly written and easy to read. The target problem is well-motivated, and the evaluation metrics are appropriately chosen to show the effectiveness of the proposed method.\n- The proposed method shows strong performance in terms of data efficiency, outperforming existing baselines both quantitatively and qualitatively.\n\n> Weaknesses\n- The rationale for choosing a light field representation is not sufficiently discussed. It's unclear what specific advantages it brings over alternative representations, especially in comparison to the baselines.\n- There is limited discussion on potential issues with using K-means clustering in latent space. It would be helpful to include analysis or experiments addressing whether the method generalizes well across diverse scene types, e.g., across the scenes of varying complexity.\n- The condensation network is introduced as a core component, but details on its architecture and design choices are insufficient. More justification or ablation results would strengthen the quality of paper.", "was_revised": true, "final_justification": "The authors have adequately addressed most of the concerns I raised in my initial review, providing appropriate results and detailed explanations in the rebuttal. I believe the paper is of a quality that can receive a rating close to acceptance.", "rebuttal": "We thank the reviewer for the constructive and detailed feedback. \n\n**W1: Advantages of light field representation.**\n\n**Compared with Reconstruction-Based Methods (e.g., DepthSplat [34]):** \n\nA key advantage of our *light field representation* is that it is *reconstruction-free*—it does not require hand-engineered intermediate geometric or photometric representations such as 3D Gaussians or spherical harmonics (L30–32). Instead, the entire pipeline operates on learned “tokens,” making the representation fully data-driven and flexible. This aligns with an emerging trend in novel view synthesis (NVS), as exemplified by methods like LVSM [12].\n\nThis token-based formulation offers two main advantages:\n\n1. It provides a straightforward path to extend NVS capabilities by integrating with other token-based networks, such as LLMs or VLMs, enabling applications like text-controlled rendering (e.g., adjusting style or lighting via prompts) or embodied AI (e.g., using NVS tokens as pretraining input for policy learning in agents, as done in VLA).\n2. It improves the system’s ability to handle complex scenes—both geometrically (e.g., thin structures) and photometrically (e.g., view-dependent effects and potentially dynamic content)—which are typically challenging for methods relying on explicit reconstruction.\n\nWe thank the reviewer for raising this point and will revise the paper to better highlight these advantages of the light field representation (L30–32).\n\n**Compared with LVSM:**\n\nLike our approach, LVSM is also reconstruction-free and operates entirely on token-based representations. The key distinction is in how the scene is represented. LVSM models the scene using all tokens from all input images, without explicit spatial organization. In contrast, our *light field tokens* (CLiFTs) are explicitly associated with rays, encoding localized appearance and geometry. This ray-based, localized design is fundamental to our contributions: it allows the scene to be represented as a variable-length list of compressed tokens and enables *compute-adaptive rendering* by selecting only the subset of tokens relevant to a target view. We will revise the paper to clarify these differences more explicitly (L91–99).\n\n**W2&Q2: Potential issues with K-means and the ablation study for its effectiveness**\n\nWe thank the reviewer for raising this important question. We agree that the latent K-means clustering may not be optimal, as it is based on heuristics and not explicitly optimized for the final objective—rendering quality. To assess its effectiveness, we compared latent K-means with a simple patch-wise clustering baseline that uses non-overlapping local patches. The table below reports PSNR scores for both methods, with and without neural condensation, under two token budget settings (256 and 1024 tokens):\n\n|  | 256 | 1024 |\n| --- | --- | --- |\n| Patch-wise | 22.64 | 27.48 |\n| Patch-wise with condenser | 22.97 (+0.33) | 27.55 (+0.07) |\n| K-means | 24.46 | 28.17 |\n| K-means with condenser | 25.21 (+0.75) | 28.41 (+0.24) |\n\nThese results show that latent K-means consistently outperforms patch-wise clustering at both budget levels, with particularly notable improvements under stronger compression (e.g., +2.24 PSNR at 256 tokens with condensation). This advantage arises because patch-wise clustering allocates capacity uniformly across regions, while latent K-means adaptively places tokens in more informative areas, thereby better preserving quality when token budgets are tight. We will include this ablation study in the final version of the paper.\n\n**W3&Q3: Motivation, design choice, and ablation of the condensation network.**\n\nWe thank the reviewer for their feedback. The goal of the neural condensation module is to extract information from *to-be-discarded tokens and* condense it into a smaller set of representative tokens. The architecture is intentionally designed to achieve this: self-attention facilitates information exchange among cluster centroids, while cross-attention allows each centroid to absorb relevant information from the tokens assigned to it. We will clarify this explanation in Lines 139–150 of the revised paper.\n\nOur ablation study in Figure 3 supports this design choice. When the condenser module is removed, model performance degrades significantly under high compression rates. This is expected, as the majority of token information is lost without condensation, highlighting the importance of this module for effective compression.\n\n**Q1: Method performs across scenes with varying complexity**\n\nWe followed prior works in selecting the RealEstate10K and DL3DV datasets, which offer varying levels of complexity. RealEstate10K consists of indoor scenes with relatively smooth camera trajectories, whereas DL3DV presents more challenging conditions, including outdoor environments and complex camera motion.\n\nOur model demonstrates strong performance on both datasets, highlighting its ability to generalize across diverse scenarios. A visual comparison of the rendering results on these datasets is included in the supplementary video (demo.mp4).\n\n**Q4: The performance of more input views.**\n\nWe conducted additional ablation studies by increasing the number of input views without retraining for that setting, and observed little to no performance gain—consistent across LVSM, DepthSplat, and our method.\n\n*RE10K (ours & LVSM, trained for a fixed 4 view count):*\n\n|  | 4 views | 6 views | 8 views |\n| --- | --- | --- | --- |\n| LVSM | 28.95 | 29.16 | 29.19 |\n| CLiFT | 29.40 | 29.48 | 28.92 |\n\n*DL3DV (DepthSplat pretrained with 2~6 views):*\n\n|  | 6 views | 8 views | 10 views |\n| --- | --- | --- | --- |\n| DepthSplat | 25.43 | 25.16 | 24.86 |\n\nIn addition, the recent paper PRoPE [A] also shows that the current camera encoding approach struggles to generalize to more input views. The proposed relative camera encoding could help address this problem.\n\n---\n\n[A] Li, R., Yi, B., Liu, J., Gao, H., Ma, Y., & Kanazawa, A. (2025). Cameras as Relative Positional Encoding. arXiv preprint arXiv:2507.10496."}, {"review_id": "abv1Yo1Dk3", "reviewer": "Reviewer_W9Zr", "summary": "This work improves large view synthesis model (LVSM) by compressing the multi-view tokens with Kmeans clustering. Previous LVSM attends on tokens of all input views or globally encoded tokens, which leads to expensive computation or pool quality due to too much or few condition tokens. This work cluster the pre-view token into a fixed set and design an attention network to refine the clustered tokens. The clustered tokens offer the decoder a smaller but diverse set token to render novel-view. Experimental results show good tradeoff of the speed and quality by varying the cluster size.", "questions": "1. Unfair evaluation. The proposed method use more images for training and testing comparing to the other baselines (L181-184). The result with aligned number of training images should be reported either by ablation of the proposed method or by increasing the number of training images of baselines. If being able to train on more views is one merit of the proposed method, the paper should explain and discuss the reason. The set of testing images should be aligned for all methods. It's not reasonable to compare numbers evaluated on different set of images.\n\n2. Paper presentation. In the method section, there is no need to filled in the hyperparameters with specific numbers if the proposed method can only work at that specific setup. Like the image resolution can be other than 256x256 so it's better to just state HxW. The figure of the Neural Condensation part in Fig.1 is hard to understand. One good figure example is VGGT to illustrate the operation of intra and inter group attention.", "limitations": "One limitation missing discussion is that the fixed number of clusters may not have good speed-quality tradeoff for different scenes. An adaptive strategy can be developed to automatically or dynamically determine the number of cluster based on the training view observations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The proposed method serves as a simple but effective efficiency improvement on the recent interesting topic: large novel-view synthesis model with minimum 3d inductive bias. The proposed clustering strategy to aggregate multi-view tokens is a reasonable extension, which is expected to give better speed-quality tradeoff and the experimental results by this work fit this expectation. Overall, I view this work as a good increment on existing trend.\n\nOn the other hand, there is also nothing very special that prompts me to give high rating to this work. Given the concerns I have in the Questions sections, I currently leans to give negative rating.", "was_revised": true, "final_justification": "My main concern about the evaluation is addressed. The discussion and the new results in the other responses are really insightful. Overall, it's a technical solid paper, the novelty and improvement is sufficient so I increase my rating from negative to positive.", "rebuttal": "We thank the reviewer for the detailed feedback.\n\n**Q1: Unfair Evaluation.**\n\nWe are sorry for the confusion. We use the same number of input views as the baselines when evaluating performance. Details of the comparison setup are provided in the baseline section (L 201–205). L181-184 describes the input setting of the original papers and leads to confusion. We will fix this in the revision.\n\n- **RealEstate10K:** For LVSM, we retrained their model from scratch using the same 4‑view input setting as ours (L 207). For MVSplat and DepthSplat, which predict Gaussian splats per pixel, the storage size grows with the number of input views and resolution. While we also provide experiments with the same input view settings on DL3DV for direct comparison, here on RE10K we use their official 2‑view checkpoints to examine **PSNR at the same storage size**, placing their storage in a comparable range.\n- **DL3DV**: Our 4-view and 6-view models are evaluated directly against the publicly available 4-view and 6-view checkpoints of MVSplat and DepthSplat (L211), ensuring fair evaluation settings.\n\n**Q2: Paper presentation.**\n\nThe footnote on page 4 explains the resolution settings. In the revision, we will use explicit hyperparameter notations (e.g., H×W) for clarity. The non-centroid tokens do not participate in the inter-cluster self-attention in our design, so the VGGT-style visualization doesn’t apply directly to our case. We will improve the clarity in the revision. \n\n**Limitations: Can not automatically select the number of clusters for different scenes.**\n\nWe thank the reviewer for raising this interesting discussion point. Our design is motivated by scenarios with predefined compute or storage constraints, such as a target FPS or GPU memory limit. Nonetheless, our system can be readily extended to automatically select the number of clusters (i.e., tokens). One simple approach is to encode the scene using multiple values of K and select the one that satisfies a desired validation PSNR threshold. A more elegant solution—such as learning to predict K during the forward pass based on a target PSNR—remains an interesting direction for future work."}, {"review_id": "EtzrjdjhCa", "reviewer": "Reviewer_6dxz", "summary": "This paper introduces CLiFT (Compressive Light-Field Tokens), a new token-based scene representation for compute-efficient and adaptive neural rendering. The method targets explicit reconstruction-free novel view synthesis (NVS), enabling a single trained model to produce high-quality renderings while flexibly trading off between data size, rendering speed, and visual fidelity. CLiFTs are created by first encoding multi-view images into dense tokens, followed by latent-space K-means clustering and a condensation network to compress these tokens into a smaller representative set. A compute-adaptive transformer renderer then uses a selected number of CLiFTs to synthesize novel views. Extensive experiments on RealEstate10K and DL3DV show that CLiFT achieves competitive rendering quality with fewer tokens or storage than baselines like LVSM, MVSplat, and DepthSplat. The paper highlights the flexibility of CLiFTs in handling different compute budgets and includes ablation studies that validate the importance of its components.", "questions": "1. Could the authors provide results comparing CLiFT to both the original (full-capacity) LVSM checkpoints and the decoder-only LVSM variant? This would help quantify the absolute performance improvement of CLiFT over the state-of-the-art and clarify whether CLiFT’s advantages persist under stronger baselines. Including these results would strengthen the evidence for CLiFT’s contribution.\n\n2. Can the authors report rendering FPS or latency metrics for MVSplat and DepthSplat under the same hardware conditions used for CLiFT? This would contextualize CLiFT’s rendering efficiency and its practical deployment value, especially for real-time or resource-constrained applications.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The proposed CLiFT framework introduces a compact, token-based light field representation that enables dynamic control over rendering quality, speed, and memory footprint. This feature is lacking in many prior approaches.\n\n2. Unlike baselines that require retraining for different compute budgets or scene complexities, CLiFT uses a single model that generalizes across different token counts and rendering settings.\n\n3. Experiments on RealEstate10K and DL3DV show that CLiFT achieves similar or better visual quality (e.g., PSNR) with significantly fewer tokens, demonstrating strong compression capability.\n\n4. The inclusion of detailed ablation studies validates the importance of the latent K-means and neural condensation stages, and shows how each component contributes to rendering quality and efficiency.\n\n5. The method is well-documented with a clean transformer-based architecture, and training/inference pipelines are described in sufficient detail to support reproducibility.\n\n**Weaknesses**\n\n1. While the paper retrains LVSM in a smaller, fairer configuration, results from the original high-capacity LVSM checkpoints should also be presented to contextualize the performance gap. Moreover, only the encoder-decoder variant is compared, despite decoder-only LVSM achieving better results in the original work.\n\n2. Rendering speed (FPS) is compared primarily across CLiFT variants, but missing comparisons to feed-forward baselines like MVSplat and DepthSplat weakens the claim of compute-efficiency, especially since those are designed for real-time rendering performance.\n\n3. Despite compression, the total number of tokens (e.g., 4096 for storage and rendering) and the transformer backbone may still be prohibitive for low-latency or mobile deployment. In addition, more input views will also significantly increase the number of tokens under the dense token setting of this paper.", "was_revised": true, "final_justification": "The authors have thoughtfully addressed my concerns, and the newly incorporated information enhances the clarity and understanding of the proposed method. In light of its potential to inspire further research in this area, I am pleased to raise my rating to accept.", "rebuttal": "We thank the reviewer for the constructive comments and the questions.\n\n**W1&Q1: Compared with LVSM decoder-only and full capacity LVSM checkpoints.**\n\n- **Compared with a decoder-only LVSM.** We trained a decoder-only LVSM model using 12 blocks and the same hyperparameter settings as our method. While it achieves strong rendering metrics due to having more decoder layers, it is extremely slow, running at only 8.7 FPS. In practice, it is **5–15× slower** than our CLiFT renderer and requires **2–5× more GFLOPs**, indicating substantially higher computational cost. As a result, it is unsuitable for real-time rendering on standard GPUs or mobile devices.\n    \n    \n    |  | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FPS ↑ | GFLOPs ↓ |\n    | --- | --- | --- | --- | --- | --- |\n    | LVSM-D | 30.83 | 0.919 | 0.088 | 8.7 | 194.2 |\n    | Ours | 29.55 | 0.906 | 0.100 | 54.3 | 70.6 |\n- **Compared with the full capacity LVSM.** The official “full capacity” LVSM decoder‑only model has 24 transformer blocks (≈2× our 12‑block model) and is trained with 2 input views. For reference, we run inference on RE10K with 4 input views using this full capacity checkpoint (no retraining). Note that this checkpoint is trained for 2‑view inputs but evaluated with 4 views, so the extra views are not fully exploited. A 24-block decoder‑only LVSM trained and tested with 4 input views would achieve higher performance.\n    \n    \n    |  | PSNR ↑ | SSIM ↑ | LPIPS ↓ |  |\n    | --- | --- | --- | --- | --- |\n    | LVSM-D (24 blocks) | 30.16 | 0.920 | 0.082 |  |\n\n**W2&Q2: Compared with rendering FPS with MVSPlat and Depthsplat.**\n\nWe tested splatting-based methods on the same hardware, and they can achieve 254.9 FPS. In general, they are significantly faster than reconstruction-free methods that use a neural renderer (LVSM and CLiFT), and this is exactly the main advantage of having explicit 3D representations. We will revise the main paper to include these results.\n\n**W3: Prohibitive for low-latency or mobile deployment as the number of input images increases.**\n\nWe respectfully disagree that this is a limitation of our method. CLiFT supports adjusting the number of tokens at inference time, enabling better speed–quality trade-offs than other reconstruction-free methods, such as LVSM. \n\nAs shown in Table 1 of the main paper, even with more input images, our rendering pipeline can select only the relevant subset of tokens for the target view, thereby significantly reducing computation while maintaining quality. This advantage would be more critical for larger scenes with more input views, as the necessary number of CLiFTs for a given query view does not increase with the scale of the scene or the number of input views.\n\n**Paper Formatting.**\n\nWe thank the reviewer for catching the typos, and we will do a careful pass in the revision to fix all these issues."}]

Author rebuttal:

Meta-review: This paper received uniformly positive reviews. It presents an interesting idea that builds upon an LVSM-like architecture by introducing compressive tokens, thereby enabling computationally efficient rendering. The method demonstrates promising trade-offs between rendering quality and efficiency, which the reviewers found compelling.

Explicit 3D representations, such as 3DGS, enable fast rendering but often with somewhat lower visual quality, whereas LVSM-like models achieve superior rendering quality at the cost of high computational expense. This paper is well positioned in this context, as it proposes a direction that narrows the gap by retaining much of the rendering quality while improving efficiency.

Overall, this paper makes a valuable contribution by advancing an important line of research and opening new opportunities for both explicit and implicit 3D representations. I support its acceptance.
Decision: Accept (spotlight)