Title: CLiFT: Compressive Light-Field Tokens for Compute Efficient and Adaptive Neural Rendering
Venue: NeurIPS 2025
Accepted: yes
Keywords: Novel View Synthesis

Scene Representation

Compression
Abstract: This paper proposes a neural rendering approach that represents a scene as "compressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene. CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent a scene or render a novel view with one trained network. Concretely, given a set of images, multi-view encoder tokenizes the images with the camera poses. Latent-space K-means selects a reduced set of rays as cluster centroids using the tokens. The multi-view ``condenser'' compresses the information of all the tokens into the centroid tokens to construct CLiFTs.  At test time, given a target view and a compute budget (i.e., the number of CLiFTs), the system collects the specified number of nearby tokens and synthesizes a novel view using a compute-adaptive renderer. trained to handle a variable number of tokens.  Extensive experiments on RealEstate10K and DL3DV datasets quantitatively and qualitatively validate our approach, achieving significant data reduction with comparable rendering quality and the highest overall rendering score, while providing trade-offs of data size, rendering quality, and rendering speed.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CLiFT (Compressive Light-Field Tokens), a new token-based scene representation for compute-efficient and adaptive neural rendering. The method targets explicit reconstruction-free novel view synthesis (NVS), enabling a single trained model to produce high-quality renderings while flexibly trading off between data size, rendering speed, and visual fidelity. CLiFTs are created by first encoding multi-view images into dense tokens, followed by latent-space K-means clustering and a condensation network to compress these tokens into a smaller representative set. A compute-adaptive transformer renderer then uses a selected number of CLiFTs to synthesize novel views. Extensive experiments on RealEstate10K and DL3DV show that CLiFT achieves competitive rendering quality with fewer tokens or storage than baselines like LVSM, MVSplat, and DepthSplat. The paper highlights the flexibility of CLiFTs in handling different compute budgets and includes ablation studies that validate the importance of its components.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed CLiFT framework introduces a compact, token-based light field representation that enables dynamic control over rendering quality, speed, and memory footprint. This feature is lacking in many prior approaches.\n\n2. Unlike baselines that require retraining for different compute budgets or scene complexities, CLiFT uses a single model that generalizes across different token counts and rendering settings.\n\n3. Experiments on RealEstate10K and DL3DV show that CLiFT achieves similar or better visual quality (e.g., PSNR) with significantly fewer tokens, demonstrating strong compression capability.\n\n4. The inclusion of detailed ablation studies validates the importance of the latent K-means and neural condensation stages, and shows how each component contributes to rendering quality and efficiency.\n\n5. The method is well-documented with a clean transformer-based architecture, and training/inference pipelines are described in sufficient detail to support reproducibility.\n\n**Weaknesses**\n\n1. While the paper retrains LVSM in a smaller, fairer configuration, results from the original high-capacity LVSM checkpoints should also be presented to contextualize the performance gap. Moreover, only the encoder-decoder variant is compared, despite decoder-only LVSM achieving better results in the original work.\n\n2. Rendering speed (FPS) is compared primarily across CLiFT variants, but missing comparisons to feed-forward baselines like MVSplat and DepthSplat weakens the claim of compute-efficiency, especially since those are designed for real-time rendering performance.\n\n3. Despite compression, the total number of tokens (e.g., 4096 for storage and rendering) and the transformer backbone may still be prohibitive for low-latency or mobile deployment. In addition, more input views will also significantly increase the number of tokens under the dense token setting of this paper.\n\nQUESTIONS:\n1. Could the authors provide results comparing CLiFT to both the original (full-capacity) LVSM checkpoints and the decoder-only LVSM variant? This would help quantify the absolute performance improvement of CLiFT over the state-of-the-art and clarify whether CLiFT’s advantages persist under stronger baselines. Including these results would strengthen the evidence for CLiFT’s contribution.\n\n2. Can the authors report rendering FPS or latency metrics for MVSplat and DepthSplat under the same hardware conditions used for CLiFT? This would contextualize CLiFT’s rendering efficiency and its practical deployment value, especially for real-time or resource-constrained applications.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work improves large view synthesis model (LVSM) by compressing the multi-view tokens with Kmeans clustering. Previous LVSM attends on tokens of all input views or globally encoded tokens, which leads to expensive computation or pool quality due to too much or few condition tokens. This work cluster the pre-view token into a fixed set and design an attention network to refine the clustered tokens. The clustered tokens offer the decoder a smaller but diverse set token to render novel-view. Experimental results show good tradeoff of the speed and quality by varying the cluster size.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed method serves as a simple but effective efficiency improvement on the recent interesting topic: large novel-view synthesis model with minimum 3d inductive bias. The proposed clustering strategy to aggregate multi-view tokens is a reasonable extension, which is expected to give better speed-quality tradeoff and the experimental results by this work fit this expectation. Overall, I view this work as a good increment on existing trend.\n\nOn the other hand, there is also nothing very special that prompts me to give high rating to this work. Given the concerns I have in the Questions sections, I currently leans to give negative rating.\n\nQUESTIONS:\n1. Unfair evaluation. The proposed method use more images for training and testing comparing to the other baselines (L181-184). The result with aligned number of training images should be reported either by ablation of the proposed method or by increasing the number of training images of baselines. If being able to train on more views is one merit of the proposed method, the paper should explain and discuss the reason. The set of testing images should be aligned for all methods. It's not reasonable to compare numbers evaluated on different set of images.\n\n2. Paper presentation. In the method section, there is no need to filled in the hyperparameters with specific numbers if the proposed method can only work at that specific setup. Like the image resolution can be other than 256x256 so it's better to just state HxW. The figure of the Neural Condensation part in Fig.1 is hard to understand. One good figure example is VGGT to illustrate the operation of intra and inter group attention.\n\nLIMITATIONS:\nOne limitation missing discussion is that the fixed number of clusters may not have good speed-quality tradeoff for different scenes. An adaptive strategy can be developed to automatically or dynamically determine the number of cluster based on the training view observations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present CLiFTs (Compressed Light-Field Tokens), a novel and data-efficient neural scene representation that addresses the high memory demands of existing neural rendering methods. Built on a Transformer encoder-decoder architecture, the framework introduces an encoding stage that constructs CLiFTs by applying K-means clustering in latent space to identify feature centroids from multi-view inputs. A proposed condensation network is then used to compress neighboring features around each centroid.\n\nThis approach enables significantly reduced memory storage compared to traditional feed-forward neural rendering pipelines, while still producing high-quality rendered images. The method is evaluated on two benchmark datasets, RealEstate10K and DL3DV, against three representative baselines, LVSM, MVSplat, and DepthSplat. CLiFTs demonstrate a good trade-off between rendering quality and data efficiency, particularly outperforming prior methods from the perspective of storage and computational cost.\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths\n- The paper is clearly written and easy to read. The target problem is well-motivated, and the evaluation metrics are appropriately chosen to show the effectiveness of the proposed method.\n- The proposed method shows strong performance in terms of data efficiency, outperforming existing baselines both quantitatively and qualitatively.\n\n> Weaknesses\n- The rationale for choosing a light field representation is not sufficiently discussed. It's unclear what specific advantages it brings over alternative representations, especially in comparison to the baselines.\n- There is limited discussion on potential issues with using K-means clustering in latent space. It would be helpful to include analysis or experiments addressing whether the method generalizes well across diverse scene types, e.g., across the scenes of varying complexity.\n- The condensation network is introduced as a core component, but details on its architecture and design choices are insufficient. More justification or ablation results would strengthen the quality of paper.\n\nQUESTIONS:\n- It would be helpful to include an experiment that evaluates how the method performs across scenes with varying complexity. Since both datasets used in the paper may contain scenes of similar visual complexity, it’s hard to tell how well CLiFT generalizes to more diverse scenarios, and its range of scenes' complexity that CLiFT can cover.\n\n- Including an ablation study that replaces the K-means clustering with simple alternative, such as grouping locally in patch style, and uses only the neural condensation component would be valuable to understand the proposed methods' contribution.\n\n- The paper could benefit from more explanation around the architecture and design choices of the condensation network. A brief motivation for the layer/module design and why certain decisions were made would help readers better understand how this component contributes to the overall method.\n\n- Does the model maintain good performance when more input views are provided? It would be good to include an experiment that tests the model’s generalization ability with a larger number of input views, especially if the number of CLiFT tokens is reduced at the same time. Showing the upper-bound of models' performance within the trade-off of the number of input views and tokens could further highlight its strength as a data-efficient approach.\n\nBased on the authors' rebuttal, I'm willing to revise my initial rating.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CLiFT, a novel approach for neural rendering that represents a scene using a set of \"compressive light-field tokens.\" The primary contribution is a framework that is both data-efficient and adaptive, allowing a single trained model to trade-off between storage size, rendering quality, and computational speed. The method consists of a three-step construction process: 1) a multi-view Transformer encoder converts input images and poses into initial \"Light Field Tokens\" (LiFTs); 2) a latent-space K-means algorithm clusters these tokens to select a compact set of representative \"centroid\" rays, effectively reducing geometric and appearance redundancy ; and 3) a \"neural condenser\" further compresses information from all tokens in a cluster into these centroids, creating the final CLiFTs. At inference time, a novel view is synthesized by a Transformer decoder that uses a subset of these CLiFTs, selected based on proximity to the target view's rays. Experiments on the RealEstate 10K and DL3DV datasets show that CLiFT achieves comparable or better rendering quality than state-of-the-art methods like LVSM and DepthSplat, but with significantly smaller data sizes.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n1. Adaptive Representation: The most compelling strength of CLiFT is its ability to operate across a spectrum of data sizes and compute budgets with a single trained model. Users can dynamically adjust the number of tokens for storage or rendering to balance quality and speed. This is a significant practical advantage over existing methods, which typically require retraining a separate model for each desired data size or quality level. \n\n2. High Compression Efficiency: The paper demonstrates impressive data compression. CLiFT achieves comparable PSNR to MVSplat and DepthSplat with 5-7x less data and to LVSM with 1.8x less data.  This efficiency is achieved through the intelligent reduction of redundant information, making the representation highly compact.\n\n3. Principled Token Selection and Condensation: The use of latent-space K-means to identify and select representative rays is a more principled approach to compression than uniform or random sampling. The ablation studies in Figure 3 clearly show that this selection strategy is critical for maintaining quality at high compression rates. The subsequent neural condensation step further refines these tokens, leading to consistent performance gains.\n\n- Weakness:\n1. Generalization Limitations: As acknowledged by the authors, the method struggles when faced with camera motions that deviate significantly from the training distribution or when target views are not well-covered by the input images.  This highlights a potential weakness in generalization, which is a common challenge for learning-based novel view synthesis methods.\n\n2. Multi-Stage Training Complexity: The training pipeline is quite complex, requiring a two-stage process where the multi-view encoder is trained first, then frozen, followed by a second stage to train the condenser using pre-computed K-means assignments.  This separation may introduce optimization challenges and could be a barrier to reproducibility and future extensions compared to a single, end-to-end trainable system.\n\nQUESTIONS:\nRegarding the Two-Stage Training: The paper describes a two-stage training process where the encoder is trained first and then frozen.  What is the motivation behind this design instead of end-to-end training? \n\nOn Failure Cases: The limitations section mentions that the method struggles with camera poses that are very different from the training data.  Is there any mechanism within the model to estimate the uncertainty or expected quality of a rendering for a given target pose? This would be very useful for practical applications.\n\nLIMITATIONS:\nYes, the authors discuss limitations in Section 5, pointing out failure cases related to out-of-distribution camera motions and insufficient view coverage in the input images.  This discussion is fair and provides a clear picture of the method's boundaries.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}