Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The presented work describes the extension of Plenoxels [1] to temporal sequences. Compared to fitting one Plenoxel grid per frame, the proposed method improves memory and training time requirements significantly, by only learning to estimate the difference between adjacent frames. 

The paper claims the following technical contributions
 - A time and memory-efficient extension of Plenoxels to dynamic scenes by fitting frame-to-frame differences.
 - Only training in a "narrow band" where scene motion is expected
 - Multi-scale training based on a "pilot model"
 - Compression of frame-to-frame differences
 - A new multi-view dataset captured with a rig of 13 Azure Kinect cameras

[1] Yu et al. Radiance Fields without Neural Networks. 2022

strengths_and_weaknesses: The paper addresses an important problem, the long training times of state of the art neural 3D video reconstruction [2], whose authors report 1,300 GPU hours of training time for a 10 second clip. The authors show how to adapt Plenoxels to video reconstruction and demonstrate training and inference times that are significantly faster than [2], and still achieve reasonable reconstruction quality. 

The quantitative results reported in Table 1 are convincing. Visually, the results on the dataset of [2] are good, although they don't achieve the quality of [2]. The results on the proposed "Meet Room" dataset exhibit more artifacts (e.g. translucent foreground and streak aftifacts along edges), which might be caused by a wider camera baseline, a less textured background, and might also be the results of some of the performance optimizations (e.g. narrow band finetuning) that the authors employ.

Overall, the paper is well structured and the discussion of related work is comprehensive. However, the discussion of the technical contributions is unfortunately lacking clarity:

## Pilot model

The authors propose to fine-tune step $N+1$ first on a lower-resolution model, before initializing the higher resolution model from the pilot model. This seems very similar to the coarse-to-fine optimization described by Plenoxels. Could the authors comment on why this should be considered a novel contribution?

## Narrow Band Finetuning

It is not clear to me why the "dense grid" column in Figure 2 fails the way it does. The failure of the "sparse grid" column is plausibile, since it is using the occupancy map of the original frame. I would expect training using the dense grid to work better (although) than selecting the narrow band. Why does using the narrow band increase quality?

The paper would benefit from a figure analyzing the performance benefit of narrow-band finetuning and the Pilot model. 

## Difference-based compression

 - Table 1 should include the model size for all techniques.
 - The compression steps in 5.2 (SH thresholding, float16 coefficients, ZLIB) are sound contributions, but are not sufficiently detailed. The paper should detail the model size reduction of each step.
  - How big is the model size of the sparse fine-tuned voxels before compression? Line 155 suggests the finetune output has a size of 5.4 MB, but this equals the size of the fully compressed model reported elsewhere.
  - How are the voxels selected for narrow band tuning different from the voxels that are included in difference-based compression? The paper would benefit from a visualization of both selection criteria, and a measurement of any extra compression gained by Eq. 6.



[2] Li et al. Neural 3D Video Synthesis from Multi-view Video. 2021

questions:  - In Eq. 2, should it be Y(d)?

limitations: Limitations are adequately discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a grid-based method to reconstruct a streaming radiance field for dynamic scenes. More precisely, this paper formulates the dynamic scene modeling problem in an incremental learning paradigm where per-frame model difference is trained to complement the adaption of a base model (the first frame). With the above design, the proposed method can efficiently handle on-the-fly video sequences without significant storage overhead. The optimization can be further accelerated via a curriculum learning strategy. Experiments on challenging video sequences demonstrate that the proposed method achieves 1500x speedup over state-of-the-art implicit models.
strengths_and_weaknesses: Strengths:
This paper proposes a novel approach of using explicit grids for reconstructing streaming radiance fields, which significantly improves over exiting implicit methods, enabling learning on-the-fly videos in a streaming manner. Also, by only storing the difference for each frame, the complete model can be efficiently compressed to reduce the memory costs brought by explicit grid models. 
The pilot model further improves the training efficiency by first learning a low-resolution pilot model, and then using the guidance from the pilot model to learn in full resolution.
Therefore, the proposed methods achieve a good balance between explicit and implicit methods in terms of learning efficiency and storage.

Weaknesses:
Although the output results show improvement over the baselines both qualitatively and quantitively, we can still see obvious artifacts in the supplementary video. There are scenes the people get transparent or overlapped in the space. The authors need to add more discussions and explanations on the artifacts and check if these phenomena are introduced by imperfect learning in model differences and error accumulation.
This paper did not have a comparison with implicit models which learns deformation between frames (e.g., NeRFies, D-NeRF, DynamicNeRF, HyperNeRF, Neural Actor, AnimatableNeRF, etc). All these methods also use the inductive bias of local continuity and use a neural network to learn the deformation field for different frames. These methods may have longer convergence (due to implicit based), however, may achieve better quality and fewer artifacts for moving objects.

Missing references:
1. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields
2. Nerfies: Deformable neural radiance fields
3. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
4. D-NeRF: Neural Radiance Fields for Dynamic Scenes
5. Neural Sparse Voxel Fields
6. Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
7. Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies


questions: (1)	Will the first frame always be optimal as the base model? Have you compared it with using different frames (middle, last) and learning the relative difference? Will this method be used with multiple base frames? For example, one alternative is to learn full grid models at several “key frames” and all other frames can be seen as interpolations from these key frames. I expect this would improve some performance especially when the objects have large movements.
(2)	How are the narrow bands hyper-parameters (rou_d and rou_e) selected for different videos? I assume the narrow bands correspond to the velocity of the object in the video, and inaccurately choosing a too small band will cause errors. Is it possible to adaptively assign the narrow bands for different parts?
(3)	Can a similar approach be used in other hybrid multi-resolution representations such as Instant-NGP? What will be the major difficulties?

limitations: This paper did not discuss any potential negative societal impact. Since the proposed method can be used to reconstructing/learning any Internet videos, it may be better to add the discussion on the social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Neural radiance fields are not directly adaptible to dynamic scenes that have a changing representation that maps a 3D position to its captured image. The paper proposes a novel approach for capturing the radiance fields of video sequences. The radiance field of the base model (first frame) is learned explicitly using a voxel-grid parameterization. Consequently, only the frame-to-frame differences in the grid representation are learned. The difference between frame-grids is identified using a narrow band focused around the contour of the dynamic segments of the scene. Furthermore, a compression strategy is proposed to handle the explosion in size caused by the use of explicit voxel grids. Finally a multi-scale learning paradigm is introduced for improved optimization during learning the proposed model. 

strengths_and_weaknesses: Strengths:
The paper is well written and the exposition is clear. 
The narrow-band tuning strategy proposed in the paper is intuitive and properly motivated within the context of local changes in adjacent dynamic video frames. 
The compression strategy based on the mask difference is also an interesting contribution of the paper and a useful construct for dynamic video representation. 

Weaknesses:
The pilot model guidance framework, on the other hand, requires better exposition and motivation. It is clear that multi-scale parameterization has a computational impact on problems that use dense grid representations. However, the qualitative impact may be larger, and possibly even negative. For example, there needs to be more discussion on the impact of coupling the pilot model approach with the narrow-band tuning. Given the dynamism of the scene and the arbitrary position of the target viewpoint, it is not difficult to imagine that full scale training is qualitatively superior to a multi-scale approach that might suffer from loss of detail.
questions: Apart from the computational benefit, what would be a theoretical motivation to compare the pilot model base training as opposed to training at full scale.
limitations: No, the authors have not adequately discussed limitations within the paper.
review_title: null

Meta-review: metareview: All reviewers agree that the paper should be accepted, despite some flaws that can be addressed in future work
confidence: Certain
Decision: Accept