Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper builds a novel indoor perception dataset on radience fields, motivated by the recent advances of NeRF. Overall, this is an interesting and novel contribution since it exhibits a notable memory compression rate from the original dataset, and also provides a unified representation of 3D and 2D information. The paper is clearly written and well-organized. 
strengths: 1. The first perception dataset built on radience fields.
2. Significant memory reduction for the storge.
3. Unified representation for 2D and 3D input.
weaknesses: The title looks quite broad, because the paper only considers indoor scenarios, and benchmark three tasks. The authors are suggested to tone down their claims.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: See weakness.
review_title: A nice perception dataset and benchmark using radiance fields

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work explores the potential of using explicit radiance fields as a unified compressed format for 2d image sequences and 3d scenes. It presents two datasets of pretrained plenoxels of CO3D (objects) and ScanNet (indoor scenes), which are used for benchmark perception tasks such as 2d/3d classification and 3d semantic segmentation, through either rendering 2d images or directly using its feature grids. It hopes to inspire future work to build larger scale 3d datasets and train bigger vision models.
strengths: - large-scale training of plenoxels on co3d and scannet, effectively compressing the video sequences into *.plenoxels format
- comprehensive study of perception tasks on those pretrained plenoxels, shows reasonable good results
- background augmentation and camera manipulation can be enabled with neural rendering, which are non-trivial for traditional 2d/3d data format
weaknesses: - for nerf-like models to be real candidate to compress data (data format), we probably need more study on the rate-distortion-compute trade-off, which is lacking in this work. e.g. the compression scheme (quantization) is quite vanilla.
- the paper claims implicit representation dataset, but plenoxels are more like explicit representation, please make it more clear in abstract and paper
- we can think of perception on trained plenoxels as two-step processes: first compress images into plenoxels, then use the compressed features for perception tasks. One can argue the compression step may lose important information for downstream perception tasks. Thus end-to-end perception models should still achieve better accuracy.

correctness: - fig 3: there is 4-5% accuracy@1 gap for models trained on co3d and test on co3d / perfception-co3d, which indicates that plenoxels lose info for classification tasks? Also the gap is larger for models trained on PeRFception-Co3D. It makes me doubt the claim "PeRFception-Co3D contains the same information as the original CO3D dataset"
- fig4: why use both density and SH makes 3d classification accuracy drop for 3dResNet101?
- table 2 & 3: scannet compression ratio is much larger than co3d, with much worse psnr (23 vs 28). This is expected from rate-distortion trade-off. The paper should make it clear. Can you adjust the trade-off for scannet? e.g. better psnr with larger model sizes.
clarity: - can you provide more details on the visuals in fig 2 (e)? what is the color for?
- Table 2 caption: C for diffused color? what do you mean?
- line 194 - saturate GPU memory?
- maybe put training time in the main time, 30min per scene
relation to prior work: maybe add a few papers related to compression with implicit neural repr. e.g. 
- Dupont, Emilien, et al. "Coin: Compression with implicit neural representations." arXiv preprint arXiv:2103.03123 (2021).

quite relevant work (maybe concurrent)
- Wang, Clinton J., and Polina Golland. "Deep Learning on Implicit Neural Datasets." arXiv preprint arXiv:2206.01178 (2022).
additional feedback: maybe provide some comparison to the stoa results on those perception tasks benchmarked in this paper
review_title: comprehensive study on perception tasks on one type of pretrained explicit radiance fields

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work introduces the recent implicit 3D scene representation method for constructing datasets. This new representation can unify 2D images and 3D scene datasets into one compact format. Specifically, the authors evaluate this idea by building two datasets upon the CO3D and ScanNet datasets. Then the authors tested the accuracy with multiple ResNet variants to show lower memory and the availability of the proposed datasets.
strengths: * The first work to use implicit representation for a large-scale dataset.
* The compression rate for the ScanNet dataset is impressive.
* In the proposed dataset, more data augmentation can be performed (e.g., render novel views, substitute the background).
weaknesses: I like the paper's motivation, and I believe there is some potential for an implicit representation of the dataset. 
However, I have some concerns about the effectiveness of the proposed method for practical usage.

* Though the data representation is unified, it seems it can not help to simplify the training process or boost performance. Especially for 2D tasks, the images should be rendered first (or generated during the training process). The rendering may also slow down the training process, which the authors do not report. There is no such step if using the original dataset. Besides the rendering time, the reconstruction error is also introduced. For the 3D task, this problem is alleviated by learning from the implicit representation directly, but it will present another problem. The network architecture of the existing method should be re-designed to adapt to this data format.

* Data compression is the core advantage of the implicit representation. But the result on the "object-centric" dataset is not good enough. As the image quality will decrease, I wonder if it is worth this trade-off.

* With background augmentation, the accuracy result (test on the original test set)  is still lower than trained with the original dataset, as shown in Fig.3.

* With the drawback of implicit representation, it seems hard to represent synthetic datasets like 3D-FUTURE/3D-FRONT well. Specifically, the scene editing (e.g., material, lighting) and domain randomization technique are not supported. Also, the advantage of generating accurate ground truth (e.g., noise-less depth) is hard to preserve.

Some minor concerns:
* ResNet is used for all experiments. Experimenting with some SOTA methods in the classification and semantic segmentation task is better.

* Fig.3 needs more discussion. It is desirable that the result trained on PeRFception has a lower accuracy than the original dataset, showing a domain gap caused by reconstruction error. However, when evaluating using the test set of PeRFception, the result (i.e., shallow green and blue) that trained on PeRFception (expected to have a lower domain gap with the test set) is still lower than the result on the original test set (i.e., dark green and blue). Please discuss.

* The choice of Plenoxels needs more discussion. The author says it has explicit geometry and consistent features, but other methods like INGP can also obtain geometry and be trained faster.
correctness: Yes.
clarity: Overall, the paper is clear and well-structured.

relation to prior work: Some minor problems:

* "Co3D" -> "CO3D"
* Table.1 only shows the specs of object-centric datasets, a similar comparison for the scene-centric dataset is preferred.
Also, the 3D-Future dataset is listed in the table, but not discussed in the main text.
* L117: "Both datasets do not include camera parameters". The NYUv2 dataset seems to contain camera parameters, as said on its [website](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html), "camera_params.m - Contains the camera parameters for the Kinect used to capture the data.".
additional feedback: 
* The authors state in L311 that there are discussions on the limitation in the supplementary document. However, I did not find it.

* Table.2: the abbreviation "pcl" for point-cloud should be noted in the caption.

* L194: It seems a bit confusing, and more details are needed. How many frames? Why need to saturate the CPU memory?
review_title: A new dataset representation using neural radiance field, but the potential for practical usage is limited.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposed a dataset called PeRFception for perception tasks such as classification and segmentation. The data is generated by a recent method called Plenoxels. This method allows a high memory compression rate while still maintaining 2D and 3D information.
strengths: 1. Table 1 shows a clear comparison with existing datasets.

2. The proposed dataset can be used for 2D image classification, 3D object classification, and 3D scene semantic segmentation.
weaknesses: 1. This paper creates an impression that the paper is good at data compression. I know this is not the intended purpose of this paper, but the paper reads like this.

2. The authors provided google drive links for URLs. People in some countries cannot access services provided by Google and hence cannot access this dataset.
correctness: The dataset construction is technically sound.
clarity: The paper reads fine, but the exposition can be improved.
relation to prior work: Related work is adequate.
additional feedback: Please see the weakness section above.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes the first large-scale neural radiance based dataset, which is generated by the Plenoxels [14]. Authors study visual perception tasks that directly process the implicit representation in their dataset. The pipeline for generating implicit is ready-to-use and provided for new data generation. 

strengths: * This paper uses the radiance field, generated by a  NeRF-based method Plenoxels [14], as a novel data structure to replace traditional 2D or 3D data structure. It is an interesting attempt, since constructing a complete 3D structure of a scene is laborious. Directly generating implicit representation from input images for 3D perception will be really useful.
* Two datasets, PeRFception-Co3D and PeRFception-ScanNet, are introduced. The implicit representation gives another choice for 3D perceptron.
* The experiment shows that previous 2D/3D methods can directly apply on the novel data structure and the performance is well.
weaknesses: * The dataset generation is totally based on the method Plenoxels [14]. All data is directly stored from the intermediate data generated by the  Plenoxels, e.g. spherical harmonic coefficients, densities.  The contribution of this paper is not substantial enough. 
* The density and color information are also generated by other NeRF-based methods [5, 6, 10, 11]. The 3D structure like points can be generated by Point-NeRF (https://arxiv.org/abs/2201.08845). The reason why we have to use the implicit representation of Plenoxels is not given. A thorough comparison between PeRFception and implicit data generated by methods mentioned before is necessary.
* The statement in L249, "PeRFception-Co3D contains the same information as the original Co3D dataset", can not be summarized from the results (Fig. 3). ResNet trained on PeRFception-Co3D has obvious performance drop compared with ResNet trained on the Co3D, when the test set is Co3D. Besides, ResNet trained on PeRFception-Co3D also has performance drop, when test set changes from Co3D to PeRFception-Co3D. I think there exists information loss after transforming original data to implicit one. A discussion about this is necessary.
*  A performance comparison between methods trained on ScanNet and  PeRFception-ScanNet is necessary, similar to Fig. 3.
correctness: The claims in the submission are correct. The construction of the dataset and the instruction for using the dataset are clear.
clarity: The paper is written clearly.
relation to prior work: Relation to most of prior work has been discussed in the relation work. The relation to prior work Plenoxels needs further discussion.
additional feedback: The main concern of mine is that the generation procedure of the dataset totally depends on Plenoxels. I will adjust my rating after the discussion with other reviewers and AC.
review_title: The idea of using radiance field as a new data structure is interesting, but the contribution of data generation is not substantial. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper focuses on the datasets in implicit 3D representation. This work creates the first large-scale implicit representation datasets called PeRFception for perception tasks, including classification and semantic segmentation in NeRF. What's more, the paper provides a ready-to-use pipeline to generate the implicit datasets automatically.
strengths: The paper introduces the first large-scale implicit datasets that can be readily used in downstream perception tasks. The dataset called PeRFception is both a timely and relevant contribution to the broader research community. Additionally, it shows a significant memory compression rate from the original dataset. Such a compression rate is beneficial for efficient research. The third strength of the paper is the pipeline to generate the implicit datasets automatically, which motivates to generate a very large scale 3D dataset in future.
weaknesses: 1. Minimal discussion of the ethical or societal implications
2. Lack the comparisons of rendering qualities between PeRFception dataset and current popular 3D datasets in NeRF research. 
correctness: 1. The dataset is constructed in a sound way. 
2. The problem definition and methodology for evaluation appear correct. 
3. The experiment is well-designed and appropriate.
clarity: The paper is well written and easy to follow.
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: It would be better to provide the source code and full data link.
review_title: Good paper that introduces the first large-scale implicit representation datasets for perception tasks

Meta-review: metareview: Authors propose a novel large-scale implicit representation dataset for perception tasks, called the PeRFception dataset, which consists of two parts that incorporate both object-centric and scene-centric scans for classification and segmentation. The dataset obtains significant memory compression rate while containing both 2D and 3D information in a unified form.

Some important pros from reviews:
+ the first large-scale perception dataset built on radiance fields, implicit representation as another choice for 3D perception
+ significant and impressive storage reduction / compression
+ comprehensive comparison with existing datasets

Authors provided a strong rebuttal, and reviewers agreed that it addressed their concerns. I believe that the dataset is a relevant contribution to the broader research community. Hence, I recommend to accept the paper.
Authors should make sure to incorporate suggestions from reviews into their final manuscript.



confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept