Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a unified framework for doing point cloud upsampling, denoising, and completion jointly. The proposed framework contains two modules: a point-to-voxel auto-encoder, and a Point Re-localization transformer. It outperforms single-task baselines on multiple benchmarks by a great margin. 
main_review: Note: I'll use the the word "multi-task" to refer to the problem setting in this work, in contrast to the single-task baselines. I understand that "multi-task" slightly abused here as one may argue that the the studied problem is a unified single task rather than three independent ones. 

Strengths:
1. The high-level idea makes lots of sense to me. Using voxelization to denoise and complete irregular point cloud is reasonable and smart. Besides, we can sample as many points as we want from a voxel grid to get a dense surface. I'm not 100% sure whether this idea has been studied before as I'm not an expert for these two tasks. If possible, the authors could further elaborate on this contribution.
2. The empirical results are strong. The proposed framework greatly out-performs single task baselines. 
3. Writing is mostly clear and proper ablation has been conducted.

Weakness:
1. The presented results are great, but the baselines are not very considerate. This paper is arguing that the three tasks should be unified and hence multi-task joint-learning baselines should be considered. The simplest setting is to ensemble all three baselines together in an optimized order. Alternatives contain training each baseline methods on all three tasks jointly. 
2. Following above point, the proposed method should also be evaluated for each single tasks to demonstrate advantages. For example,  I would be only curious about how good is the proposed framework for point completion if my point cloud isn't noisy. Does this method improved over specified point-completion network?  
3. I'm not fully convinced by the story of positional encoding. In the original form, the distance information is already expressed as the frequency of the signal. The magnitude is just redundant information. Why is it important? And what kind of effect it will cause precisely?  
4. Lots of details are unclear. I list several that confuses me most: 
    - How many voxels are processed in the transformer on average? Since the complexity of the attention function is $O(N^2)$, does the proposed framework suffer from big memory consumption?
    - For the tested dataset, especially the synthetic ones, how does the input noise/sparsity/incompleteness are created? ScanNet has lots of incomplete objects (eg., chair with no legs) but I don't find any qualitative results showing that the network can complete them, which is unsatisfactory. Also, regular synthetic noise (eg.,  Gaussian noise) are much simpler than real-world noise. How does the proposed framework deal with real-world noise? For example, MVS reconstructed point-clouds. Similar thing also holds for sparsity. It's unclear how does this framework work for real-world sparse point clouds (eg., the remote areas in Lidar data). If the authors think these are out-of-the-scope, then please clearly introduce the problem setting and limitations.
   - How is the dimension of $P_out$ decided? I thought arbitrary number of points can be generated. If this is true, how are the numbers in Tab.1 decided? I found these values of $P_out$ are confusing and they are not consistent with prior work.

Misc
1. The difference between sparse (point up-sampling) and incomplete (point-completion) is not very clear. Very specific definition should be made in the very early stage of the writing so that there is no ambiguity between the challenges that these two problems pose. Besides, the experimental results should clearly demonstrate how does the proposed framework address each sub-problem (above two+de-noising) individually. Unfortunately, these results are either missing or not very clear right now.
2. If find it mis-leading to term this task "point-cloud reconstruction". A maybe improper analogy is that we don't call joint image de-noising and super-resolution and image generation. If the authors really want to use the term reconstruction, maybe call it "conditioned point cloud reconstruction"? Or something else that is more accurate like "point cloud refinement"? I'm just making examples and throwing out random names. 
3. It's unclear to me what's the motivation of clustering (voxel hashing), and how important is this step? Why do we need a clustering before transformer?  And where are the cluster information used in the later framework? If I understand correctly, this is used because the voxel-to-point step only operates on center voxels and hence we want a locally augmented feature representation?

Minor issues:
1. Fig.2 input figure is barely visible. A better color-map should be used. 
2. 3D sparse Hourglass network is not a super popular network. It's also not implemented in popular sparse conv net framework (eg., Minkowski engine). Why this framework is adopted? Maybe the authors can share some related work to justify the advantage. It's not a common choice.

Suggestions:
1. In Tab.1, maybe the light color can be used to high-light the 2nd best among the baseline method? High-lighting the 2nd best among all including "ours" cannot fully demonstrate the advantages.
summary_of_the_review: Summary: My current rating is mainly based on two reasons: 1) The experimental results are strong and the idea of utilizing voxelization for de-noising/refining/densify-ing point clouds makes lots of sense to me. In addition, the proposed transformer works well for further improving the results and recovering points from voxels. 2) Even though the numbers are great, I do think the experiments can be improved. Multi-task baselines should be devised and compared. It's not enough to only demonstrate the proposed method out-performs single-task baselines. Moreover, fine-grained experiments should also be conducted to help us understand the performance on each single tasks. 

I'll also use this section to justify my ratings below. This paper is mainly proposing a new problem setting and develops a new framework correspondingly.  Technical wise, this new framework doesn't carry significant new components. It does a great job choosing the proper sub-modules,  combining and slightly improving them for the final tasks. However, most of these components existed in the literature. Empirical wise, the results in Tab.1 is exciting and the proposed framework greatly improves over singe-task baselines. More strictly speaking, single tasks and two-task (X--> score-PD) baselines. The qualitative results in Fig.5 are also encouraging and clearly show the advantages. 

I have worked in the field of point-cloud recognition and pretty familiar with the recognition frameworks.  However, I'm not an expert of point-cloud de-noising/completion/densify-ing, and not very familiar with the SOTA methods in these fields.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an end-to-end unified approach to solve the subtasks in point cloud completion. The main selling point of the paper is that combine two existing methods for point cloud densification  & denoising (Stage 1) and point cloud completion (Stage 2). The paper does extensive experiments on 3 datasets and compares to 3 different baselines. They achieve SOTA performance on these datasets. The use of sparse convolution to process point clouds is quite interesting as an autoencoder (termed as hour glass) aligns with the task at hand.
main_review: 

Strengths:-
1. The paper brings a new perspective solving two old problems in computer vision, and this is useful. I like the idea it is clean and simple.
2. The two-stage novel architecture is simple, but apart from the additional positional embedding I don’t there is any other addition here. But the stage1 and stage2 architectures have been used in the past to solve problems. My understanding is that the network majorly befits from stage1 as a lot of qualitative results concentrate on completion.
3. The numbers are better and change with voxel resolution but significantly better than the baseline approaches.
4. Qualitative examples are good, and show the performance improvement but are limited, and would like to see more of them in the supplemental (which is missing)





Weakness:-
1. This paper needs a quite few presentation changes to make the idea clear and easy flowing. The initial section of the method section of the paper is up until 3.1 is clear and well explained while 3.2 needs a better explanation. I still do not understand the exact intuition behind the amplitude-based positional embedding and logic is not very clear. 
2. The baselines are not explained well at all, I’m not sure if the comparison to the baselines is fair because the exact characteristics of the Snowflake-PC → Score-PD are not explained, and likewise for Dis-PU. These sections need rewriting.
3. There is no evaluation independently for Stage-1 all evaluations in the paper use Stage-1 and Stage-2. I think to justify the need for Stage-2 and see its impact it is important to set up some evaluation criteria for Stage-1 only. I understand that maybe doing the full evaluation on Stage-1 is not possible. 
4. Reiterating a point I mentioned earlier → More qualitative examples are needed.
5. The paper claims like past work when they upsample they don’t have this scaling-up factor of $r$ but I think in this case this $r$ just manifests as the voxel density in stage 1.
6. I’m also not clear what happens when multiple voxel centers move to the same location in 3D points.





summary_of_the_review: 

Justification:-
I think the paper does interesting contributions but lacks extensive qualitative results and the “Snowflake-PC → Score-PD” and similar things make me wonder if things are fair. The performance gap between the method and baselines is quite huge and I’m not able to completely understand the exact difference in impact between Stage 1 and Stage 2. Depending upon the rebuttal by the authors I will change my rating upwards if given reasonable explanations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an elegant two-stage pipeline to jointly solve the three tasks of point cloud densification, denoising and reconstruction at one time. Voxel densification and denoising are performed using a 3D spark stacked hourglass network, and then voxels are reconstructed into dense point clouds using transformers. The experimental results of ShapeNet exceed most existing methods, and have strong generalization ability.
main_review: This paper mainly has two strengths:
First, a two-stage pipeline is innovatively proposed, which can enable the network to solve multiple tasks such as point cloud denoising, completion and reconstruction at one time, and the reconstruction accuracy and visualization results of this method can surpass the latest methods in various sub fields.
Second, the innovative use of amplified positive encoding and transformers to compute the relationship between a center voxel and its neighbor voxels has certain enlightening significance for the follow-up work.

But at the same time, this paper also has some shortcomings.
From the experimental results, ShapeNet-Part dataset is comparable to SOTA results, and the generalization ability of ScanNet and ICL-NUIM datasets is significantly better than other methods, but the paper only compares the experimental results. It would be better if the paper could explain why the proposed new method can achieve such strong generalization ability, and why the generalization ability of the existing methods is poor.

summary_of_the_review: This paper creatively integrates different tasks such as point cloud denoising, completion and reconstruction into a two-stage task, and the generalization ability of the experimental results on ScanNet and ICL-NUIM datasets is much better than the latest existing methods. This paper creatively uses amplified positive encoding to compute the relationship between a center voxel and its neighbor voxels, which has certain enlightening significance for the follow-up work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method to jointly address the tasks of point cloud denoising, completion and upsampling. They call this joint problem "Point cloud reconstruction". The method consists of two main modules - a voxel generation module (to increase voxel density and remove outliers) implemented as an hourglass sparse convolutional neural network and a point relocalization module (to convert the discrete voxels back to point clouds) implemented using a transformer network. The results show good performance in the chosen metrics and datasets. 
main_review: The problem of point cloud reconstruction combines several existing important challenges in point cloud processing. The paper is well written and easy to follow, however, some details remain unclear and the evaluation choices inconclusive. 

Major issues: 


- Since many scans are combined into a single scan, there is another challenge that is entirely overlooked - misalignment. If the idea is to combine all challenges into one, this should be another challange to address.  
- Missing related work - There is a very large body of recent works on neural shape implicit representation which focuses on the surface reconstruction task. While I understand how this is a bit different from the proposed point cloud reconstruction, one could simply sample points on the zero level set and achieve point cloud reconstruction. Among these works are DeepSDF, Occupancy networks, SAL, SALD, DPDist, IGR,  Convolutional occupancy networks, PHASE, SIREN, DiGS, etc. These are missing particularly since in section 3.2 it is claimed that the local 3D shape is estimated using neighbour voxels. 
- Approach - in the described approach it was not clear to me how the final number of points is specified. Also, not clear how color information is determined for the newly generated points
- Evaluation baselines: I find the choice of the baseline questionable. First, there was no unified approach used (denoising + completion + sampling). Second, It would make more sense to me to compare the proposed method on the other's task since it is expected that a denoising method will not do well for upsampling. Third, in the combined baselines (Snowflake-PC->Score-PD and Dis-PU->scorePD) it is not clear the order of application (it makes sense to first denoise and then upsample/complete but the text suggests the opposite which should also be reported). Finally, as mentioned in the related work comment, the paper lacks comparison to implicit representations surface reconstruction. 
- Evaluation datasets - The chosen datasets are commonly used and an understandable choice for this task. However, they are strongly biased towards planar geometries. For example, most shapes in ShapeNet and most scenes in ScanNet have large planes. I expected to see results on the Surface Reconstruction Benchmark (Berger et. al.) which is small but has a mix of complex geometries. 
-Evaluation - time and memory requirements are missing. From the implementation details it seems to be resource-heavy, how does that compare to the other baselines? 
- Lack of insight - The paper is missing an important subsection that provides some insight into what the different modules have learned and how is that beneficial to the task. This raises some question about novelty - how is this approach better than the sum of its technical parts? 
- In its current form, the paper is not self-contained and the method is not reproducible due to a lack of technical details. 

Minor issues: 
 - There is an issue with the term "continuous 3D points" which repeats itself in the paper (section 3.2 for example). Point clouds are not continuous, they are sampled on a continuous surface. 
- Evaluation: The evaluation metrics were not properly specified. When performing standardized experiments it may be sometimes acceptable to commit this, however, in these case, the paper proposes a new task, therefore, the evaluation measures should be specified. 
- $V_{mid}$ is missing from Fig 2 and Fig3. One can only assume where it is in the architecture. It should be better specified.
- Section 2 -> Point cloud upsampling "input points input points" typo. 


summary_of_the_review: In summary, I find the paper well written and the problem interesting. The approach is well presented and technically sound for the most part. However, due to the lack of a significant body of works in the related work section as well as the missing evaluations I can not recommend this paper for acceptance at this stage.
review_title: null

Meta-review: The paper proposes a unified framework for point cloud upsampling, denoising, and completion through a two-stage approach. It receives three reviews with three leaning to accept and one leaning to reject. Most of the reviewers like the proposed two-stage approach for its simplicity and demonstrated strong performance. The reviewer recommending marginally below the acceptance threshold expresses concerns about missing comparison to neural shape implicit representation and a lack of insights on what is learned by individual layers in the network. While the meta-reviewer agrees that having both would make the paper stronger, the meta-reviewer feels the paper has enough merit and would like to recommend its acceptance.
Decision: Accept (Poster)