Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework called Temporal Point cloud Up-sampling to capture temporal and spatial features, in light of the difficulty to obtain point correspondence annotation. The framework is constituted by a generator for refining coarse point cloud, a temporal, and a spatial discriminator for feature extracting. Experiments conducted on two datasets towards human action recognition and fluid particles analysis demonstrate the effectiveness of this method.
main_review: Overall, the main issues of this paper are missing hyper-parameters experiments and unsatisfactory presentation quality. The authors are suggested to provide fair comparison under reasonable experimental settings, and improve the readability of the paper and further investigate the hyper-parameters to verify the robustness of the proposed method.


-Methods
- In Sec.3.1, the author mentioned only Inception DenseGCN is applied to the generator for efficiency. I am wondering how is the performance using the entire PU-GCN in the feature extractor? albeit the computational cost would increase.
- In Sec.3.2, it is not clear that how the author obtain the $r_{j}^{t+1}$  according to $r_{i}^{t}$. Do you require mapping points in the current frame into the corresponding points in the next frame, or using the same position in two frames directly?
- In Sec. 3.4, two parameters are introduced to the training process,(the predefined threshold $\epsilon$ in equal.3 and the number ’3’ in equal.4 ), please justify your hyperparameter choice in the paper. 
- In Sec.3.5, the CD loss is used to evaluate the distance between generator output and ground truth due to its efficiency. How about using other losses? Could it bring huge computational costs?
- In Sec.3.5, equal.6 and equal.12 have the same symbol $L_G$, which may cause confusion, It seems an adversarial loss was missed in your paper, please clarify.


Experiments
1. Missing ablation:
- Explore up-sampling strategy: For up-sampling points, equal.5 requires the points to satisfy two conditions simultaneously. How about using equal.3 or equal.4 for up-sampling solely?
- Each component of equal.13 shall be studied to verify the effectiveness of each block.
2. Add visualization results: It is interesting that this work adopts a GAN to achieve super-resolution of points cloud. For this block, It would be better and convincing if the input and output point cloud during the training stage (for example, the first epoch and the final epoch) could be provided.

summary_of_the_review: As mentioned in the main review, I am inclined to weakly reject this paper due to the missing hyper-parameters experiments, unsatisfactory presentation quality.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use a discriminator to enhance the temporal coherence for point cloud upsampling. The technique is tested in two scenarios: fluid mechanics simulation and human scan.
main_review: Strengths:

1. The problem addressed by the paper is significant. Leveraging temporal cues for point cloud upsampling is interesting and meaningful.

2. The approach makes sense. Approaches that leverage scene flow is likely to add too much computational burden and the annotation is very expensive to obtain. Using a discriminator can be a useful alternative.

3. The tasks are very interesting and commercially valuable. I am especially fond of the fluid mechnics simulation, and the potential commercial value behind it is immense.

4. The paper is coherent and clear, it is easy to follow.

5. Effective results in the visualization (Figure 2 and 3) where the proposed method clearly has a competitive edge. 

6. Suitable baselines are chosen in the experiments: PU-GCN (static SOTA) and Tranquil Clouds (first learning-based temporal method).

Weaknesses:

1. The method is largely a combination of existing works. The basic modules may take less space and have their own "preliminaries" section: set abstraction that is taken from PointNet++, multi-scale graph convolution from PU-GCN, and temporal feature extraction module flow embedding from Flownet3d. More importantly, the critical part of the nolvelty, the temporal and spatial discriminators have simplistic designs by putting the above-mentioned modules together. This is may be the first work to leverage temporal information with discriminators, having a straight-forward design is understandable, but adaptation to the given tasks may improve the relevance.

2. The experiment setting may raise some questions, and the quantitive evaluation seems to have some performance gap. I understand that existing datasets for point cloud upsampling are all static, but I would like to hear the authors' comments on if it makes sense to tweak the conventional setting and introduce motion into the existing datasets (e.g. sample point clouds with a moving virtual camera or sample point clouds from a moving object, as these settings have some use cases in real life). Moreover, the performance of the proposed methods are not outperforming existing methods entirely (Table 1 and 2).


Minor comments:

1. Please elaborate on how duplicate points are removed in Figure 1a)

2. Sec 3.3: "often no correspondences" -> "often no clear correspondences". As there are definately some correspondences but we are unable to know.

3. Sec 3.3: "... remains unknown" -> "are not annotated"

4. Sec 3.4: "However, not all point clouds are distributed in a uniform way such that every point has similar amount of points in their neighborhood": solid observation, but this is also the case for static point cloud upsampling. Does point cloud sequence makes this problem more prominent? I would assume adaptive upsampling is especially critical for point cloud sequences as it is highlighted in the abstract.

5. Sec 3.5: lambda 1 and lambda 2 are different for different datasets, does this mean the hyperparameters are sensitive?

6. Sec 4.1 Dataset: why do you use different upsampling ratio for two tasks?

7. Would authors release the dataset for fluid simulation?

8. Sec 4.2 Comparison with other models: it is claimed that TPU-GAN outperforms PU-GCN on "high-level" distance metrics, does this make CD not an effective metric?

9. Page 8 footnote 2: input size is a critical factor, would it be possible to adapt the code of Tranquil Clouds to allow for 16x upsampling?

10. Conclusion and Figure 4 captions are excessively close to each other.
summary_of_the_review: The paper has clearly several weaknesses, but it is undeniable that it has brought forward an interesting topic of point cloud upsampling with temporal cues, and developed two relevant tasks that are useful in real life. I thus opt for "marginally above the acceptance threshold" as the initial rating and hope the authors could address my concerns during the rebuttal period.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In the paper, the authors firstly introduced TPU-GAN, a GAN that provides a temporal coherence for point cloud sequence.  Another contribution is an upsampling module built into TPU-GAN. In contrast to other methods that produce almost uniform sampling, this probability module provides a more reasonable distribution of upsampling points.
The authors investigate proposed methods both quantitatively and qualitatively on two different datasets. They show that TPU-GAN performs better than other state-of-the-art methods in terms of static view (per-frame) and temporal coherence. They also prove that the upsampling model in TPU-GAN provides distribution closer to ground truth distribution.

main_review: The paper proposes TPU-GAN,  an approach for point cloud generation concerning temporal coherences. In addition, the authors introduce a new trainable mask for controlling the distribution of sampled points. Both methods are well motivated. The authors provide experiments investigating various aspects of the proposed approaches. While the results seem to be reasonable, there are several concerns stated below.

Major comments:

1. I find methods to be well explained and experimental part to be sufficient. The authors' results are reasonably better than others analyzed methods. Nevertheless, I find that the main contribution, TPU-GAN, is based on several well-known ideas or techniques with small, in my opinion, novelty. Here is how I understand the novelty of the work: there already exist approaches that use GAN, for example, PU-GAN and PU-GCN. The novelty here is to use a GAN that incorporates learning temporal coherence. However, in my opinion, the background behind the TPU-GAN is based on existing approaches: feature extractor follows PU-GCN,  Temporal Embedding Layers used in the architecture are from work [4] with a small difference in input features ($ F_t, F_{t + 1}$). Spatial and temporal loss functions are common for GANs. In addition, the idea of using the triple moment($t - 1, t, t + 1$) for training the model appears in the paper [5].  Please, give clear arguments to show the significant novelty of TPU-GAN. 

2. It’s claimed in the text that “Tranquil Clouds requires point correspondence across frames to evaluate the first-order and second-order finite difference loss, which our method (TPU-GAN) does not require during the training phase.” However, TPU-GAN’s loss function still relies on ground truth in $L_{CD}$ . Please, clarify what exactly was meant by this statement? 

Minor comments:

1. I recommend the authors look through the recent work "Sequential point cloud upsampling by exploiting multi-scale temporal dependency" by Wang K [3] and discuss the similarities and differences with the proposed TPU-GAN. In this paper, the authors also proposed an approach for non-uniform sparse upsampling with preserving temporal dependence. Although their results for MSR-Action 3D are a little bit lower in terms of Chamfer Distance, they don't estimate temporal coherence, and probably the TPU-GAN is significantly better in this metric. Of course, the authors could find even more advantages.
2. Papers [1, 2, 3] consider Hausdorff Distance and Point-to-surface distance as metrics. Is it reasonable to include them in the analysis? 
3. In equation (2), what is $\gamma$ and $h$? Are they similar to the same in the equation (1)? 
4. In equation (3), why is it used 3 as the threshold? 
5. Please, proofread the work. Examples: 
Abstract. “In this work…” - should be separated by a comma; 
Section 2. “Compared to PU-GAN…” - seems should be to PU-GCN or the reference on PU-GAN should be added;  
Section 2. “...the change of points with respect to time are not taken into consideration” - use “is” instead of “are”;

References:
1. Li, Ruihui, et al. "Pu-gan: a point cloud upsampling adversarial network." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
2. Qian, Yue, et al. "Pugeo-net: A geometry-centric network for 3D point cloud upsampling." European Conference on Computer Vision. Springer, Cham, 2020.
3. Wang, Kaisiyuan, et al. "Sequential point cloud upsampling by exploiting multi-scale temporal dependency." IEEE Transactions on Circuits and Systems for Video Technology (2021).
4. Liu, Xingyu, Charles R. Qi, and Leonidas J. Guibas. "Flownet3d: Learning scene flow in 3d point clouds." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
5. Prantl L. et al. Tranquil clouds: Neural networks for learning temporally coherent features in point clouds //arXiv preprint arXiv:1907.05279. – 2019.

summary_of_the_review: The proposed methods are well-investigated from different aspects in two different datasets. The paper is well-structured and well-written; most of the claims and hypotheses are supported by good arguments. However, it remains the question about the novelty's significance. If the authors provide reasonable explanations, I am ready to recommend the paper for acceptance.
 
Please, let me add a comment that I’m not an expert in the field of models for point clouds and admit that I could understand something wrong, but I've spent a lot of time reviewing the literature.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a GAN framework for super-resolution on the dynamic point cloud sequences. 
The main purpose is to avoid explicit supervision on the point displacements (scene flows) for this task.
Specifically, the Generator aims to produce super-resolution points for each individual frame. Then, a Temporal Discriminator and a Spatial Discriminator are used to measure the confidence of temporally coherent and spatially coherent, respectively. 
Experiments on fluid particles and 3D scanned human action that the proposed upsampling model can produce spatially and temporally point cloud sequences consistent with target data without explicit supervision on the point displacements.
main_review: Strengths:

1. Avoiding explicit supervision on the point displacements (scene flows)  is an important and big challenge for point cloud sequence modelling.  The direction that spatial-temporally upsamples point cloud sequences without scene flow supervision is correct and promising. 

2. The proposed method that employs a GAN framework for the super-resolution of dynamic point cloud sequence is technically sound.

3. The proposed method is evaluated on two very different datasets and experiments demonstrates the effectiveness of the proposed methods. 

Weaknesses:

1. Although addressing an interesting problem, the novelty and contribution seem a bit limited. GAN frameworks, such as PU-GAN,  have been used for point cloud upsampling. Modules, such as Set Abstraction (PointNet++) and Flow Module (FlowNet3D), are also widely used for static point cloud and point cloud sequence processing.  Combining these techniques to solve a new problem seems not that novel. 

2. The learnable masking module, which adaptively upsamples points, is a difference from previous works. It cloud be better to provide the visualization for this module to show what it learns. 

3. Self-driving is a more important application for point clouds. It cloud be better to apply the proposed method on self-driving car datasets to demonstrate its effectiveness. 

4. Upsampling three frames is somewhat short. It could be interesting to investigate the influence of sequence length on this task. 

5. A few related works are missing. For example, [a] interpolates frames into point cloud sequences, which can be seen as temporally upsampling.  [b] and [c] also learn Spatio-temporal representations of point cloud sequences without point displacement supervision. 

[a] PointINet: Point Cloud Frame Interpolation Network (AAAI21)

[b] Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos (CVPR21)

[c] Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences (TIP21)
summary_of_the_review: The paper presents an interesting problem. The proposed method is effective. However, the novelty seems a bit limited. Some experiments and related works are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced a method of point cloud upsampling using GAN, which can learn the underlying temporal coherence from a point cloud sequence.  TPU-GAN can determine for each input point whether it should be upsampled or not by introducing a masking module into the conventional point clouds upsampling model. In addition, TPU-GAN can learn the underlying temporal coherence using temporal discriminator that takes temporal sequential point clouds as input. The experiments are evaluated with particle-based fluid dynamics simulation dataset and scanned human action dataset. In the comparison,  the method proposed in this paper outperformed PU-GCN and Tranquil Clouds. Moreover, to confirm the performance variation with the number of input frames and to confirm the necessity of the both temporal discriminator and spatial discriminator in the learning, some other experiments are conducted. 

main_review: strengths 
- TPU-GAN can learn temporal information from sequential point clouds and the experiments show that TPU-GAN can learn the underlying temporal coherence. 
- TPU-GAN can determine for each point whether it should be upsampled or not, and perform upsampling only for the points that need upsampling.

weaknesses 
- There is no ablation study to confirm whether the function to determine for each point whether it should be upsampled or not is realized by the masking module. 
- The reason for employing the spatial discriminator should be explained.  I could not understand the reason why calculating temporal discriminator loss and spatial discriminator loss because temporal discriminator seems to include spatial discriminator in its structure.

Comment 
- The explanations of (b) and (c) in Figure 1 is reversed.
- I think "Table 4.3" in the fourth line of the subsection "Self-supervised temporal feature learning" in section 4.2 is a mistake for "Table 3".
- In the fourth line of "Self-supervised temporal feature learning" in section 4.2 "Table 4.3" may be "Table 3".

summary_of_the_review: Please provide a short summary justifying your recommendation of the paper.  TPU-GAN is well designed to learn the underlying temporal coherence from a point cloud sequence, but additional experiments and additional explanations about the model and modules are necessary.
review_title: null

Meta-review: The paper proposes a GAN framework for dynamic point cloud superresolution. It does not need scene flow supervision for training and has an interesting adaptive upsampling mechanism. Results are shown on several datasets and are reasonably convincing. Overall, all the reviewers are slightly positive about the work. After the rebuttal, all the five reviewers converged to a marginally-above-the-threshold recommendation. The meta-reviewer agreed with their assessment and would like to recommend accepting the paper.
Decision: Accept (Poster)