Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies scene-flow estimation. Following the idea of deep-image prior, it proposes an unsupervised way to learn scene flow. The results are competitive, or even better than supervised flow-estimation methods trained on very different training set. Moreover, it also out-performs existing optimization based unsupervised flow-estimation methods.
main_review: I'm not an expert in the field of 3D flow estimation, and not super familiar with the SOTA unsupervised or self-supervised methods for this tasks. It's likely that my interpretation or understanding is biased or wrong. For the paper authors and other people, feel free to correct me if you find such things.

Strength:
1. This paper identifies the poor generalizability of one established supervised method. The results are supervised and it performs very poor on new data domain. If one or two more baselines could be studied, this result would be more convincing. 
2. The proposed method out-performs existing work "graph prior" significantly and consistently. Another good property of this work is that many prior-deep learning works are mentioned or compared, which is rarely now days and show be encouraged.
3. The experiments are conducted on multiple different experiments, and promising results are presented and visualized.
4. Writing is mostly clearly and easy to follow. 

Weakness:
1. The supervised baselines seem to be very weak. On the unseen dataset, the Acc_5 could be as low as 1.34%. Even on the training set (FyingThings), Acc_5 is only 22.64%. As a comparison, the proposed method can achieve 80% Acc on other dataset. Therefore, I don't think these supervised baselines are strong, which makes the results less convincing.
2. For 3D flow estimation, shouldn't EPE the major metric? 
3. Recently, the RAFT model become the most popular method for 3D flow estimation. For 3D flow, RAFT-3D [1] is the current SOTA method. I think it's a more proper baseline than FlowNet3D, and I'm also wondering how well will it generalize to new dataset. In fact, the provides numbers/models for both FlyingThings and Kitti, which help us to understand both the generalizability and performance upper bound.
[1] RAFT-3D: Scene Flow using Rigid-Motion Embeddings. Teed and Deng. CVPR 2021
4. In Deep image prior, it tricky to find an optimal stopping iteration during training time. It's unclear to me how does this work handle this problem. 
5. This work should be classified as self-supervised learning approach IMO. It's essentially corresponding learning, during which the 3d flow is also learned as an architectural prior. 
6. The cycle-consistency loss is an important innovation but not ablated.
7. The architectures are inspired by recent implicit models but lack of motivation. I'm not convinced by the advantages of doing this and why this framework is proper for flow-estimation.

Misc / Questions:
1. In Fig.3, why does one line of graph priors decrease over time?
2. In Tab.1, mean/var of "graph prior" and "ours"should be reported for all metrics for completeness.
3. Deep image prior trains one model for each picture. Does this work also train one model for each scene?

limitations_and_societal_impact: 1. Limitations: one failure case is shown in Sec 4.5 and Fig.6 to show the limitation of the NN loss.
2. Societal impacts are properly discussed (L309ff).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission presents a method for estimating scene flow between two point clouds that uses a neural network as an implicit regularizer during optimization. The submission revisits the classical optimization-based approach where instead of training a neural net to estimate flow for a given scene, it optimizes the flow per scene and does not need to train the network beforehand. Instead, the network is “trained” to produce flow for the given scene, such that representing the flow with the network introduces some form of implicit smoothness regularization.
main_review: # Originality

The idea presented in the submission is interesting and while neural networks have been applied as implicit functions in other domains, their application to scene flow is novel as far as I know. The presented method connects modern neural network based approaches to flow estimation to classical flow estimation approaches. The listed related work is extensive. However, I believe that the comparison to related approaches has important gaps – I will give more details about this in the next point.

# Quality

The description of the method is technically sound. The experimental evaluation of the proposed method and its components (apart from the comparison to other work in Table 1) is extensive and insightful, e.g. Sections 4.1, 4.3, 4.4, and 4.6. I also appreciate the discussion of the failure case in Section 4.5 – more examples would be even better (maybe in the supplemental material). I find Figure 5 especially helpful to understand how the model is optimized over time and how it interpolates between points. From that figure, I’m curious if the removal of the ground plane is essential for the method to work well. Maybe the authors could comment on that if they have done those experiments.

Unfortunately, some claims seem overly general and are not well supported by evidence: a) L26-28 portrays the need for data in itself as a problem, including training on synthetic scenarios and then fine-tuning on small real datasets. But this in itself is not a problem, at least not for domains where such data exists, as for autonomous driving which this paper focuses on. b) L29 claims that poor generalization is a problem but fails to give evidence for this in general. Yes, generalization is poor when one goes from FlyingThings3D to KITTI, but this looks like a strawman because the more relevant question would be how well the models generalize between KITTI, nuScenes, Argoverse, and other autonomous driving scenarios. In contrast to the statement here, e.g. [28] states that “we show that our network, even trained on synthetic data, is able to robustly estimate scene flow in point clouds from real scans, showing its great generalizability.” c) L13-15 make the argument that because optimization happens at runtime and no offline data is needed, this approach would be ideal for deployment in autonomous driving – ignoring the problem that optimization at runtime is much slower compared to performing optimization during training time, which will in fact make it very challenging to apply this method for autonomous driving to produce scene flow in real time. Based on Figure 3, inference takes > 5s for a single scene. Here the discussion should be more balanced and mention both the advantages and the disadvantages of performing optimization at runtime. Figure 1 is also misleading in this regard as it cherry-picks one alternative method that is slower but disregards other related methods that are much faster than the proposed approach. L168 states that the evaluation focuses on both accuracy and computational cost but does not take the cost at inference into account when comparing to other methods in Table 1.

The comparison to SOTA in Table 1 does not appear fair to me for a number of reasons:
a) The SOTA comparison only includes a single (and no longer SOTA) supervised method FlowNet3D and ignores important related work with published results on FlyingThings3D and KITTI, e.g. HPLFlowNet (CVPR 2019), PointPWC-Net (ECCV 2020), FLOT (ECCV 2020)
b) The supervised method FlowNet3D is only evaluated with domain transfer from FlyingThings3D to driving scenes even though a model trained on KITTI is available. This is stated in the text but not in the caption of Table 1 making it easy to miss. Testing generalization is important but to give the full picture, the KITTI-trained model should be included and it should be made clear when a model is tested outside of the training domain.
c) Self-supervised methods here are only optimized on the training scenes and need to generalize to the test scenes. For a fair comparison, it would be good to also report their scores after fine-tuning on the test scenes in order to have access to the same information that the presented method has during optimization.
d) To adequately show the advantages and disadvantages of these methods, an estimate of the inference time should be added to the table as this will vary strongly between the compared methods and orders of magnitude differences in inference time substantially change the applicability of a method for estimating motion (which often needs to be done in real time).

L162: I’m not convinced that comparing the number of parameters is very useful here. Parameters have very different semantics in both cases. For a learned model, they correspond to the memory required to store the model, but in the proposed method, they need to be optimized for inference. As the learned model represents a flow estimation model for “all” scenes it should not be surprising that this needs to be larger than a model that represents the flow for a single scene. If one wants to make the comparison, both memory and inference time should be taken into account.

# Clarity

The submission is written clearly and well organized. All code is included in the supplementary material, which should make it easy to reproduce the work.

# Significance

The presented method and results are important to the community and could be the basis for further exploration into this approach of estimating scene flow during inference using neural networks as implicit functions. My main reservation regarding this work regards the comparison to existing learning-based approaches both in the text and in the empirical results. Here the paper seems to focus too much on the advantages of the presented method and does not always give the whole picture. 

To make this very clear: I think that the paper has very valuable contributions as it shows progress along a direction that is not the main path taken by related work right now. Even if a complete comparison to related work (as described above) shows that this approach is not the most practical method (in terms of accuracy and / or inference time) it would still be a very valuable contribution because it shows progress along an approach that is very different from most existing work. But only with a fair portrayal and comparison of related work will I be able to recommend accepting the paper.

EDIT: My main concerns have been addressed in the post-review discussion with the authors.
limitations_and_societal_impact: Limitations are not fully discussed (see above).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of scene flow: estimating 3D motion fields between given point clouds from a dynamic scene. The proposed method predicts scene flows by optimizing a neural network at execution time, using randomly initialized weights. 

This approach of using a network architecture as a prior is shown to have better generalization performance for out-of-distribution samples compared to learning-based methods which require offline datasets. In contrast to another recently introduced non-learning approach which uses graph Laplacian priors, this method provides a continuous scene flow representation, and has better time and accuracy performances in large scale scenes.

main_review: This is a well-written paper, which I enjoyed reading. While the problem studied in the paper has been around for a long time, the approach for solving the task is novel and interesting. The claims in the paper are supported by comprehensive experimental results across four different datasets.

My questions to the authors along with some comments are listed below.
 
- Selection of MLP networks to represent scene flows: Processing point clouds (or point sets) with MLPs has a few drawbacks that are already well-known, such as imposing an ordering of the points and requiring a fixed cardinality for the point set. While the distance function used in Eq. 4 allows point sets of different sizes, the MLP inputs point clouds of fixed size (set as 2048 in the experiments). A discussion about how to take point clouds of differing sizes to compute the scene flow would be useful -- especially for the case when computing over the sequence of M point sets. 

- Related to the previous comment, could optimizing the network weights at execution time using an architecture designed to operate on point clouds (e.g. [1]) regularize the motion field better? Alternatively, can the method be modified to take a single point as input to the MLP at a time for allowing variable point cloud sizes?

- Can you comment on the computation/inference time of other learning- and non-learning-based methods? Adding how much time the non-rigid ICP and learning-based take would be helpful.

- What is meant by the rigid accumulation of point clouds, shown in Figure 7? Does it refer to computing the flow in a discrete fashion using nearest neighbors as opposed to the continuous flow estimation? 
 

Minor comments:

- In line 110, I believe the reference to [15, 24] should be [15, 27] instead.
- You may consider adding a zoomed inset image in the top-down views of Figure 5 for better visual clarity.


References

1- Qi, C.R., Yi, L., Su, H. and Guibas, L.J., 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413.

limitations_and_societal_impact: The authors have assessed the limitations of their work, including a failure case due to the nearest-neighbor distance term in the loss function. Figure 6 shows an example of this failure mode. Other than that, the paper contains an analysis for the degradation of the computation-time performance of the presented method, along with the baseline’s performance.

The paper also includes a broader impact section, for discussing potential outcomes of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 
This paper highlights the fact that deep learning has caused a paradigm shift in many computer vision applications from methods based on optimization with strong priors to ones that optimize  feed-forward neural networks in an offline process. The paper argues that these feed-forward models are bad at generalization and proposes to use the network in a radically different way. In essence, the network are used as a parametrization for a traditional optimization solution - allowing the network's inductive biases to be used as a “prior”. While using a network structure as a prior for optimization is not new (ala Deep Image Prior, DeepSDF, NeRF etc.), applying this type of parametrization to the problem of optical flow is new. The results are competitive with trained approaches and quite impressive for such a simple solution.
main_review: 
Overall this is a well-written and interesting paper. Please find my detailed comments below where I have labelled each point as either positive (+), mixed (+/-) or negative (-).
 
**Originality**: 

(+) The method is new in the sense that a neural parameterization of this type has not been used before for scene flow estimation. This type of parameterization has multiple advantages such as the fact that the scene flow becomes a continuous field that can be sampled at any point in space which is a much more natural way to represent the flow field. In contrast traditional approaches based on variational methods would typically solve for the flow field only on 2D frames of projected points. 

(+) In my view, the work ***is*** a novel combination of well-known techniques. Optimization-based approaches for scene flow have existed for a long time and quite sophisticated methods have been developed. Similarly, the use of the inductive biases of deep networks as a prior to guide optimization was first described in “deep image prior” and has been exploited in multiple subsequent works. However, the application of these to the problem of scene flow estimation is novel and the authors have done a very good job in describing how their approach bridges the different paradigms.

(+/-) Overall, the work is well-positioned with respect to previous contributions in the space (like Deep image prior, supervised methods, and traditional optimization approaches). However, there is one very related work that seems to be missing and that is DeepMapping [A]. Like the proposed approach, DeepMapping also aligns point clouds in an “unsupervised manner” (i.e. test time optimization) and uses a deep network for the parametrization for the pose. I do think there are clear differences between the proposed method and DeepMapping in the sense that the proposed method recovers a full, continuous flow field whereas DeepMapping only recovers the pose. This means that the proposed method can compute the flow for dynamic objects whereas DeepMapping is limited to rigid scenes. The proposed method also appears to be much simpler than DeepMapping in terms of its design and training. However, it would probably be good to reference and compare DeepMapping to the proposed approach (in terms of a description of the differences) as it probably is the closest existing method. It would also be interesting to see how DeepMapping performs compared to the proposed method on rigid scenes.

**Quality**: 

(+) As far as I can tell, the submission is technically sound and there are no major errors in the method. 
The claims related to the proposed approach (in terms of performance compared to existing methods) are well supported. The numerical experimentation is sufficient to allow the reader to place the method with respect to recent supervised approaches and the results do indicate that the method is very competitive with these approaches. 

(+) The paper is relatively polished and complete in the sense that the idea is fully described and evaluated to an extent that allows the reader to see that the approach works and what problems it is suited for.  It would have been nice to see more demos on different types of data (currently only lidar point clouds from vehicles are shown) and I’m really curious how this would work on denser point clouds from RGB-D sensors in dynamic indoor scenes. The attached video clip in the supplementary showing the approach in-action is a bit difficult to understand as it’s just a screen grab of the python program running with no editing. However, I don’t think these issues detract significantly from the quality of the paper.

(-) The authors do a good job of highlighting the strengths of the method, but fall a bit short in describing the weaknesses. The main advantage of approaches that are only optimized at training time is of course their inference speed and this has been one of the major factors that has led to the widespread adoption of deep learning models. Whereas traditional state-of-the-art scene flow methods would take minutes or hours to process a single example, methods like FlowNet3D can do it in seconds. In essence, there exists a strong tradeoff between generalization and inference speed and I think it is quite important for the authors to make this clear in the paper. Inference time is currently only compared to GraphPrior in Figure 3, but I think inference times should be given for other methods like FlowNet3D as well (perhaps by adding them to Table 1).

**Clarity**: 

(+) In general the submission is very clearly written, and makes a good argument for the neural prior and optimization-based approach that is proposed. The paper is well organized and contains all the details to replicate the approach (the authors have also provided the source code for reproducibility).

**Significance**: 

(+) In my view the results are quite important for two reasons. It shows that a very simple optimizer (Adam) combined with an easy to implement loss function and neural prior can achieve very good results on the scene flow task - even beating supervised approaches in some cases. Secondly, it shows that a continuous representation is the best way to represent 3D flow fields (similar to what has been shown on other 3D tasks too - like for volume rendering in Nerf). Therefore I think these result could make a large impact in this research area too.

(+) Furthermore, I do think other researchers are very likely to be inspired by the approach and try to build on top of it to overcome its currently limitations (like inference speed and pushing the accuracy to maybe beat the current supervised methods). This is because the proposed method opens up many interesting possibilities for future research. 

(+) Although the method does not advance the state of the art in terms of numerical performance, it absolutely presents a method that is better in terms of complexity and ease of implementation compared to previous work. In my view, the simplicity of the approach, and the advantages afforded by being able to reason about a full, continuous flow field are what really make this a great paper.

[A] Li Ding, Chen Feng, DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds, CVPR 2019.

# Post-rebuttal and discussion: 

After considering the authors' response and the other reviews, I am still positive about this paper and therefore give it a final Rating of "7: Good paper, accept". I think the idea of using a neural network parameterization for test-time optimization to solve for non-rigid flow is a very interesting contribution and promising area of research and the paper shows strong results to back this up. The authors have promised to include the missing reference and discussion about DeepMapping [A], as well as clarifying the limitations of the approach.

limitations_and_societal_impact: 
The discussion of the potential negative societal impacts is limited to "as an AI-based research, it could be misused" - which I don't feel is satisfactory. Being a method for tracking people, objects and things accurately I think the authors should at least mention immediate ways in which it could be misused for nefarious purposed - e.g. invading privacy by tracking pedestrians across the city, or it's potential use in autonomous weapons and drones. I only mention these as the authors have described many potential positive impacts but I feel the discussion should be more balanced.
review_title: null

Meta-review: This iinteresting and well written paper proposes to use neural network training at runtime (i.e., during inference) to fit a scene flow function g (a plain MLP) that maps 3D points from a point cloud at time t to time t'. The function is formulated per and applied to individual points as an alternative formulation flow regularisation, meaning that the several thousand points in typical self-driving car dataset are sufficient for it to converge. Strikingly, the method is very competitive to supervised methods (that are faster at runtime but need to be trained) as well as to ICP.

After a lengthy discussion between the 4 reviewers and the authors (longer that the paper itself), the consensus was to accept the paper with scores 6, 7, 7, 7, once the specifics of the evaluations (training the supervised methods on 2k points vs. training on 8k points) were elucidated and the authors accepted to do some revisions to the manuscript. I personally commend all reviewers and authors for exemplary thoroughness and collaboration on this review.
Decision: Accept (Spotlight)