Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies how to learn single image dense keypoint prediction by leveraging multiple-view constraints. Compared to prior works in this space, the key contribution is the modeling of the epipolar geometry constraint across different views.
main_review: This paper studies an important problem of learning keypoint detectors by using multi-view supervision. Compared to prior works, the key contribution is the learning of dense keypoints and using epipolar constraints. 

The epipolar constraint provides some supervision for the dense keypoint prediction. However, it is not a direct supervision as we do not know which point on the epipolar line is the corresponding point. This paper proposes a solution, which is similar to an attention mechanism, by matching points in some feature space. The proposed solution is plausible. 

The network training incorporates other losses such as photometric multi-view consistency and regualrizations using a pre-trained model.

The experimental results show that the proposed approaches outperform state-of-the-art approaches on two benchmark datasets. 

There are a couple of issues that prevent me from giving higher scores:
1. It is not entirely clear how to address the partial similarity across the multiple views. The paper does mention that this is an issue. However, it is unclear how the definition of $P(u,u')$ and equations (2) and (3) address this. 
2. The epipolar constraints state that two lines of the two images are in correspondence, and correspondences are expected to be continuous. The current formulation in the paper only use such signals partially.
3. It is unclear how to the canonical model is defined. Figure 2 shows that it may be a 2D surface parameterization of a 3D mesh. If so, why the phi map is not invertible? 
4. The current setup requires calibrated images with known intrinsics and extrinsics. What about if the estimations of those have errors and inaccurate. 

limitations_and_societal_impact: The limitations are not discussed explicitly, but the experimental results are analyzed in depth, including the pros and cons.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a novel method to learn dense keypoints in a semi-supervised manner using unlabeled multiview images. They propose to leverage the epipolar constraint of dense correspondences given a known fundamental matrix. In order to make epipolar constraint learnable, they propose a probabilistic epipolar constraint, incorporating uncertainty in correspondences using soft correspondences. The proposed method is proven to improve dense keypoint detectors in multiple datasets over reasonable baseline methods.

main_review: The proposed method is reasonable, well written, and well studied. I cannot comment much about the originality of the work since this is out of my field of research, however the paper was easy to follow. Given that dense labeling of correspondence is not an easily affordable label, I have no doubt that self-supervised learning with known geometric constraints definitely helps.

Here are some minor comments that may potentially help me understand this work better:

It would be helpful to ablate the effect of L_R. Although L_R is supposed to be a regularization, there may be some knowledge distillation from the pretrained model on unlabeled dataset that may potentially help improve the keypoint accuracy by itself. L_L + L_R would make the effectiveness of L_M obvious, especially since L_L + L_M does not outperform L_L (although I believe is reasonable).

How does the choice of $\sigma$ in soft correspondence affect the performance of the model? Which value did you choose for $\sigma$? How did you find the optimal $\sigma$? Some ablation study on the effect of $\sigma$on the performance would be helpful.

limitations_and_societal_impact: This paper thoroughly study the limitation of the proposed method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a method for learning continuous dense keypoint fields with part-supervised/labelled data and part-unlabelled data. The authors derive a probabilistic matchability constraint and define a loss that includes multi-view consistency, photometric consistency and a supervised loss from the labelled data. The method is evaluated quantitatively by comparing to a few other baselines and doing ablation studies on various loss terms and train/test splits. Geometric consistency and reconstruction accuracy are used as metrics to measure the accuracy of the proposed method and a few qualitative results are provided. 
main_review: **Main paper**

1. Motivation: It might be good to elaborate a bit on what the applications of the proposed method would look like, in order to motivate the method soundly. Typically, keypoint matching whether sparse/dense and discete or continuous is used as a first step in learning the transformation between a pair of images. While there are several classical methods to solve this problem, there is also a rich body of learnt methods such as Spatial Transformer Network etc, that learn and predict the transformation directly. Could you provide example scenarios where having keypoint matches (i.e. the intermediate output) is necessary? 

2. Related work: What is the downside of applying sparse keypoint matching? Alternatively, what motivates the need for dense keypoint matching? There are many learned keypoint detection/matching methods in literature such as 3DFeatNet, R2D2, SKD, USIP. Would the authors be able to provide some clarity on where these methods fall short (and therefore what the USP of the proposed method is)? In summary, it would be helpful to motivate clearly why dense keypoint matching is needed and sparse matching is not sufficient? Sparse matches help recover an accurate transformation matrix, which gives dense mapping between images.

3. Section 3.2 [Loss definition]: Loss equation could be written out more clearly. Why are some equations missing numbering? It's unclear whether $\lambda_M$ and $\lambda_R$ are inside the summation or pre-multipliers to the next term such as $\mathcal{L}_M$ and $\mathcal{L}_R$. Reading the description later in the paper clarifies this, but it would be good to sort out correct spacing or multi-line equation here to help resolve this.

4. Is there a constraint on $\lambda_*$ as weights. Looking at the supplementary information suggests that there is no constraint and these are multipliers of the implicit weight of $\lambda_L = 1$ for supervised loss $\mathcal{L}_L$. Please clarify why a constraint on the weights is not needed and provide some intuition or explanation for the _very_ different values of the relative weights for the various terms in the loss function. 

5. Section 3.2 [Loss definition]: I think multi-view consistency loss is not really working because it is naïve. The true match for a keypoint lies anywhere on the epipolar line if only two images are available. If three images and the fundamental matrix $F$ is available, then a point-point constraint can be derived and used. The equations and training details seem to suggest that only two images are being used. Can the authors confirm if two images or three images are used for multi-view consistency loss? The regularizing loss $\mathcal{L}_R$ is formulated as 'not too far from the pretrained keypoint prediction'. Given that this is defined on labeled data with ground truth, it effectively imposes the multiview consistency constraint correctly and hence improves the results (as evidenced in ablation study on loss terms in Table 4). This may explain why results for $\mathcal{L}_L + \mathcal{L}_M$ are consistently much worse than those for $\mathcal{L}_L$ alone and those for $\mathcal{L}_L + \mathcal{L}_M + \mathcal{L}_R$. I suspect, $\mathcal{L}_M$ can just be dropped and $\mathcal{L}_R$ could be used instead. To this end, it would be interesting to see an ablation for $\mathcal{L}_L + \mathcal{L}_R$. Hence or otherwise, it would be good to see a decent justification for the inclusion of $\mathcal{L}_M$ term in it's curent form. See also my comment regarding Supplementary info Section A.

6. Photometric consistency loss: I wonder if results from an HSV or albedo related colour space insteadof RGB are better in any noticeable way? They may not be, if the network has sufficient capacity to learnt the RGB to HSV conversion implicitly. Allowing the colour components to change less than the lightness or brightness components should lead to better robustness for lighting changes. I suspect this is not an experiment worth doing unless dealing with datasets with extreme lighting changes. Since only a handful of examples of failure case are provided, I leave it to the authors to judge whether this is an area they think needs improvement. The goodness of the term $\mathbb{T}(x,x')$ is entirely dependent on the goodness of the norm $||\mathcal{I}(x)-\mathcal{I'}(x')||$ and what representation is chosen for $\mathcal{I}(x)$ in general. I do acknowledge that changing the representation of $\mathcal{I}(x)$ will affect rest of the network too, so perhaps only the distance metric here should be updated instead. 

7. Evaluation: The method is trained on Human3.6M and then evaluated on SkiPose and OpenMonkeyStudio datasets. all three datasets have a reasonably long range or static background and a moving object in the foreground. It could be useful to see results on a different kind of dataset to disamiguate whether the proposed approach relies on the foreground/background setting in order to be sure that the background pixels are also contributing to the learning process. Another way to verify this is to plot saliency maps for network weights. If the method does capture the fact that the foreground moves against a static background, then that would be fine, but then the paper should specify what kinds of applications such a network is suited to. Ultimately, such analysis will help answer the question 'how general is the approach?'.  

8. Evaluation: Figure 5: While it's good to see examples of dense keypoint matching results, it's unclear how to interpret the general goodness of the algorithm from the results in Figure 5. An alternative way to show the results could be to show the calculated transformation between the images and possibly a comparison against a pair of images where the ground truth transformation is available. If there are any patterns in the errors, they will likely become clear from these mappings. The continuous dense keypoint field could be sampled on a regular grid in the source image to acquire some grid points in order to enable comparison to classical multiview geometric methods. 

9. Results [Table 1]: I am confused about why 'Ours' does better on geodesic point similarity despite being worse on AUC10 compared to HMR. 'Ours' does do better on AUC30 compared to HMR, which suggests that 'Ours' has a flatter error profile but that does not correlate with high geodesic point similarity. It dependes somewhat on the value of $\kappa$ used. Would the authors comment on scenarios where AUC30 is preferred over AUC10? It's a little bit counter intuitive that 'Ours' does better on geodesic point similarity (RCI is just anothermanifestation of GPS) and epipolar error, yet does not do well on AUC10. Evaluating something like AUC15 or AUC20 will help uncover if it's just that the crossover point between HMR and 'Ours' is just past 10cm. 

10. Line 142-143: "A point that is not visible in a corresponding image does not have to be forced to match" - A no match found is a valid outcome. This claim is misleading, because authors use this outcome themselves later in the paper. 

11. Line 144: I find the argument about geometric error being discrete and hence non-differentiable a bit arduous. The expected geometric error (i.e. the version using argmin) is discrete only by choice. It is perfectly possible to define continuous versions of the same quantity as indeed the authors do later in the paper. But of course, if one wants to select a single correspondence, then we end up with a discontinuity. I am not sure what purpose this point is serving. I suggest rewriting this. 

**Supplementary information:**

1. Section A [Training details] mentions you chose $\mathcal{L}_M = 1.0$, $\mathcal{L}_R = 2000$, $\mathcal{L}_T = 10$. 
    Did you mean the weights $\lambda_M = 1.0, \lambda_R = 2000, \lambda_T = 10$ instead? 
    If yes, then perhaps this supports my suggestion for dropping $\mathcal{L}_M$ term completely. $\mathcal{L}_R$ is the term that is actually imposing the multi-view consistency constraint. I think that $\mathcal{L}_T$ is too strict (or restricitve), and that's why you 
    (a) have accuracy issues where the photoconsistency is poor and 
    (b) only a small amount of correction is allowed in the training (i.e. $\lambda_T << \lambda_R$)
    This also raises another question that the supervised loss $\mathcal{L}_T$ (aka label loss) has a very small contribution to the total loss, and yet, we see from ablation studies that $\mathcal{L}_T$ alone has reasonable performance.
    
2. Section A [Training details]: For reproducibility, it would be helpful to have a few more training details such as dataset splits, data augmentations if any, pre-processing of the data, cleaning, imputation and view selection for multi-view consistency loss where appropriate. 
    
3. Section B [Failure cases] provides some explanations for failure cases. I find these puzzling. There must be enough corresponding points between the two views. If you output a probabilitistic matching score, then perhaps it is possible to set a threshold for accepting the matches. For various values of the acceptability threshold, you would be able to determine the fraction of 'accepted matches' vs 'total matches' and also determine the point beyond which the separation in the two views is so large that no acceptable matches can be found. Something like that may be a better measure of the quality of matching rather than saying that the network fails due to one of these three reasons. In other words it would be good to evaluate the failure modes of the dense keypoint matching network more systematically. The separation in views can be measured in terms of the angular difference in 3D. Although the proposed method is for continuous dense keypoints, the said continuous field can be sampled at discrete locations based on pixel locations of the source image and all evaluation metrics for sparse/discrete keypoint methods can then apply. 

4. Section C [Further results] provides further dense keypoint matching results in Figure 2. How should the figure be interpreted? Are the corresponding pairs arranged vertically or horizontally? Is the colour map same as that in the paper. A caption, even if it repeats text from the paper, would be helpful here. 

**Notes**

1. Equation numbering: It would be good to have _all_ equations or math expressions numbered. The equations between Eq(1) and Eq(2) are not numbered? The loss definition in Section 3.2 is also not numbered. 

2. Typos: I think in quite a fewplaces where you refer to 'synchronized images', you actually mean 'corresponding images'. 

3. Typos: Line 289: 'We evaluation', Line 296: 'margain', Line 299: 'does not seen any', Line 300: 'still get improved'

4. Typo? Line 251: Am I correct in understanding that authors mean to refer to DensePose-COCO as the first model in the comparison rather than initial model.

**Overall assessment**
The paper presents a new method for continuous dense keypoint matching that leverages existing ideas to build a system with improved accuracy while using unlabelled images for training. The motivation for the work could be substantiated better. The method is evaluated quantitatively; improvement over existing methods is a little incremental, but noticeable nevertheless. A thorough analysis of what makes the system work or not work (limitations) and some comments about the statistical significance of the results would be welcome. There are a few fundamental issues in the development of the theory that are as yet not very well supported by the results or the reasoning presented in the paper. 
limitations_and_societal_impact: The datasets used are cited but the licenses of the datasets are not cited. A reader can look up the licences though. 
The authors do not mention any negative societal impact, which is largely true because the proposed method will be used as a building block in larger systems. It is the larger systems or the applications that may have positive/negative societal impact. However, the limitations of the method could have been made clearer in the paper. A more thorough evaluation will naturally provide the evidence required to draw conclusions about the limitations of the method. Combinations of limitations of the building blocks used in a larger systems can come together to create blind spots in larger systems, so each block has a role to play. Being upfront about the limitations of each method definitely helps assess the impact of larger systems.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new method for the semi-supervised learning of keypoints using multiview images. The main challenge addressed is finding exact correspondences betweenkeypoints in multiple views, since the inverse of keypoint matching cannot be analytically derived or differentiated. The proposed probabilistic epipolar constraint encodes a soft correspondence and geometric consistency in the correspondence field. A distillation-based regularization is also proposed to avoid degenerative cases. Each contribution is ablated and show significant improvement over the baseline and relative to other methods. 
main_review: This paper's contributions are:

- A novel formulation of probabilistic epipolar constraints that can be used to enforce multiview consistency on continuous dense keypoints fields
- A new neural network design that enable the precise measurement of of probabilistic epipolar error
- A distillation-based regularization term
- Strong performance on real-world multiview image data 

I believe this paper makes a convincing argument for all these claimed contributions, however some further clarification in some points would be useful for my final decision, as described below:

The idea of probabilistic epipolar constraints is well-motivated in terms of multiview keypoint matching, and is well positioned relative to other methods in the related work section. The motivation behind precise measurement of epipolar error, although not particularly novel, has many practical applications beyond improving performance, as a tool for uncertainty estimation. 

The distillation-based regularization term addresses a practical limitation of the method, and multiple papers have already explored distillation / teacher networks to improve the performance of multi-frame networks in such degenerate cases. Although seems like the coefficient for this loss has to be carefully determined, so the network is allowed to deviate from it starting values, but not too much in the degenerate cases. Some analysis on the impact of varying this loss would have been interesting. Is there any sort of masking to determine in which areas this regularization should be applied, or is it applied equally?

Performance is evaluated on multiple large-scale datasets, with several baselines (including [22], the method that more closely relates to the setting the proposed method operates on), metrics, and qualitative results. The ablation table shows the impact of each component, however seems like L_T (photometric consistency loss) has very minor impact on the final performance. The same can be said for the generalization performance and comparison with other methods, improvement seems to be much higher for geometric and reconstruction accuracy than for keypoint accuracy. Some reasoning as to why that's the case would be interesting. 


limitations_and_societal_impact: The authors mention scalable behavioral analysis as a potential societal impact of this work, since it will enable the study of animal life's individual and social behaviors by eliminating the need of annotated data for image processing.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a technique that exploits multi-view consistency for learning dense correspondences in images coupled with a densepose-like mapping in texture space. They performed ablations and SOTA comparisons on H3.6, and further SOTA comparisons on Ski-pose and Monkey.
main_review: The paper proposes an interesting technique that introduces multi-view consistency for dense pose estimation. What I would have liked to see is that multi-view consistency allows one to further increase the precision of computed correspondences, but what the authors discover is that by accounting for multi-view consistency... you get more consistent mappings (duh?) – in other words it is a paper that does well on a metric, because they introduced a metric for which other techniques would fail.
So overall, the emerging question is `why should we care?` and `can you show an application scenario where multi-view consistent keypoints are useful?`.
Other than this core questions, I have other concerns about the paper itemized in the remarks below (to which I expect an answer). Note that I am *more than happy* to raise the score if my concerns are properly addressed (the opposite is also true).

## Core remarks

1. My core complaint relates to eq(5,7), and unless properly addressed would prevent me from letting this paper being accepted. First of all, I was left wondering why you needed both of them until I read L217-L224 (this should be addressed). But most importantly, I do not buy the argument provided by (6), and the massive drop in performance in the ablation in Table.1 (from 0.5 to 0.12).... even more so given that the supervised method performs much better (at 0.48)!!! In particular, (5) provides pointwise supervision to the value of the maps, leaving no affine nullspace in the supervision of $\varphi$. To prove this, let us assume the ideal mapping function is an identity function $\varphi(.)=I(.)$, then your statement would say $I(x)==2I(x)$ which is only true for $x=0$ (hence generally false)? So, while I think this argument is flawed, I can guess what happened in Table.1: the aggregated magnitude of (5) is significantly higher in value than the one in (7) (e.g. you forgot normalization?), and hence you are simply retrieving different pareto-optimal solutions (i.e. the magnitude of $\mathcal{L}_R$ and $\mathcal{L}_M$ are similar, while $\mathcal{L}_L$ is much smaller... in other words $\mathcal{L}_M$ is overpowering $\mathcal{L}_L$, which is also consistent with the fact that the geometric consistency score is the lowest across the entire set of experiments).
1. I find it quite disturbing that the paper continuously refers to "geodesic" distances and metric. More specifically, despite the fact that given a piecewise parameterization of geometry, distances between texture map coordinates $u$ *do not* represent geodesic distances. Even if you used small patches and quasi-isometric parameterization, this would only be a decent approximation in small neighborhoods (as isometric maps are only possible for developable surfaces with zero Gaussian curvature). The fact that [8] was sloppy in employing this term does not justify the entire field to adopt it too.
1. L144: `argmin is not differentiable` this is only partially true, as otherwise any other paper that uses Chamfer Distance would not exist in geometric deep learning? Further, the *necessity* to convert from min to softmin has not been ablated at all... why?
1. L158: It is not clear at all to me where the visibility $v$ comes from? Usually pairwise visibility is ensured by cycle consistency... but I see no such loss. Are you assuming visibility to be provided as part of the dataset for every single image? This is a severe limitation, and should be stated as such.
1. Considering part of the paper is related to uncertainty for keypoints, I would consider appropriate an acknowledgement of works in this area (e.g. UGGLI amongst many others... Google it?).

## Further remarks
1. Sec.4: The writing in the technical section could be improved a lot... it would have been a lot faster to digest eq(2) directly, and saying the first term measures for the difference in how two points mapped in texture space, and the latter measures epipolar correctness, and then diving into details.
1. Table.1: it would have been a lot more informative if the ablation would disable one loss at a time, rather than progressively adding them? Further, Why the scores in Ours(comparison) and Ours(ablation/last row) *do not match*?
1. Fig.1 in print you can barely see anything, recommend increasing the image transparency, or increase brightness
1. L57, L??: you state that you generalize the Samson distance (twice), yet never point out the difference
1. L138: missing eq# and $\Theta$ is undefined
1. L149: what's the `range of the body surface?`
1. L162-166: this seems repeated text with zero new information? (repeated earlier in the main body, as well as in Fig.2)
1. L181, L192: I am not exactly sure what `when DensePose is used the field is defined for each body part` and `we augment the loss for body parts` mean. Are you intending the the function $\Phi$ is actually a collection of mapping functions $\{\Phi_k\}$, one per body part (indexed by `k`)?
1. L203: if that's the cardinality of the foreground pixels, wouldn't that make your method choke in terms of memory on reasonably (e.g. 1MP) size images with $\approx O(1e9^2)$ complexity?
1. L207: `design a new operation` not sure I see anything new, isn't this just a vectorization of the same operation? Seems like an implementation detail more than anything else. Am I mistaken?
1. L207-8: missing equation number!!!
1. L187: infinitely → infinite
1. Tab.1: accruacy → accuracy
1. L173-5: if you use $\underbrace{\mathcal{L}}_{\text{photometric loss}}$ you can omit this piece of text and save some space for more useful insight
1. L131: consider explaining why this mapping is injective in your setting?
1. L289: We evaluation → We evaluate

# UPDATE raised my score 6→7 thanks to the rebuttal+discussion
limitations_and_societal_impact: I am not sure people tracking cannot be seen has having some impact on society (monitoring, recognition, tracking), and not acknowledging this seems a bit naive.

Limitations are in the supplementary, but a short discussion should be moved to the main paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a semi-supervised approach for dense keypoints detection (assining a unique ID or a UV coordinate to an image of a human body). The key idea of the paper is an epipolar inconsistency loss function which can be computed and trained with multiview datasets. The approach has been evalauted on multi-view human datasets compared against single-view and multi-view techniques. Experimental results are OK, but not very great.
main_review: I was not very excited with the paper. I initially liked 2 ideas of the paper: 1) Using epipolar errors does not require extra labels. 2) The keypoint detection engine uses only a single image, and the model can be used for monocular sequences.

However, I was not really convinced (before seeing results) that the first idea is a good one or not. The epipolar error seems very weak and I was in doubt if this really does anything on top of exact supervision. Since the paper is all about semi-supervised, this additional epipolar loss must add significant performance boost against a system without this epipolar loss. Then, I look at the experimantal results and found that the effects of the epipolar loss is not really verified as I suspected. The ablation study must show their system just without L_M in Table 1. This ablation study basically says that L_M does not matter.

I do not really care about geometric consistency metric (geom consistency metric in table 1 and the entire Table 2), because this is the error which the proposed method seeks to minimize. The real performance should be mesure by the real pin-point answers. On this point, Table 1 (Comparison block) is not very exciting. Ignoring (geom consistency column), the proposed approach is the best in 3 out of 6 columns only. And even where they win, the performance gap is very minor at the point that I would rather suspect the engineering difference (likely authors spent more time tweaking their system on getting good numbers). Overall, the performance improvement does not seem to justify a bit complex technical formulation from not-so-convincing idea.

Lastly, on my second point above, the architecture does not mix 2 images which means that the system can be used for monocular sequences. I thought that this is the exciting piece and I wished to see evaluations on monocular sequences say optical flow datasets. The evaluations are solely based on mult-view datasets, which is understandable as generalization is often a problem and they have to train and test on the same dataset. But, a potentially interesting aspect of the paper diminished and I got less excited.

Overall, I did not find this paper exciting. I started with a strong doubt that the main idea may not be very effective. And a bit complicated thing must happen to implement this idea, but overall, the metrics of the idea are not demonstrated and I beleive that there are no such benefits.

--------------
The rebuttal provided new experimental validations and clarifications. My concenrs are resolved and I support the acceptance of the paper.
limitations_and_societal_impact: The paper says that the negaive impact is minimum in onen sentence. It rather focuses on descriving postive aspects. This is not a problem to me at all. I do not see issues on negative impacts from this paper.
review_title: null

Meta-review: Reviewers agreed that this is a solid paper that deserves acceptance. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version. 
Decision: Accept (Spotlight)