Dense Keypoints via Multiview Supervision
Authors: Zhixuan Yu
Haozheng Yu
Long Sha
Sujoy Ganguly
Hyun Soo Park
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Dense keypoint estimation
Multiview supervision
Dense epipolar geometry
Semi-supervised learning

Abstract: This paper presents a new end-to-end semi-supervised framework to learn a dense keypoint detector using unlabeled multiview images. A key challenge lies in ﬁnding the exact correspondences between the dense keypoints in multiple views since the inverse of the keypoint mapping can be neither analytically derived nor differentiated. This limits applying existing multiview supervision approaches used to learn sparse keypoints that rely on the exact correspondences. To address this challenge, we derive a new probabilistic epipolar constraint that encodes the two desired properties. (1) Soft correspondence: we deﬁne a matchability, which measures a likelihood of a point matching to the other image’s corresponding point, thus relaxing the requirement of the exact correspondences. (2) Geometric consistency: every point in the continuous correspondence ﬁelds must satisfy the multiview consistency collectively. We formulate a probabilistic epipolar constraint using a weighted average of epipolar errors through the matchability thereby generalizing the point-to-point geometric error to the ﬁeld-to-ﬁeld geometric error. This generalization facilitates learning a geometrically coherent dense keypoint detection model by utilizing a large number of unlabeled multiview images. Additionally, to prevent degenerative cases, we employ a distillation-based regularization by using a pretrained model. Finally, we design a new neural network architecture, made of twin networks, that effectively minimizes the probabilistic epipolar errors of all possible correspondences between two view images by building afﬁnity matrices. Our method shows superior performance compared to existing methods, including non-differentiable bootstrapping in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.

Semi-supervised Dense Keypoints using Unlabeled Multiview Images
Zhixuan Yu University of Minnesota
Haozheng Yu University of Minnesota yu00042 4 dumn. edu
Long Sha TuSimple long. sha @ usimple.... ai
yu00006 46 (mmn edu
Sujoy Ganguly Unity
Hyun Soo Park
University of Minnesota hspark@umn edu
sujoy. sujoy.gangulyyuninn... www.ggagglyymmiiyyd.comm
Abstract
This paper presents end-to-end semi-supervised framework to learn dense keypoint detector using unlaheled multiview images. A key challenge lies in finding the exact correspondences between the dense keypoints in multiple views since the inverse of the keypoint mapping ated. This limits applying existing multiview supervision approaches used learn sparse keypoints that rely on the exact correspondences. To address this challenge, we derive a new probabilistic epipolar constraint that encodes the two desired properties. (1) Soft correspondence: we define a matchability, which measures a likelihood of a point matching to the other image's ' corresponding point, thus relaxing the requirement of the exact correspondences. (2) Geometric consistency: every point in the continuous correspondence fields must satisfy the multiview consistency collectively. We formulate a probabilistic epipolar constraint using a weighted average of epipolar errors through the matchability thereby generalizing the point-to-point geometric error the field-to-fiedd geometric error. This gen- eralization facilitates learning a geometrically coherent dense keypoint detection model by utilizing large number of unlabeled multiview images. Additionally, prevent degenerative cases, employ distillation-beal based regularization by using a pretrained model. Finally, we design new neural network architecture, made of twin networks, that effectively minimizes the probabilistic epipolar errors of all possible correspondences between two view images by building affinity matrices. Our method shows superior performance compared existing methods, including non-differentiable bootstrapping in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.
be neither analytically derived nor differenti-
Introduction
The spatial arrangement of keypoints of dynamic organisms characterizes their complex pose, pro- viding a computational representation the way they behave. Recently, computer vision models offer fine grained behavioral modeling through dense keypoints that establish an injective mapping from the image coordinates to the continuous body surface of humans [8] and chimpanzees [36].
These models predict the continuous keypoint field from an image, supervised by a set of densely annotated keypoints, which shows remarkable performance on real-world imagery and brings out
number of applications including 3D mesh reconstruction [54] [53 35 50, 7 21]. texture/style
transfer 226] 37 and geometry learning [14 [1. Nonetheless, attaining such densely annotated data is labor intensive, and more importantly, the quality of the annotations is fundamentally bounded by the visual ambiguity of keypoints, c.g., points on textureless shirt. This visual ambiguity leads suboptimal model when applying out-of -a-mmll distributions. this paper, we present
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Multiviow camera
Multiview images
Geometrically consistent dense keypoints
3D reconstruction
We use unlabeled multiview images to learn dense keypoint model via the epipolar geometry in an end-to-end fashion. As byproduct, we can reconstruct the 3D body surface by triangulating visible regions of body parts.
Figure 1:
new semi-supervised method to learn a dense keypoint detection model from the unlabeled multiview images via the epipolar constraint as shown in Figure
Our main conjecture is that the dense keypoint model is optimal only if is geometrically consistent across views. That is, every pair of corresponding keypoints, independently predicted by two views, must satisfy the epipolar constraint [9]. However, enforcing the epipolar constraint to learn a dense keypoint model is challenging because (1) the ground truth 3D model is unknown and thus the projections of the 3D model cannot be used as the ground truth dense keypoints; (2) the predicted dense keypoints are inaccurate and continuous over the body surface, and therefore, existing multiview supervision approaches [51 38 45 for sparse keypoints are not applicable. In these previous methods, the epipolar constraint was enforced between two keypoints (or features) of which semantic meaning was explicitly defined by finite set of joints (c.g., elbow channel in network); and (3) establishing correspondences across views requires knowing an inverse mapping from the body surface to the image that can be neither analytically derived nor differentiable. These challenges limit the performance of previous work [22 that relies on iterative offline bootstrapping. which is not end-to-end trainable, or requires additional parameters to learn for 3D reconstructior
We tackle these challenges through a probabilistic epipolar constraint by incorporating an uncertainty in correspondences. This new constraint encodes the two desired properties. (1) Soft correspondence: given a keypoint in one image, we define matchability-thh the likelihood of correspondence for all predicted keypoints in another image based on the distance in the canonical body surface coordinate (c.g., texture coordinate). This allows evaluating geometric consistency in the form of a weighted average of epipolar errors over continuous body surface coordinates, eliminating the requirement of exact correspondences. (2) Geometric consistency: we generalize symmetric Sampson distance [9] for all possible pairs of keypoints from two views to enforce the epipolar constraint, collectively. With these properties, we derive a new differentiable multiview consistency measure that is label- agnostic, allowing us to utilize large number of the unlabeled multiview images without explicit 3D reconstruction.
We design an end-to-end trainable twin network architecture that takes pair of images as input and outputs geometrically consistent dense keypoint fields. This network design builds the affinity maps between two keypoint fields based on the matchability and epipolar errors, which facilitates measuring the probabilistic epipolar errors for all possible correspondences. In addition, inspired by knowledge distillation, we use a pretrained model to regularize network learning, which can prevent degenerate cases, Our method shows superior performance compared to existing methods, including lon-iifferentiable bootstrapping [22) in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.
Our contributions include: (1) a novel formulation of probabilistic epipolar constraint that can be used to enforce multiview consistency on continuous dense keypoint fields in a differentiable way; (2) new design of the neural network that enable to precisely measure the probabilistic epipolar error, which allows utilizing a large number of the unlabeled multiview images; (3) a distillation-hasee -based regularization to prevent degenerate model learning; (4) strong performance on real-world multiview image data, including Human3. ..6 [IT). Ski-Pose [40] and OpenMonkeyPoss [3) outperforming
existing methods including non-differentiaalle dense keypoint learning (22).
Broader Impact Statement The ability to understand animals' individual and social behaviors is of central importance to multiple disciplines such as biology, neuroscience, and behavioral science.
An analogous insight has been used for fundamental matrix, directly computed from correspondences that does not require additional variables for 3D reconstruction.

---

Measuring their behaviors has been extremely challenge due to limited annotated data. This approach offers way address this challenge with limited number of annotated data, which will lead to scalable behavioral analysis. The negative societal impact of this work is minimum.
2
Related Work
Our framework aims at training a dense keypoint field estimation model via multiview supervision. We briefly review the related works.
Dense Keypoint Field Estimation Finding dense correspondence fields between two images is challenging problem in computer vision. 3D measurements (e.g.. depth and pointcloud) can provide a strong geometric cues which enable matching of deformable shapes [42] 29 48]. Similarly, in 2D, visual and geometric cues have been used to find the dense correspondence fields in an unsupervised 4 Notably, for special foreground targets such as humans, the dense matching problem can be cast as finding a dense keypoint field that maps pixel coordinates to a canonical body surface coordinates (24] (e.g., DensePose (8). These works were built upon large amount of data labeled by crokesworkers and generalized learn the correspondence fields for face [2] and chimpanzees [36]. Key limitations of these approaches are the inaccuracy of labeling and requirement of large labeled data. We address these limitations by formulating multiview supervision that can enforce geometric consistency, which allows utiliziig a large amount of unlabeled multiview images.
learning [56] 6 4
Multiview Feature Learning Epipolar geometry can be used learn a visual representation by transferring visual information from one image to another via epipolar lines or 3D reconstruction. For instance, fusion layer can be learned to fuse feature maps across views [30] Such fusion models can be factored into shape and camera components to reduce the number of learnable parameters and improve generalizability [49] Given the camera calibration, a light fusion module can be learned to directly fuse deep features [0]] or heatmaps (55) from other view along corresponding epipolar
line. Several works combine multiview image features to form 3D features 13 46
view-invarian
feature in 2D (32].
possess a unique geometric property: images are visually similar yet geometrically distinctive, provided by stereo parallax. Such property offers new opportunity to learn geometrically coherent representation without labels. Bootstrapping by 3D reconstruction [2 can be used to learn keypoint detector supervised by the projection of the 3D reconstruction to enforce cross-view consistency. MONET (51] enables an end-to-end learning by eliminating the necessity of 3D reconstruction and directly minimizing
Multiview Supervision Synchronized multiview images M 15 52
epipolar error. Learning keypoints can be combined with 3D pose estimation 34 12] by enforcing
predicting the same pose in all views while using a few labeled examples with 3D or 2D pose
annotations to prevent degeneration. One can alleviate the need for large amounts of annotations by matching the predicted 3D pose with the triangulated pose 19] Several works further learn latent representation encoding 3D geometry from images [ 25 or 2D pose 33
41 by enforcing
consistent embedding, texture (28). and view synthesis [45] across views. Unlike these approaches
designed for sparse keypoints where the geometric consistency applied on finite points, we study geometric consistency on continuous dense keypoint fields. Capture Dense [22] is the closest work to ours, which uses bootstrapping through 3D reconstruction of human mesh model [16]. However, due the non-differentiable nature of bootstrapping, it is not end-to-end trainable. A temporal consistency has been also used for self-supervise the dense keypoint detector [27].
Method
We present a novel method to learn a dense keypoint detector by using unlabeled multiview images. We formulate the epipolar constraint for continuous correspondence fields, which allows us to enforce geometric consistency between views.
3.1 Dense Epipolar Geometry
Given a pair of corresponding images from two different views, a pair of corresponding points
X ++ x', are related by fundamental matrix given the calibrated cameras, i.c. X 0, where F e 3x3 is the fundamental matrix, and x 12 is a homogeneous representation of x. The
3

---

Fx
X
U
MXX
d(xx''F))
Ex(x))
Figure 2: A dense keypoint field maps point in an image to the canonical body surface coordinate, Le., o(x: I). Establishing correspondence between two view images requires the analytic inverse é which does not exist in general. We present a matchability Mxx, x'; I, ",T"), a likelihood of matching through the body surface coordinate. We combine the matchability with the epipolar error d(x, x' F) to obtain probabilistic epipolar error E(x,x').
measure of geometric consistency between the two images can written as 9
X FX
d(x, x);)
X X+
(FX) (F
where (Fx), the jth entry of Fx. between x' and the epipolar line Fx.
This measures epipolar error that is the Euclidean distance
Consider an injective dense keypoint mapping R + R that maps a pixel coordinate canonical 2D body surface coordinate, i.e., o(x; where x E ((Z) the set of the foreground pixels in the image I, and E R is the 2D coordinate in the body surface as shown in Figure 2 This dense keypoint mapping can be learned from annotations, e.g., DensePose [8] for humans where it maps to the texture coordinate of 3D human body surfaces. This mapping is incomplete because there exists missing correspondences in an image due to occlusion. One can find correspondence between the two images through u, i.e., Φ Iu; + o (u; where and the two view images. However, is injective mapping where the analytic inverse does not exist in general.
Given point in the image I, one measure the expectation of geometrie error by nearest neighbor search on the body surface space:
(xx) = d(x, F), x'= argmin 1Ø(x: - o(x': x'E((Z')
where F(x) the expectation of geometric error at x. The expectation of the geometric error measures the epipolar error over all possible matches in the other view image, i.e., Vx E @
There are two limitations in the nearest neighbor search: (1) The correspondences are not exact, leading biased estimate of the geometric differentiable, which cannot be used to learn a dense keypoint detector in an end-to-end fashion.
expectation. (2) The argmin operation is not
Instead, we address these limitations by making use of a soft correspondence, or matchability likelihood of point being matched to another point as shown in Figure 2
M M(x, ,,,, P (ø(x; (x': I'); (0ly; I') (yee(2))
(3)
where P(u, (2) is the probability of matching between and u':
u u'113
u
v|1
P(u, u' =exp
Σ VEB
exp
(4)
20
20
Ω is the domain of the canonical coordinates, a is the standard deviation that controls the smoothness of matching, e.g., when -> 0, it approximates the nearest neighbor search.
We the matchability form probabilistic epipolar error:
E E(x, x') = M (x, x': I. Ix
(5)
where Exx, x') is the epipolar error between x and weighted by the matchability. The error expectation of x can be computed by marginalizing over all x':
E(x)
Σ
E(x,x').
x'E

---

Iter O
Iter 600
Iter 1200
Iter 2400
Iter 4800
Figure 3: Our multiview supervision progressively minimizes the epipolar error between two views (top and bottom) as learning the dense keypoint detection model. The keypoint detection, indepen- dently by a pretrained model (Iter 0), is not geometrically consistent. As the optimization progresses, the error is significantly reduced, resulting in a geometrically coherent model.
That is. the expectation of the geometric error of x measures a weighted average of epipolar errors for all possible correspondences. The higher matchability, the more contribution to the error expectation. Unlike Equation 2 Equation (6) does not include the non-differentiable argmin operation. Further, it takes into account all possible pairs of correspondences between two images, which eliminates the bias introduced by the nearest neighbor search.
Given the expectations eтror that measures geometric consistency between two images:
the dense keypoint field, derive a symmetric dense epipolar
E(I,T')
1
Σ XEE(I)
v(x; o. d, + V
Σ
x'E((I)
where vxx: o, ",I'' E {0,1} } is visibility indicator:
1, 7xe O(T)) llox),, ((x';I')Il <E
o(x; (x
0. otherwise.
V= V= ExerI) vxx; o, D,)) and V Ex , are the numbers of foreground pixels visible in I' and I, respectively. is correspondence tolerance defined in the canonical surface coordinate.
In fact, this dense epipolar error is generalization of Sampson distance p that defines epipolar error between explicit point correspondences (point-to-point). We generalize it to model probabilistic correspondence between
of points (feld-to-field).
Existing approaches such as bootstrapping 222 establish the matching through 3D reconstruction of mesh and enforce the geometric error in an alternating fashion due to the non-differentiability of matching. Our differentiable formulation allows learning the dense keypoint detector in an end-to-end manner, which is flexible and shows superior performance.
3.2 Multiview Semi-supervised Learning
We learn the dense keypoint detector by minimizing the following error:
(9) where Li. is the labeled data loss, LM is the multiview geometric consistency loss, LR is the regularization loss, and Lx is the multiview photometric consistency loss. >₁ AM, ÅR and Xy are the weights to control their relative importance.
L = >r.Ct. + AMeM + ARCR ATLT,
Supervised Loss We learn the dense keypoint detector from the labeled dataset Dt, by minimizing the following error:
L1
Σ
Σ
Ux 6(x: D)
(10)
{I,UHED, xEO(I)
5

---

where U E R2x is the ground truth canonical surface coordinates for dense keypoints, Ux is the ground truth canonical surface coordinate x.
Multiview Geometric Consistency Loss We learn the dense keypoint detector from the unlabeled multiview dataset Du by minimizing dense epipolar error over image pairs:
LM
E EII,T'),
(11)
(I.I')EDo
where EE I') is the dense epipolar error between two corresponding images and I', defined in Equation (1.. This loss progressively minimizes the dense epipolar error between image pair as learning the dense keypoint detection model as shown in Figure
Distillation Based Regularization Loss Enforcing multiview consistency alone can lead to de- generate cases. For instance, consider linear transformation in the body surface, e.g., v Tu where T E Rx is similarity transformation. Any 6 that satisfies the following condition can be equivalent dense keypoint detector:
(12) This indicates that there exist an infinite number of dense keypoint detectors that satisfy the epipolar geometry. To alleviate this geometric ambiguity, we use a ioiillation-baseee regularization using pretrained model. Let Φp be dense keypoint detector pretrained by the labeled data. We prevent the learned detector Φ from deviating too much from the pretrained detector do for the unlabeled data:
(xx: I) = T),
LR Σ
Σ
[]do(x: 6(x;
(13)
TEDO XEB(I)
where CR is the loss for the distillation based regularization, minimizing the difference from the pretrained model.
Multiview Photometric Consistency Loss We leverage photometric consistency across views. As- suming ambient light, pixels across views corresponding to the same 3D point in space should have the same RGB value. Similar to dense epipolar error E (I,T'), we define dense photometric error TII as:
1 Σ ,, V
TIIIT)=
o(x':,,,,II)T(x')
(14)
V
xEB(I)
x'EO(I')
where T(x) is the expectation of photometric eπror of x similar to E(x). i.e. T(x) =
Exee(z) T(x, x') where x'x, = M (x, x': ó [z,'))(( T'x'']13
Then we compute multiview photometric consistency loss as:
CT
Σ T(I,I', (L.
(15)
3.3 Network Design
We design new network architecture composed of twin networks to learn the dense keypoint detector by enforcing multiview consistency over dense keypoint fields shown in Figure Each network is made fully convolutional network that outputs the dense keypoint field per body part.
Given two dense keypoint fields, we compute the probabilistic epipolar error by constructing two affinity matrices: matchability matrix and epipolar matrix, M, E E R ([(Z)) where (O(Z) is the cardinality of the range of foreground pixels. These two matrices are defined by:
Mi- = M(,,, Xj; o, I'), Ei = d(x,, xx; F)
(16)
where Mij is the i, entry of the matrix M. x; and x, are the tth and jth foreground pixels from the images I and I', respectively. We also compute the visibility maps V, = v(x; o. I,T' and V v(x:; o, I'I"..
We design operation to measure the dense epipolar error in Equation
E(I,T)
( E)
+ V
T (M O E) 1
(17)
where 1, is the nn-iimenionnal vector of which entries are all one. @ is the element-wise multiplication of matrices. Note that dense epipolar error TII,I') can be computed following the same operations.

---

x u
4
Flatten
I
E(X
M
E
x
C
E
Figure 4: We design a new architecture composed of twin networks that detect dense keypoint fields. The dense keypoint fields from two views are combined to form two affinity matrices: matchability M and epipolar error E. M is obtained from the dense keypoint fields (u and u'), and E is obtained from the epipolar еπrоr of pixel coordinates (x and x'). These matrices allow us to compute dense epipolar errors and subsequent multiview geometric consistency loss LM- Same operations are applied to compute Lr. addition, we make use of distillation- based regularization using a pretrained model on to avoid degenerate cases (CR). We measure the labeled loss LL if the ground truth dense keypoint field is available. @ is the clement-wise multiplication of matrices. @ is the minus operation between dense keypoint predictions.
Experiments
We perform experiments human and monkey targets as two example applications to evaluate the effectiveness of our proposed superviee learning pipeline.
4.1 Implementation Details
We use HRNet [18
the backbone network followed by four head networks made up of convolutional layers to predict foreground mask, body part index, and UV coordinates on the canonical body surface, respectively. Each network takes an input a x24x222 image and outputs 15-channel (for foreground mask head only) or 25-channel 56 feature maps S
We train the network in two stages. In the first stage, train an initial model using the labeled data by the labeled data loss Lt. Specifically, for the human dense keypoints, we use 48K human instances in DensePose-COO COCO { training set to train the initial model. For the monkey dense keypoints, we directly the pretrained model for chimpanzees (36) the initial model since do not have an access to the labeled data. the second stage, we train our model on unlabeled multiview data leveraging all losses. The initial model is used for two purposes:
(1) the pre- pre-trained model do with weights fixed for distillation-ban biailatioo----ddddd- regularization and (2) the initial model to be refined via multiview supervision.
4.2 Evaluation Datasets
Human3.6M [I]
a large-scale indoor multiview dataset captured by 4 cameras for 3D human pose estimation. It contains 3.6 millions of images captured from 7 subjects performing 15 different daily activities. e.g. Walking, Greeting, and Discussion. Following common protocols, we use subject SI, S5, S6, S7 and S8 for training, and reserve subject $9 and $11 for testing. Following 54 and (33 we leverage SMPL 24 parameters generated by HMR [17] via applying MoSh [23] to the sparse 3D MoCap marker data recover ground truth 3D human meshes. We further perform Procrustes analysis [9] to align them with ground truth 3D poses in global coordinate system and then render ground truth IUV maps using Py Torch31) (31. Since it is the only multiview human dataset that we have access to ground truth 3D mesh IUV maps, we use this dataset to perform comprehensive experiment and studies.
3DPW [47 an in-the-wild dataset made of single view images with accurate 3D human pose and shape annotations. We use the test split of this dataset for dense keypoint accuracy evaluations (31) metrics do not apply). Ground truth IUV maps are rendered from 3D pose and shape annotations by leveraging the provided camera parameters.
Ski-Pose PTZ-Camera Dataset 40 a multiview dataset capturing competitive skiers performing giant slalom runs. 6 synchronized and calibrated pant-tile-zoom-cane (PTZ) cameras are used to track single skier at a time. The global locations of the cameras were measured by tachymeter
7

---

theodolite. It contains 8.5K training images and 1.7K testing images. We use this dataset to evaluate generalization towards in-the-wild multiview settings. We use its standard train/test split to train and evaluation our model. We select adjacent view pairs to form training samples.
OpenMonkeyPoose [3] is a large landmark dataset of rhesus macaques captured by 62 synchronized multiview cameras. It consists of nearly 200K labeled images with four macaque subjects that freely move in large cage while performing foraging tasks. Each monkey instance is annotated with 13 2D and 3D joints. We use this dataset to show our model's ability on transferring dense keypoints to monkey data, We split about 64K images for training and 12K images for testing. For training, we generate the densepose of monkey data using a pretrained model [36] pseudo-label These pseudo-labels are then used for refining the pretrained model.
4.3 Baselines
In our experiments, we consider four baselines: (1) A model fully-supervised by labeled data, i.c.
DensePose-COCO [8]. which is also our initial model for semi- cemi-supervised learning. We refer to this as Supervised in the following. (2) A model trained using multiview bootstrap strategy [38] 22]. where multiview triangulation results from previous stage
used as the pseudo ground truth. (3) A learnable 3D mesh estimation method where dense keypoint estimation can be acquired by reprojecting 3D mesh to image domain, c.g. HMR [17]. (4) A 3D mesh estimation framework similar to HMR but additionally incorporating Model-fittin fittinggiigigg in the Loop, e.g. SPIN (20). Supervised is also used as an initial model for our approach.
4.4 Metrics
We evaluate the performance of our dense keypoint model using metrics from three aspects: (1) geometric consistency, (2) accuracy of dense keypoints, and (3) accuracy of 3D reconstruction from multiple views.
Geometric Consistency We use epipolar distance (unit: pixel) averaged over views and frames the metric for evaluating multiview geometric consistency. Ideally, two dense keypoints corresponding to the same point on 3D surface should have epipolar distance equal to This metric can be evaluated on any multiview dataset with ground truth camera parameters available.
Dense Keypoint Accuracy We evaluate the model's performance dense keypoint accuracy from two aspects: (1) Ratio of Correct Point (RCP) and (2) Ratio of Correct Instances (RCI). RCP evaluates correspondence accuracy over the whole image domain. Specifically, it records the ratio of foreground pixels on images with corresponding 3D body surface correctly predicted as function of geodesic distance threshold. where the prediction is considered correct if its geodesic distance to the ground truth is below the threshold (10em and 30cm). RCIs consider instance- wise accuracy where an instance is declared to be correct if its geodesic point similarity (GPS) [8] is above the threshold. We also report the mean RCI (mRCI) and the mean GPS for all instances (mGPS).
Reconstruction Accuracy Given dense keypoints, we measure 3D reconstruction error by trian- gulating them in 3D. We compute Mean Per mesh Vertex Position Error (MPVPE) as the metric for reconstruction accuracy, which is defined as the mean euclidean distance between triangulated vertices and corresponding ground truth ones. In addition, inspired by geodesic point similarity (8),
(x (_d((e-v.) where d(v; -v)) is the we define vertex similarity as: VS = / Σv,ev exp (--22- 2x
euclidean distance between triangulated vertex V; and corresponding ground truth one v;, and V is the set of visible ground truth vertices from both views. K is a normalizing parameter. For v; that does not correspond the triangulated vertex, it is set to infinity. Further, account for false positives in triangulated vertices (vertices not visible from both views), we define masked vertex similarity (MVS) as MVS = vVS. I, where I is the intersection over union between the set of triangulated vertices and V. We report mean MVS (mMVS) over all instances.
4.5 Evaluation on Human3.6M Dataset
We use Human3.66 dataset to perform (1) comprehensive cross-method evaluations, (2) study on model's generalizability towards new views, and (3) ablation study on losses used for training. Results are summarized in Table with all metrics reported.
Cross-method Evaluation We evaluate the performance of our model against other methods as shown in the comparison block in Table and show qualitative results (1st column in Figure 5). For the metrics of keypoint accuracy and geometry consistency, ours outperforms all other methods
8

---

Keypoint accuracy
Goom, consistency
Recon. accuracy
Method
AUC
AUC
mBC1
mGPS
Epi. error
MPVPE
mMVS
Goneral. Comparison
Supervised
0.445
0.729
0.761
0.856
6.09
60.04
0.498
Bootstrapping
0.732
5,90
58.56
HMR DN
0,513
0.701
0.610
0,780
3.67
50.10
0.831
SPIN A0
0,472
0.633
0.459
0.704
3.32
50,46
0.725
Ours
0.485
0.745
0.770
0.861
2,05
53.04
0.561
Supervised (Test view 1,3)
0.428
0.724
0.760
0.855
5.87
58.98
0.521
Ours (Train view 0,27 Test view 1,3)
0.454
0.734
0.766
0.858
3.62
55.38
0.555
Supervised (Test view 0,2)
0.460
0.735
0.762
0,856
5,83
$7.90
0.462
Ours (Train view 1,3/ Test view 0,2)
0.519
0.756
0.771
Supervised (C1)
0,445
0.729
0.761
0,856
6.09
60.04
0.498
Lt. + LM
0.113
0.438
0.710
167.61
Ablation
Ct. + CT
0.164
0,519
0.569
0.759
6.03
128.26
0.510
+ Caa + fr
0.126
0.471
0.509
0.730
1.97
138.43
0.409
Ct. + LR
0.445
0.730
0.762
0.857
5.94
59.64
0.492
Lt. + Ln Exr
0.475
0.741
0.768
0,860
2,06
57.92
0.535
LL + La CT
0.458
0.734
0.764
0.858
5.13
54.88
0.540
Ct. + CR Cx Cr
0.486
0.745
0.770
0.861
2.05
$3.04
0.561
Table 1: We performance cross-methen evaluation, study on model's generalizability towards new
views and ablation study on Human3,6 6M dataset and report performance on keypoint accuracy, geometric consistency and reconstruction accuracy, Note that HMR and SPIN are trained with the 3D ground truth eettmmmtttted mesh) of Human3 3.6M while other algorithms including ours predict dense keypoints without the 3D ground truth. (Epipolar error unit: pixel; MPVPE unit: mm)
(except for AUC only second to HMR 17)) Note that HMR and SPIN are trained with the 3D ground truth (pre-estimated mesh) of Human3.6M while other algorithms including ours predict dense keypoints without the 3D ground truth. Having the 3D ground truth is a significant advantage while it requires substantial additional amount of 3D annotation effort. It is expected to outperform the ones without it, in particular, on reconstruction accuracy metrics. We included these baselines to provide the upper bound of our semi-supervised method as a reference. Note that despite the lack of the 3D ground truth, ours still outperforms HMR and SPIN by margin of up 26.2% and 67.89 67.8% on keypoint accuracy metries on three keypoint accuracy metries. Compared to "'Supervised" and "Bootstrapping" which are close to our setup, ours outperform on all metrics by a margin of up to 9. 9.2% and 7.0% keypoint accuracy metrics.
Generalizability towards new views We evaluate generalization by testing on different views: two views are used for training and other two views are used for testing. The results are summarized in the generalization block in Table As can be seen, although model trained purely on one camera pair does not use any sample captured by the other pair, its performance still improves on all metrics on top of Supervised model by margin of 0.6%- geometric consistency, and 6.5% 13.7% on reconstruction accuracy. This shows that model trained by our semi-supervised approach can be generalized to new views.
on keypoint accuracy, 38.3% 47.9% on
Ablation Study We conduct an ablation study to evaluate the impact of each loss. The results reported the ablation block in Table Note that we propose semi-super learning thus LL is always used. The inferior performance of model trained with L1. + LM or Lt. + LT or Lt. + Lm + LT proves that refining initial model by multiview loss alone suffers from degeneration (Section 3.2). This limitation can be addressed by adding the regularization CR. Note that regularization itself does not add any value: it is identical to the supervised model in theory. This is empirically verified since LT, + LR performs similarly to L, Both multiview consistency losses show effectiveness: L1, + LR. + L'M and L1, + LR + LT outperform Lt. + LR all metrics by margin up 6.5% 2.7% for keypoint accuracy metrics and 8.7% 9.8% for reconstruction accuracy metrics respectively. LL + LR + Lm + Lr further achieves better results on all metrics. Multiview geometric consistency loss mainly contributes to the improvement on keypoint accuracy and geometric consistency metrics, while multiview photometric consistency loss mainly contributes to reconstruction accruacy metrics.
4.6 Evalution on 3DPW Dataset
Keypoint accuracy
To further validate the usefulness of our method in terms of dense keypoint ac- curacy, in addition to detection accuracy on multiview image data, we conduct an-
Method
AUC11
AUCaa
mRCI
mGPS
Compursonn
Supervised
0.398
0.678
0.645
0.786
Rootsrapping
0.397
0.677
0.640
0.784
HMR [7
0.378
0.607
0.472
0.697
0.591
0.656
other cross method evaluation on 3DPW dataset (single-view dense keypoint de- tection) on the keypoint accuracy met-
Ours
0.432
0.693
0.653
0.790
Table 2: We performance cross-method evaluation on 3DPW dataset and report performance on dense keypoint accuracy.

---

Human3.66 6M
SKI-PosePTZ
Open Monkey Studio
Supervised
Bootstrap
HMR
N/A
SPIN
N/A
Ours
Figure 5: Qualitative results Human3.6M, Ski-Pose PTZ-Camera and OpenMonkeyPose Datasets.
Heatmaps overlapping on images indicate epipolar error for each pixels.
rics, i.e., training on Denselose--COe
with Human3.63 multiview supervision and testing on 3DPW without an adapta- tion, as shown in Table 2 The results show that our trained model with geometric consistency is more generalizable than the baselines.
4.7 Evalution on Ski-Pose PTZ-Camera Dataset and OpenMonkeyPose Dataset
We evaluate our method multiview in-the-wild datasets: Ski-pose and OpenMonkeyPosee Since no ground truth is available, we evaluate geometric consistency summarized in Table and show qualitative results (2nd and 3rd columns in Figure 5). The results show that our model outperforms other baselines by a large margin: 37.2%-54. % on Ski-Pose and 31. 31.9%-47.5% on OpenMonkey which can be visually identified by qualitative results.
Method
Ski-Pose (10)
Monkey 3
Supervised
11.83
Bootstrap
22
8.64
9.12
HMR
11
13.40
N/A
SPIN
20
11.83
N/A
Our
5.43
6,21
Table Comparison on geome- try consistency for in-the-wild data (Epipolar error unit: pixel).
5
Conclusion
We present a novel end-to-end semi-supervised approach to learn dense keypoint detector by leverage a large amount of unlabeled multiview images. Due to the nature of continuous keypoint representation, finding exact correspondences between views is challenging unlike sparse keypoints. We address this challenge by formulating a new dense epipolar constraint that allows measuring field-to-field geometric еттоr without knowing exact correspondences. Additionally, we proposed distillation-bases regularization to prevent degenerated cases. We design made of twin networks that can effectively measure the dense epipolar error by considering all possible correspondences using affinity matrices. We show that our method outperforms the baseline approaches in keypoint accuracy, multiview consistency, and reconstruction accuracy.
network architecture
10

---

Acknowledgement
This project is partially supported by NSF IIS 1846031 and NSF IIS 2022894.
References
[1] T. Alldieck, G. Pons-Moll, C. Theobalt, and M. Magnor. Tex2shape: Detailed full human body geometry from single image. In ICCV, 2019.
[2] R. Alp Guler, G. Trigeorgis, E. Antonakos, P. Snape, S. Zafeiriou, and I. Kokkinos. Densereg: Fully convolutional dense shape regression wn-the-wild. In CVPR, 2017.
[3] P. C. Bala, B. R. Eisenreich, S. B. M. Yoo, B. Y. Hayden. H. S. Park, and J. Zimmermann. Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nature Communications, 2020.
[4] H. Bristow, J. Valmadre, and S. Lucey. Dense semantic correspondence where every pixel is a classifier. In ICCV, 2015.
X. Chen, K.-Y. Lin, W. Liu, C. Qian, and L. Lin. Weakly-supervised discovery of geometry-aware representation for 3d human pose estimation. CVPR, 2019.
[6] U. Gaur and B. Manjunath. Weakly supervised manifold learning for dense semantic object correspondecce. In ICCV, 2017.
[7] R. A. Guler and I. Kokkinos. Holopose: Holistic 3d human reconstruction in-the-wild. In CVPR, 2019.
[8] R. A. Güler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, 2018.
[9] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge University Press, second edition, 2004.
[10] Y. Hc, R. Yan, K. Fragkiadaki, and S.-I. Yu. Epipolar transformers. In CVPR, 2020.
[11] C. lonescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3, 66m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2013.
[12] U. Iqbal, P. Molchanov, and J. Kautz. Weaaly-supervised 3d human pose learning via multi-view images in the wild. In CVPR, 2020.
[13] K. Iskakov, E. Burkov, V. Lempitsky, and Y. Malkov. Learnable triangulation of human pose. In ICCV, 2019.
[14] Y. Jafarian H. S. Park. Learning high fidelity depths dressed humans by watching social media dance vidcos. In CVPR, 2021.
[15] H. Joo, Simon. X. Li, H. Liu, L. Tan, Gui, Banerjee, T. Godisart, B. Nabbe, Matthews, et Panoptic studio: massively multiview system for social interaction capture. TPAMI, 2017.
[16] H. Joo, T. Simon, and Y. Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In CVPR, 2018.
[17] A. Kanazawa, M. J. Black, D. Jacobs, and J. Malik. End-to-end recovery of human shape and pose. In CVPR, 2018
[18] D. M. B. x. D. I. 7. Z. W. Ke Sun, Zigang Geng. Bottom-up human pose estimation by ranking heatmap- headmap-guided adaptive keypoint estimates. arXiv, 2020.
[19] M. Kocabas, S. Karagoz, and E. Akbas. Self-supervised learning of 3d human pose using multi-view geometry. In CVPR, 2019.
[20] N. Kolotouros, Pavlakos, M. Black, and K. Daniilidis, Learning to reconstruct 3d human pose and shape via model-fitting the loop. ICCV, 2019.
[21] N. Kolotouros, G. Pavlakos, and K. Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. In CVPR, 2019.
[22] X. Li, Y. Liu, H. Joo, Q. Dai, and Y. Sheikh. Capture dense: Markerless motion capture meets dense pose estimation. arXiv, 2018

---

[23] M. Loper. N. Mahmood, and M. J. Black. Mosh: Motion and shape capture from sparse markers. SIGGRAPHI Asia, 2014.
[24] M. Loper. N. Mahmood, J. Romero, G. Pons-Moll and M. J. Black. Smpl: A skinned mallti-peroon linear model. SIGGRAPH Asia, 2015.
[25] Mitra, B. Gundavarapu, A. Sharma, and A. Jain, Multiview-coonsisten semi- supervised learning for human pose estimation. In CVPR, 2020.
[26] N. Neverova, R. A. Guler, and Kokkinos. Dense pose transfer. In ECCV, 2018.
[27] N. Neverova, J. Thewlis, R. A. Guler, I. Kokkinos, and A. Vodaldi. Slim densepose: Thrifty learning from sparse annotations and motion cues. In CVPR, 2019.
[28] G. Pavlakos, N. Kolotouros, and Daniilidis. Texturepose: Supervising human mesh estimation with texture consistency. In ICCV, 2019.
[29] G. Pons-Moll, J. Taylor, J. Shotton, A. Hertzmann, and Fitzgibbon. Metric regression forests for correspondence estimation. IJCV, 2015.
[30] H. Qiu, C. Wang. J. Wang, N. Wang, and W. Zeng. Cross view fusion for 3d human pose estimation. In ICCV, 2019.
[31] N. Ravi, J. Reizenstein, D. D. Novotay, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv, 2020.
[32] E. Remelli, S. Han, Honari, P. Fua, R. Wang. Lightweight multi-view 3d pose estimation through camera-disentangleed representation. In CVPR, 2020.
[33] H. Rhodin, M. Salzmann, and P. Fua. Unsupervised geometry representation for 3d human pose estimation. In ECCV, 2018.
[34] H. Rhodin, J. Spörri, I. Katircioglu, V. Constantin, F. Meyer, E. Müller, M. Salzmann, and P. Fua. Learning monocular 3d human pose estimation from multi-view images. In CVPR, 2018.
[35] Y. Rong, Liu, C. Li, Cao, and C. C. Loy. Delving deep into hybrid annotations 3d human recovery in the wild. In ICCV, 2019.
[36] A. Sanakoyeu, V. Khalidov. M. S. McCarthy, A. Vedaldi, and N. Neverova. Transferring dense pose proximal animal classes. In CVPR, 2020.
[37] A. Shysheya, E. Zakharov, K.-A. Aliev, R. Bashirov, E. Burkov, K. Iskakov, A. Ivakhnenko, Y. Malkov,
I. Pasechnik, D. Ulyanov, et al. Textured neural avatars. In CVPR, 2019.
[38] T. Simon, H. Joo, L. Matthews, and Y. Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR, 2017.
[39] O. Sorkine- Sorkine-Horuung M. Rabinovich. Least-squares rigid motion using svd. Computing, 2017.
[40] J. Spörri. Reasearch dedicated to sports injury prevention-the sequencee of prevention 'on the example of alpine ski racing. Habilitation with Venia Docendi in Biomechanics, 2016.
[41] J. Sun, J. Zhao, L.-C. Chen, F. Schroff, H. Adam, and T. Liu, View-invariant probabilistic emhedding for human pose. In ECCV, 2020.
[42] Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. CVPR, 2012.
[43] J. Thewlis, S. Albanie, H. Bilen, and A. Vedaldi. Unsupervised learning of landmarks by descriptor vector exchange. In ICCV, 2019.
[44] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object frames by dense equivariant image labelling, arXiv, 2017.
[45] J. Thewlis. H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks by factorized spatial embeddings. In ICCV, 2017.
[46] H. Tu. C. Wang, and Zeng. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment, arXiv, 2020.
[47] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
12

---

[48] L. Wei. Q. Huang, D. Ceylan, E. Vouga, and H. Li. Dense human body correspondences using convolutional networks. In CVPR, 2016.
[49] R. Xic, Wang, Y. Wang. Metafuse: pre-trained fusion model for human pose estimation. In CVPR, 2020.
[50] Y. Xu, S.-C. Zhu, and T. Tung. Denserac: Joint 3d pose and shape estimation by dense render-and-compare. In ICCV, 2019.
[51] Y. Yao, Y. Jafarian, and H. S. Park. Monet: Multiview semi cemi-supervised keypoint detection via epipolar divergence. In ICCV, 2019.
[52] Z. Yu, S. Yoon, Lee, P. Venkatesh, J. Park, J. Yu, S. Park. Humbi: large multiview dataset of human body expressions. CVPR, 2020.
[53] W. Zeng, W. Ouyang, P. Luo, W. Liu, and X. Wang. 3d human mesh regression with dense correspondence. In CVPR, 2020.
[54] H. Zhang, J. Cao, G. Lu, W. Ouyang, and Z. Sun. Learning 3d human shape and pose from dense body parts. TPAMI, 2020.
[55] Z. Zhang, C. Wang. W. Qiu, W. Qin, and W. Zeng. Adafuse: Adaptive multiview fusion for accurate human pose estimation in the wild. IJCV, 2020.
[56] T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, and A. A. Efros. Learning dense correspondence via 3d-guided cycle consistency. In CVPR, 2016.
13


Reviews:
[{"review_id": "vAw2gZAt10", "reviewer": "Reviewer_xjUf", "summary": "This paper proposes a new method for the semi-supervised learning of keypoints using multiview images. The main challenge addressed is finding exact correspondences betweenkeypoints in multiple views, since the inverse of keypoint matching cannot be analytically derived or differentiated. The proposed probabilistic epipolar constraint encodes a soft correspondence and geometric consistency in the correspondence field. A distillation-based regularization is also proposed to avoid degenerative cases. Each contribution is ablated and show significant improvement over the baseline and relative to other methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper's contributions are:\n\n- A novel formulation of probabilistic epipolar constraints that can be used to enforce multiview consistency on continuous dense keypoints fields\n- A new neural network design that enable the precise measurement of of probabilistic epipolar error\n- A distillation-based regularization term\n- Strong performance on real-world multiview image data \n\nI believe this paper makes a convincing argument for all these claimed contributions, however some further clarification in some points would be useful for my final decision, as described below:\n\nThe idea of probabilistic epipolar constraints is well-motivated in terms of multiview keypoint matching, and is well positioned relative to other methods in the related work section. The motivation behind precise measurement of epipolar error, although not particularly novel, has many practical applications beyond improving performance, as a tool for uncertainty estimation. \n\nThe distillation-based regularization term addresses a practical limitation of the method, and multiple papers have already explored distillation / teacher networks to improve the performance of multi-frame networks in such degenerate cases. Although seems like the coefficient for this loss has to be carefully determined, so the network is allowed to deviate from it starting values, but not too much in the degenerate cases. Some analysis on the impact of varying this loss would have been interesting. Is there any sort of masking to determine in which areas this regularization should be applied, or is it applied equally?\n\nPerformance is evaluated on multiple large-scale datasets, with several baselines (including [22], the method that more closely relates to the setting the proposed method operates on), metrics, and qualitative results. The ablation table shows the impact of each component, however seems like L_T (photometric consistency loss) has very minor impact on the final performance. The same can be said for the generalization performance and comparison with other methods, improvement seems to be much higher for geometric and reconstruction accuracy than for keypoint accuracy. Some reasoning as to why that's the case would be interesting. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention scalable behavioral analysis as a potential societal impact of this work, since it will enable the study of animal life's individual and social behaviors by eliminating the need of annotated data for image processing. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "egQsz_Ue4V", "reviewer": "Reviewer_FiQR", "summary": "The paper proposes a semi-supervised approach for dense keypoints detection (assining a unique ID or a UV coordinate to an image of a human body). The key idea of the paper is an epipolar inconsistency loss function which can be computed and trained with multiview datasets. The approach has been evalauted on multi-view human datasets compared against single-view and multi-view techniques. Experimental results are OK, but not very great.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I was not very excited with the paper. I initially liked 2 ideas of the paper: 1) Using epipolar errors does not require extra labels. 2) The keypoint detection engine uses only a single image, and the model can be used for monocular sequences.\n\nHowever, I was not really convinced (before seeing results) that the first idea is a good one or not. The epipolar error seems very weak and I was in doubt if this really does anything on top of exact supervision. Since the paper is all about semi-supervised, this additional epipolar loss must add significant performance boost against a system without this epipolar loss. Then, I look at the experimantal results and found that the effects of the epipolar loss is not really verified as I suspected. The ablation study must show their system just without L_M in Table 1. This ablation study basically says that L_M does not matter.\n\nI do not really care about geometric consistency metric (geom consistency metric in table 1 and the entire Table 2), because this is the error which the proposed method seeks to minimize. The real performance should be mesure by the real pin-point answers. On this point, Table 1 (Comparison block) is not very exciting. Ignoring (geom consistency column), the proposed approach is the best in 3 out of 6 columns only. And even where they win, the performance gap is very minor at the point that I would rather suspect the engineering difference (likely authors spent more time tweaking their system on getting good numbers). Overall, the performance improvement does not seem to justify a bit complex technical formulation from not-so-convincing idea.\n\nLastly, on my second point above, the architecture does not mix 2 images which means that the system can be used for monocular sequences. I thought that this is the exciting piece and I wished to see evaluations on monocular sequences say optical flow datasets. The evaluations are solely based on mult-view datasets, which is understandable as generalization is often a problem and they have to train and test on the same dataset. But, a potentially interesting aspect of the paper diminished and I got less excited.\n\nOverall, I did not find this paper exciting. I started with a strong doubt that the main idea may not be very effective. And a bit complicated thing must happen to implement this idea, but overall, the metrics of the idea are not demonstrated and I beleive that there are no such benefits.\n\n--------------\nThe rebuttal provided new experimental validations and clarifications. My concenrs are resolved and I support the acceptance of the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper says that the negaive impact is minimum in onen sentence. It rather focuses on descriving postive aspects. This is not a problem to me at all. I do not see issues on negative impacts from this paper.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "XYM2JNnBazS", "reviewer": "Reviewer_fHWo", "summary": "The paper presents a method for learning continuous dense keypoint fields with part-supervised/labelled data and part-unlabelled data. The authors derive a probabilistic matchability constraint and define a loss that includes multi-view consistency, photometric consistency and a supervised loss from the labelled data. The method is evaluated quantitatively by comparing to a few other baselines and doing ablation studies on various loss terms and train/test splits. Geometric consistency and reconstruction accuracy are used as metrics to measure the accuracy of the proposed method and a few qualitative results are provided. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Main paper**\n\n1. Motivation: It might be good to elaborate a bit on what the applications of the proposed method would look like, in order to motivate the method soundly. Typically, keypoint matching whether sparse/dense and discete or continuous is used as a first step in learning the transformation between a pair of images. While there are several classical methods to solve this problem, there is also a rich body of learnt methods such as Spatial Transformer Network etc, that learn and predict the transformation directly. Could you provide example scenarios where having keypoint matches (i.e. the intermediate output) is necessary? \n\n2. Related work: What is the downside of applying sparse keypoint matching? Alternatively, what motivates the need for dense keypoint matching? There are many learned keypoint detection/matching methods in literature such as 3DFeatNet, R2D2, SKD, USIP. Would the authors be able to provide some clarity on where these methods fall short (and therefore what the USP of the proposed method is)? In summary, it would be helpful to motivate clearly why dense keypoint matching is needed and sparse matching is not sufficient? Sparse matches help recover an accurate transformation matrix, which gives dense mapping between images.\n\n3. Section 3.2 [Loss definition]: Loss equation could be written out more clearly. Why are some equations missing numbering? It's unclear whether $\\lambda_M$ and $\\lambda_R$ are inside the summation or pre-multipliers to the next term such as $\\mathcal{L}_M$ and $\\mathcal{L}_R$. Reading the description later in the paper clarifies this, but it would be good to sort out correct spacing or multi-line equation here to help resolve this.\n\n4. Is there a constraint on $\\lambda_*$ as weights. Looking at the supplementary information suggests that there is no constraint and these are multipliers of the implicit weight of $\\lambda_L = 1$ for supervised loss $\\mathcal{L}_L$. Please clarify why a constraint on the weights is not needed and provide some intuition or explanation for the _very_ different values of the relative weights for the various terms in the loss function. \n\n5. Section 3.2 [Loss definition]: I think multi-view consistency loss is not really working because it is naïve. The true match for a keypoint lies anywhere on the epipolar line if only two images are available. If three images and the fundamental matrix $F$ is available, then a point-point constraint can be derived and used. The equations and training details seem to suggest that only two images are being used. Can the authors confirm if two images or three images are used for multi-view consistency loss? The regularizing loss $\\mathcal{L}_R$ is formulated as 'not too far from the pretrained keypoint prediction'. Given that this is defined on labeled data with ground truth, it effectively imposes the multiview consistency constraint correctly and hence improves the results (as evidenced in ablation study on loss terms in Table 4). This may explain why results for $\\mathcal{L}_L + \\mathcal{L}_M$ are consistently much worse than those for $\\mathcal{L}_L$ alone and those for $\\mathcal{L}_L + \\mathcal{L}_M + \\mathcal{L}_R$. I suspect, $\\mathcal{L}_M$ can just be dropped and $\\mathcal{L}_R$ could be used instead. To this end, it would be interesting to see an ablation for $\\mathcal{L}_L + \\mathcal{L}_R$. Hence or otherwise, it would be good to see a decent justification for the inclusion of $\\mathcal{L}_M$ term in it's curent form. See also my comment regarding Supplementary info Section A.\n\n6. Photometric consistency loss: I wonder if results from an HSV or albedo related colour space insteadof RGB are better in any noticeable way? They may not be, if the network has sufficient capacity to learnt the RGB to HSV conversion implicitly. Allowing the colour components to change less than the lightness or brightness components should lead to better robustness for lighting changes. I suspect this is not an experiment worth doing unless dealing with datasets with extreme lighting changes. Since only a handful of examples of failure case are provided, I leave it to the authors to judge whether this is an area they think needs improvement. The goodness of the term $\\mathbb{T}(x,x')$ is entirely dependent on the goodness of the norm $||\\mathcal{I}(x)-\\mathcal{I'}(x')||$ and what representation is chosen for $\\mathcal{I}(x)$ in general. I do acknowledge that changing the representation of $\\mathcal{I}(x)$ will affect rest of the network too, so perhaps only the distance metric here should be updated instead. \n\n7. Evaluation: The method is trained on Human3.6M and then evaluated on SkiPose and OpenMonkeyStudio datasets. all three datasets have a reasonably long range or static background and a moving object in the foreground. It could be useful to see results on a different kind of dataset to disamiguate whether the proposed approach relies on the foreground/background setting in order to be sure that the background pixels are also contributing to the learning process. Another way to verify this is to plot saliency maps for network weights. If the method does capture the fact that the foreground moves against a static background, then that would be fine, but then the paper should specify what kinds of applications such a network is suited to. Ultimately, such analysis will help answer the question 'how general is the approach?'.  \n\n8. Evaluation: Figure 5: While it's good to see examples of dense keypoint matching results, it's unclear how to interpret the general goodness of the algorithm from the results in Figure 5. An alternative way to show the results could be to show the calculated transformation between the images and possibly a comparison against a pair of images where the ground truth transformation is available. If there are any patterns in the errors, they will likely become clear from these mappings. The continuous dense keypoint field could be sampled on a regular grid in the source image to acquire some grid points in order to enable comparison to classical multiview geometric methods. \n\n9. Results [Table 1]: I am confused about why 'Ours' does better on geodesic point similarity despite being worse on AUC10 compared to HMR. 'Ours' does do better on AUC30 compared to HMR, which suggests that 'Ours' has a flatter error profile but that does not correlate with high geodesic point similarity. It dependes somewhat on the value of $\\kappa$ used. Would the authors comment on scenarios where AUC30 is preferred over AUC10? It's a little bit counter intuitive that 'Ours' does better on geodesic point similarity (RCI is just anothermanifestation of GPS) and epipolar error, yet does not do well on AUC10. Evaluating something like AUC15 or AUC20 will help uncover if it's just that the crossover point between HMR and 'Ours' is just past 10cm. \n\n10. Line 142-143: \"A point that is not visible in a corresponding image does not have to be forced to match\" - A no match found is a valid outcome. This claim is misleading, because authors use this outcome themselves later in the paper. \n\n11. Line 144: I find the argument about geometric error being discrete and hence non-differentiable a bit arduous. The expected geometric error (i.e. the version using argmin) is discrete only by choice. It is perfectly possible to define continuous versions of the same quantity as indeed the authors do later in the paper. But of course, if one wants to select a single correspondence, then we end up with a discontinuity. I am not sure what purpose this point is serving. I suggest rewriting this. \n\n**Supplementary information:**\n\n1. Section A [Training details] mentions you chose $\\mathcal{L}_M = 1.0$, $\\mathcal{L}_R = 2000$, $\\mathcal{L}_T = 10$. \n    Did you mean the weights $\\lambda_M = 1.0, \\lambda_R = 2000, \\lambda_T = 10$ instead? \n    If yes, then perhaps this supports my suggestion for dropping $\\mathcal{L}_M$ term completely. $\\mathcal{L}_R$ is the term that is actually imposing the multi-view consistency constraint. I think that $\\mathcal{L}_T$ is too strict (or restricitve), and that's why you \n    (a) have accuracy issues where the photoconsistency is poor and \n    (b) only a small amount of correction is allowed in the training (i.e. $\\lambda_T << \\lambda_R$)\n    This also raises another question that the supervised loss $\\mathcal{L}_T$ (aka label loss) has a very small contribution to the total loss, and yet, we see from ablation studies that $\\mathcal{L}_T$ alone has reasonable performance.\n    \n2. Section A [Training details]: For reproducibility, it would be helpful to have a few more training details such as dataset splits, data augmentations if any, pre-processing of the data, cleaning, imputation and view selection for multi-view consistency loss where appropriate. \n    \n3. Section B [Failure cases] provides some explanations for failure cases. I find these puzzling. There must be enough corresponding points between the two views. If you output a probabilitistic matching score, then perhaps it is possible to set a threshold for accepting the matches. For various values of the acceptability threshold, you would be able to determine the fraction of 'accepted matches' vs 'total matches' and also determine the point beyond which the separation in the two views is so large that no acceptable matches can be found. Something like that may be a better measure of the quality of matching rather than saying that the network fails due to one of these three reasons. In other words it would be good to evaluate the failure modes of the dense keypoint matching network more systematically. The separation in views can be measured in terms of the angular difference in 3D. Although the proposed method is for continuous dense keypoints, the said continuous field can be sampled at discrete locations based on pixel locations of the source image and all evaluation metrics for sparse/discrete keypoint methods can then apply. \n\n4. Section C [Further results] provides further dense keypoint matching results in Figure 2. How should the figure be interpreted? Are the corresponding pairs arranged vertically or horizontally? Is the colour map same as that in the paper. A caption, even if it repeats text from the paper, would be helpful here. \n\n**Notes**\n\n1. Equation numbering: It would be good to have _all_ equations or math expressions numbered. The equations between Eq(1) and Eq(2) are not numbered? The loss definition in Section 3.2 is also not numbered. \n\n2. Typos: I think in quite a fewplaces where you refer to 'synchronized images', you actually mean 'corresponding images'. \n\n3. Typos: Line 289: 'We evaluation', Line 296: 'margain', Line 299: 'does not seen any', Line 300: 'still get improved'\n\n4. Typo? Line 251: Am I correct in understanding that authors mean to refer to DensePose-COCO as the first model in the comparison rather than initial model.\n\n**Overall assessment**\nThe paper presents a new method for continuous dense keypoint matching that leverages existing ideas to build a system with improved accuracy while using unlabelled images for training. The motivation for the work could be substantiated better. The method is evaluated quantitatively; improvement over existing methods is a little incremental, but noticeable nevertheless. A thorough analysis of what makes the system work or not work (limitations) and some comments about the statistical significance of the results would be welcome. There are a few fundamental issues in the development of the theory that are as yet not very well supported by the results or the reasoning presented in the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The datasets used are cited but the licenses of the datasets are not cited. A reader can look up the licences though. \nThe authors do not mention any negative societal impact, which is largely true because the proposed method will be used as a building block in larger systems. It is the larger systems or the applications that may have positive/negative societal impact. However, the limitations of the method could have been made clearer in the paper. A more thorough evaluation will naturally provide the evidence required to draw conclusions about the limitations of the method. Combinations of limitations of the building blocks used in a larger systems can come together to create blind spots in larger systems, so each block has a role to play. Being upfront about the limitations of each method definitely helps assess the impact of larger systems. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "SdUhkGdGTd", "reviewer": "Reviewer_Tdyv", "summary": "This paper proposes a novel method to learn dense keypoints in a semi-supervised manner using unlabeled multiview images. They propose to leverage the epipolar constraint of dense correspondences given a known fundamental matrix. In order to make epipolar constraint learnable, they propose a probabilistic epipolar constraint, incorporating uncertainty in correspondences using soft correspondences. The proposed method is proven to improve dense keypoint detectors in multiple datasets over reasonable baseline methods.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is reasonable, well written, and well studied. I cannot comment much about the originality of the work since this is out of my field of research, however the paper was easy to follow. Given that dense labeling of correspondence is not an easily affordable label, I have no doubt that self-supervised learning with known geometric constraints definitely helps.\n\nHere are some minor comments that may potentially help me understand this work better:\n\nIt would be helpful to ablate the effect of L_R. Although L_R is supposed to be a regularization, there may be some knowledge distillation from the pretrained model on unlabeled dataset that may potentially help improve the keypoint accuracy by itself. L_L + L_R would make the effectiveness of L_M obvious, especially since L_L + L_M does not outperform L_L (although I believe is reasonable).\n\nHow does the choice of $\\sigma$ in soft correspondence affect the performance of the model? Which value did you choose for $\\sigma$? How did you find the optimal $\\sigma$? Some ablation study on the effect of $\\sigma$on the performance would be helpful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper thoroughly study the limitation of the proposed method.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3hrs"}, "extra_text": {}}, {"review_id": "Q2v9zTuDytl", "reviewer": "Reviewer_Muxb", "summary": "This paper studies how to learn single image dense keypoint prediction by leveraging multiple-view constraints. Compared to prior works in this space, the key contribution is the modeling of the epipolar geometry constraint across different views.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an important problem of learning keypoint detectors by using multi-view supervision. Compared to prior works, the key contribution is the learning of dense keypoints and using epipolar constraints. \n\nThe epipolar constraint provides some supervision for the dense keypoint prediction. However, it is not a direct supervision as we do not know which point on the epipolar line is the corresponding point. This paper proposes a solution, which is similar to an attention mechanism, by matching points in some feature space. The proposed solution is plausible. \n\nThe network training incorporates other losses such as photometric multi-view consistency and regualrizations using a pre-trained model.\n\nThe experimental results show that the proposed approaches outperform state-of-the-art approaches on two benchmark datasets. \n\nThere are a couple of issues that prevent me from giving higher scores:\n1. It is not entirely clear how to address the partial similarity across the multiple views. The paper does mention that this is an issue. However, it is unclear how the definition of $P(u,u')$ and equations (2) and (3) address this. \n2. The epipolar constraints state that two lines of the two images are in correspondence, and correspondences are expected to be continuous. The current formulation in the paper only use such signals partially.\n3. It is unclear how to the canonical model is defined. Figure 2 shows that it may be a 2D surface parameterization of a 3D mesh. If so, why the phi map is not invertible? \n4. The current setup requires calibrated images with known intrinsics and extrinsics. What about if the estimations of those have errors and inaccurate. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not discussed explicitly, but the experimental results are analyzed in depth, including the pros and cons.", "review_text": "", "extra_scores": {"ethical_concerns": "NA.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "8iSu-N7BwB", "reviewer": "Reviewer_Ao3W", "summary": "The authors propose a technique that exploits multi-view consistency for learning dense correspondences in images coupled with a densepose-like mapping in texture space. They performed ablations and SOTA comparisons on H3.6, and further SOTA comparisons on Ski-pose and Monkey.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting technique that introduces multi-view consistency for dense pose estimation. What I would have liked to see is that multi-view consistency allows one to further increase the precision of computed correspondences, but what the authors discover is that by accounting for multi-view consistency... you get more consistent mappings (duh?) – in other words it is a paper that does well on a metric, because they introduced a metric for which other techniques would fail.\nSo overall, the emerging question is `why should we care?` and `can you show an application scenario where multi-view consistent keypoints are useful?`.\nOther than this core questions, I have other concerns about the paper itemized in the remarks below (to which I expect an answer). Note that I am *more than happy* to raise the score if my concerns are properly addressed (the opposite is also true).\n\n## Core remarks\n\n1. My core complaint relates to eq(5,7), and unless properly addressed would prevent me from letting this paper being accepted. First of all, I was left wondering why you needed both of them until I read L217-L224 (this should be addressed). But most importantly, I do not buy the argument provided by (6), and the massive drop in performance in the ablation in Table.1 (from 0.5 to 0.12).... even more so given that the supervised method performs much better (at 0.48)!!! In particular, (5) provides pointwise supervision to the value of the maps, leaving no affine nullspace in the supervision of $\\varphi$. To prove this, let us assume the ideal mapping function is an identity function $\\varphi(.)=I(.)$, then your statement would say $I(x)==2I(x)$ which is only true for $x=0$ (hence generally false)? So, while I think this argument is flawed, I can guess what happened in Table.1: the aggregated magnitude of (5) is significantly higher in value than the one in (7) (e.g. you forgot normalization?), and hence you are simply retrieving different pareto-optimal solutions (i.e. the magnitude of $\\mathcal{L}_R$ and $\\mathcal{L}_M$ are similar, while $\\mathcal{L}_L$ is much smaller... in other words $\\mathcal{L}_M$ is overpowering $\\mathcal{L}_L$, which is also consistent with the fact that the geometric consistency score is the lowest across the entire set of experiments).\n1. I find it quite disturbing that the paper continuously refers to \"geodesic\" distances and metric. More specifically, despite the fact that given a piecewise parameterization of geometry, distances between texture map coordinates $u$ *do not* represent geodesic distances. Even if you used small patches and quasi-isometric parameterization, this would only be a decent approximation in small neighborhoods (as isometric maps are only possible for developable surfaces with zero Gaussian curvature). The fact that [8] was sloppy in employing this term does not justify the entire field to adopt it too.\n1. L144: `argmin is not differentiable` this is only partially true, as otherwise any other paper that uses Chamfer Distance would not exist in geometric deep learning? Further, the *necessity* to convert from min to softmin has not been ablated at all... why?\n1. L158: It is not clear at all to me where the visibility $v$ comes from? Usually pairwise visibility is ensured by cycle consistency... but I see no such loss. Are you assuming visibility to be provided as part of the dataset for every single image? This is a severe limitation, and should be stated as such.\n1. Considering part of the paper is related to uncertainty for keypoints, I would consider appropriate an acknowledgement of works in this area (e.g. UGGLI amongst many others... Google it?).\n\n## Further remarks\n1. Sec.4: The writing in the technical section could be improved a lot... it would have been a lot faster to digest eq(2) directly, and saying the first term measures for the difference in how two points mapped in texture space, and the latter measures epipolar correctness, and then diving into details.\n1. Table.1: it would have been a lot more informative if the ablation would disable one loss at a time, rather than progressively adding them? Further, Why the scores in Ours(comparison) and Ours(ablation/last row) *do not match*?\n1. Fig.1 in print you can barely see anything, recommend increasing the image transparency, or increase brightness\n1. L57, L??: you state that you generalize the Samson distance (twice), yet never point out the difference\n1. L138: missing eq# and $\\Theta$ is undefined\n1. L149: what's the `range of the body surface?`\n1. L162-166: this seems repeated text with zero new information? (repeated earlier in the main body, as well as in Fig.2)\n1. L181, L192: I am not exactly sure what `when DensePose is used the field is defined for each body part` and `we augment the loss for body parts` mean. Are you intending the the function $\\Phi$ is actually a collection of mapping functions $\\{\\Phi_k\\}$, one per body part (indexed by `k`)?\n1. L203: if that's the cardinality of the foreground pixels, wouldn't that make your method choke in terms of memory on reasonably (e.g. 1MP) size images with $\\approx O(1e9^2)$ complexity?\n1. L207: `design a new operation` not sure I see anything new, isn't this just a vectorization of the same operation? Seems like an implementation detail more than anything else. Am I mistaken?\n1. L207-8: missing equation number!!!\n1. L187: infinitely → infinite\n1. Tab.1: accruacy → accuracy\n1. L173-5: if you use $\\underbrace{\\mathcal{L}}_{\\text{photometric loss}}$ you can omit this piece of text and save some space for more useful insight\n1. L131: consider explaining why this mapping is injective in your setting?\n1. L289: We evaluation → We evaluate\n\n# UPDATE raised my score 6→7 thanks to the rebuttal+discussion", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not sure people tracking cannot be seen has having some impact on society (monitoring, recognition, tracking), and not acknowledging this seems a bit naive.\n\nLimitations are in the supplementary, but a short discussion should be moved to the main paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that this is a solid paper that deserves acceptance. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version. 
Decision: Accept (Spotlight)