Semi-supervised Dense Keypoints using Unlabeled Multiview Images
Year: 2022

Abstract:
This paper presents a new end-to-end semi-supervised framework to learn a dense keypoint detector using unlabeled multiview images. A key challenge lies in finding the exact correspondences between the dense keypoints in multiple views since the inverse of the keypoint mapping can be neither analytically derived nor differentiated. This limits applying existing multiview supervision approaches used to learn sparse keypoints that rely on the exact correspondences. To address this challenge, we derive a new probabilistic epipolar constraint that encodes the two desired properties. (1) Soft correspondence: we define a matchability, which measures a likelihood of a point matching to the other image’s corresponding point, thus relaxing the requirement of the exact correspondences. (2) Geometric consistency: every point in the continuous correspondence fields must satisfy the multiview consistency collectively. We formulate a probabilistic epipolar constraint using a weighted average of epipolar errors through the matchability thereby generalizing the point-to-point geometric error to the field-to-field geometric error. This generalization facilitates learning a geometrically coherent dense keypoint detection model by utilizing a large number of unlabeled multiview images. Additionally, to prevent degenerative cases, we employ a distillation-based regularization by using a pretrained model. Finally, we design a new neural network architecture, made of twin networks, that effectively minimizes the probabilistic epipolar errors of all possible correspondences between two view images by building affinity matrices. Our method shows superior performance compared to existing methods, including non-differentiable bootstrapping in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.

heading: 1 Introduction
text: The spatial arrangement of keypoints of dynamic organisms characterizes their complex pose, providing a computational representation of the way they behave. Recently, computer vision models offer fine grained behavioral modeling through dense keypoints that establish an injective mapping from the image coordinates to the continuous body surface of humans [8] and chimpanzees [36]. These models predict the continuous keypoint field from an image, supervised by a set of densely annotated keypoints, which shows remarkable performance on real-world imagery and brings out a number of applications including 3D mesh reconstruction [54, 53, 35, 50, 7, 21], texture/style transfer [26, 37], and geometry learning [14, 1]. Nonetheless, attaining such densely annotated data is labor intensive, and more importantly, the quality of the annotations is fundamentally bounded by the visual ambiguity of keypoints, e.g., points on textureless shirt. This visual ambiguity leads to a suboptimal model when applying it to out-of-sample distributions. In this paper, we present a
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
new semi-supervised method to learn a dense keypoint detection model from the unlabeled multiview images via the epipolar constraint as shown in Figure 1.
Our main conjecture is that the dense keypoint model is optimal only if it is geometrically consistent across views. That is, every pair of corresponding keypoints, independently predicted by two views, must satisfy the epipolar constraint [9]. However, enforcing the epipolar constraint to learn a dense keypoint model is challenging because (1) the ground truth 3D model is unknown and thus the projections of the 3D model cannot be used as the ground truth dense keypoints; (2) the predicted dense keypoints are inaccurate and continuous over the body surface, and therefore, existing multiview supervision approaches [51, 38, 45] for sparse keypoints are not applicable. In these previous methods, the epipolar constraint was enforced between two keypoints (or features) of which semantic meaning was explicitly defined by a finite set of joints (e.g., elbow channel in a network); and (3) establishing correspondences across views requires knowing an inverse mapping from the body surface to the image that can be neither analytically derived nor differentiable. These challenges limit the performance of previous work [22] that relies on iterative offline bootstrapping, which is not end-to-end trainable, or requires additional parameters to learn for 3D reconstruction1.
We tackle these challenges through a probabilistic epipolar constraint by incorporating an uncertainty in correspondences. This new constraint encodes the two desired properties. (1) Soft correspondence: given a keypoint in one image, we define matchability—the likelihood of correspondence for all predicted keypoints in another image based on the distance in the canonical body surface coordinate (e.g., texture coordinate). This allows evaluating geometric consistency in the form of a weighted average of epipolar errors over continuous body surface coordinates, eliminating the requirement of exact correspondences. (2) Geometric consistency: we generalize symmetric Sampson distance [9] for all possible pairs of keypoints from two views to enforce the epipolar constraint, collectively. With these properties, we derive a new differentiable multiview consistency measure that is labelagnostic, allowing us to utilize a large number of the unlabeled multiview images without explicit 3D reconstruction.
We design an end-to-end trainable twin network architecture that takes a pair of images as an input and outputs geometrically consistent dense keypoint fields. This network design builds the affinity maps between two keypoint fields based on the matchability and epipolar errors, which facilitates measuring the probabilistic epipolar errors for all possible correspondences. In addition, inspired by knowledge distillation, we use a pretrained model to regularize network learning, which can prevent degenerate cases. Our method shows superior performance compared to existing methods, including non-differentiable bootstrapping [22] in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.
Our contributions include: (1) a novel formulation of probabilistic epipolar constraint that can be used to enforce multiview consistency on continuous dense keypoint fields in a differentiable way; (2) a new design of the neural network that enable to precisely measure the probabilistic epipolar error, which allows utilizing a large number of the unlabeled multiview images; (3) a distillation-based regularization to prevent degenerate model learning; (4) strong performance on real-world multiview image data, including Human3.6M [11], Ski-Pose [40], and OpenMonkeyPose [3], outperforming existing methods including non-differentiable dense keypoint learning [22].
Broader Impact Statement The ability to understand animals’ individual and social behaviors is of central importance to multiple disciplines such as biology, neuroscience, and behavioral science.
1An analogous insight has been used for fundamental matrix, directly computed from correspondences that does not require additional variables for 3D reconstruction.
Measuring their behaviors has been extremely challenge due to limited annotated data. This approach offers a way to address this challenge with a limited number of annotated data, which will lead to a scalable behavioral analysis. The negative societal impact of this work is minimum.

heading: 2 Related Work
text: Our framework aims at training a dense keypoint field estimation model via multiview supervision. We briefly review the related works.
Dense Keypoint Field Estimation Finding dense correspondence fields between two images is a challenging problem in computer vision. 3D measurements (e.g., depth and pointcloud) can provide a strong geometric cues which enable matching of deformable shapes [42, 29, 48]. Similarly, in 2D, visual and geometric cues have been used to find the dense correspondence fields in an unsupervised learning [56, 6, 4, 44, 43]. Notably, for special foreground targets such as humans, the dense matching problem can be cast as finding a dense keypoint field that maps pixel coordinates to a canonical body surface coordinates [24] (e.g., DensePose [8]). These works were built upon a large amount of data labeled by crowd-workers and generalized to learn the correspondence fields for face [2] and chimpanzees [36]. Key limitations of these approaches are the inaccuracy of labeling and requirement of large labeled data. We address these limitations by formulating multiview supervision that can enforce geometric consistency, which allows utilizing a large amount of unlabeled multiview images.
Multiview Feature Learning Epipolar geometry can be used to learn a visual representation by transferring visual information from one image to another via epipolar lines or 3D reconstruction. For instance, a fusion layer can be learned to fuse feature maps across views [30]. Such fusion models can be factored into shape and camera components to reduce the number of learnable parameters and improve generalizability [49]. Given the camera calibration, a light fusion module can be learned to directly fuse deep features [10] or heatmaps [55] from other view along corresponding epipolar line. Several works combine multiview image features to form 3D features [13, 46] or view-invariant feature in 2D [32].
Multiview Supervision Synchronized multiview images [11, 15, 52] possess a unique geometric property: images are visually similar yet geometrically distinctive, provided by stereo parallax. Such property offers a new opportunity to learn a geometrically coherent representation without labels. Bootstrapping by 3D reconstruction [22] can be used to learn a keypoint detector supervised by the projection of the 3D reconstruction to enforce cross-view consistency. MONET [51] enables an end-to-end learning by eliminating the necessity of 3D reconstruction and directly minimizing epipolar error. Learning keypoints can be combined with 3D pose estimation [34, 12] by enforcing predicting the same pose in all views while using a few labeled examples with 3D or 2D pose annotations to prevent degeneration. One can alleviate the need for large amounts of annotations by matching the predicted 3D pose with the triangulated pose [19]. Several works further learn a latent representation encoding 3D geometry from images [5, 25] or 2D pose [33, 41] by enforcing consistent embedding, texture [28], and view synthesis [45] across views. Unlike these approaches designed for sparse keypoints where the geometric consistency is applied on finite points, we study geometric consistency on continuous dense keypoint fields. Capture Dense [22] is the closest work to ours, which uses bootstrapping through 3D reconstruction of human mesh model [16]. However, due to the non-differentiable nature of bootstrapping, it is not end-to-end trainable. A temporal consistency has been also used for self-supervise the dense keypoint detector [27].

heading: 3 Method
text: We present a novel method to learn a dense keypoint detector by using unlabeled multiview images. We formulate the epipolar constraint for continuous correspondence fields, which allows us to enforce geometric consistency between views.

heading: 3.1 Dense Epipolar Geometry
text: Given a pair of corresponding images from two different views, a pair of corresponding points x↔ x′, are related by a fundamental matrix given the calibrated cameras, i.e. x̃′TFx̃ = 0, where F ∈ R3×3 is the fundamental matrix, and x̃ ∈ P2 is a homogeneous representation of x. The
measure of geometric consistency between the two images can be written as [9]:
d(x,x′;F) = |x̃′TFx̃|√
(Fx̃)21 + (Fx̃) 2 2
, x↔ x′, (1)
where (Fx)i is the ith entry of Fx. This measures epipolar error that is the Euclidean distance between x′ and the epipolar line Fx̃.
Consider an injective dense keypoint mapping φ : R2 → R2 that maps a pixel coordinate to a canonical 2D body surface coordinate, i.e., u = φ(x; I) where x ∈ Θ(I) is the set of the foreground pixels in the image I, and u ∈ R2 is the 2D coordinate in the body surface as shown in Figure 2. This dense keypoint mapping can be learned from annotations, e.g., DensePose [8] for humans where it maps to the texture coordinate of 3D human body surfaces. This mapping is incomplete because there exists missing correspondences in an image due to occlusion. One can find a correspondence between the two images through u, i.e., φ−1(u; I) ↔ φ−1(u; I ′) where I and I ′ the two view images. However, φ is an injective mapping where the analytic inverse does not exist in general.
Given a point x in the image I, one can measure the expectation of geometric error by a nearest neighbor search on the body surface space:
E(x) = d(x,x′;F), x′ = argmin x′∈Θ(I′) ‖φ(x; I)− φ(x′; I ′)‖, (2)
where E(x) is the expectation of geometric error at x. The expectation of the geometric error measures the epipolar error over all possible matches in the other view image, i.e., ∀x′ ∈ Θ(I ′). There are two limitations in the nearest neighbor search: (1) The correspondences are not exact, leading to a biased estimate of the geometric error expectation. (2) The argmin operation is not differentiable, which cannot be used to learn a dense keypoint detector in an end-to-end fashion.
Instead, we address these limitations by making use of a soft correspondence, or matchability—a likelihood of a point being matched to another point as shown in Figure 2:
M(x,x′;φ, I, I ′) = P ( φ(x; I), φ(x′; I ′); {φ(y; I ′)}y∈Θ(I′) ) , (3)
where P (u,u′; Ω) is the probability of matching between u and u′:
P (u,u′; Ω) = exp ( −‖u− u ′‖2
2σ2
) / ∑ v∈Ω exp ( −‖u− v‖ 2 2σ2 ) . (4)
Ω is the domain of the canonical coordinates, σ is the standard deviation that controls the smoothness of matching, e.g., when σ → 0, it approximates the nearest neighbor search. We use the matchability to form a probabilistic epipolar error:
E(x,x′) = M(x,x′;φ, I, I ′)d(x,x′;F), (5)
where E(x,x′) is the epipolar error between x and x’ weighted by the matchability.
The error expectation of x can be computed by marginalizing over all x′: E(x) = ∑
x′∈Θ(I′)
E(x,x′). (6)
That is, the expectation of the geometric error of x measures a weighted average of epipolar errors for all possible correspondences. The higher matchability, the more contribution to the error expectation. Unlike Equation (2), Equation (6) does not include the non-differentiable argmin operation. Further, it takes into account all possible pairs of correspondences between two images, which eliminates the bias introduced by the nearest neighbor search.
Given the error expectations over the dense keypoint field, we derive a symmetric dense epipolar error that measures geometric consistency between two images:
E(I, I ′) = 1 V ∑ x∈Θ(I) v(x;φ, I, I ′)E(x) + 1 V ′ ∑ x′∈Θ(I′) v(x′;φ, I, I ′)E(x′), (7)
where v(x;φ, I, I ′) ∈ {0, 1} is a visibility indicator: v(x;φ, I, I ′) = {
1, ∃x′ ∈ Θ(I ′), ‖φ(x; I)− φ(x′; I ′)‖ < 0, otherwise.
(8)
V = ∑ x∈Θ(I) v(x;φ, I, I ′) and V ′ = ∑ x′∈Θ(I′) v(x ′;φ, I, I ′) are the numbers of foreground pixels visible in I ′ and I, respectively. is a correspondence tolerance defined in the canonical surface coordinate.
In fact, this dense epipolar error is a generalization of Sampson distance [9] that defines epipolar error between explicit point correspondences (point-to-point). We generalize it to model probabilistic correspondence between two set of points (field-to-field).
Existing approaches such as bootstrapping [22] establish the matching through 3D reconstruction of mesh and enforce the geometric error in an alternating fashion due to the non-differentiability of matching. Our differentiable formulation allows learning the dense keypoint detector in an end-to-end manner, which is flexible and shows superior performance.

heading: 3.2 Multiview Semi-supervised Learning
text: We learn the dense keypoint detector φ by minimizing the following error:
L = λLLL + λMLM + λRLR + λTLT, (9) where LL is the labeled data loss, LM is the multiview geometric consistency loss, LR is the regularization loss, and LT is the multiview photometric consistency loss. λL, λM, λR and λT are the weights to control their relative importance.
Supervised Loss We learn the dense keypoint detector from the labeled dataset DL by minimizing the following error:
LL = ∑
{I,U}∈DL ∑ x∈Θ(I) ‖Ux − φ(x; I)‖1 (10)
where U ∈ R2×H×W is the ground truth canonical surface coordinates for dense keypoints, Ux is the ground truth canonical surface coordinate at x.
Multiview Geometric Consistency Loss We learn the dense keypoint detector from the unlabeled multiview dataset DU by minimizing dense epipolar error over image pairs:
LM = ∑
{I,I′}∈DU
E(I, I ′), (11)
where E(I, I ′) is the dense epipolar error between two corresponding images I and I ′, defined in Equation (14). This loss progressively minimizes the dense epipolar error between image pair as learning the dense keypoint detection model as shown in Figure 3.
Distillation Based Regularization Loss Enforcing multiview consistency alone can lead to degenerate cases. For instance, consider a linear transformation in the body surface, e.g., v = Tu where T ∈ R2×2 is a similarity transformation. Any φ that satisfies the following condition can be equivalent dense keypoint detector:
φ(x; I) ≡ Tφ(x; I). (12) This indicates that there exist an infinite number of dense keypoint detectors that satisfy the epipolar geometry. To alleviate this geometric ambiguity, we use a distillation-based regularization using a pretrained model. Let φ0 be a dense keypoint detector pretrained by the labeled data. We prevent the learned detector φ from deviating too much from the pretrained detector φ0 for the unlabeled data:
LR = ∑ I∈DU ∑ x∈Θ(I) ‖φ0(x; I)− φ(x; I)‖1, (13)
where LR is the loss for the distillation-based regularization, minimizing the difference from the pretrained model.
Multiview Photometric Consistency Loss We leverage photometric consistency across views. Assuming ambient light, pixels across views corresponding to the same 3D point in space should have the same RGB value. Similar to dense epipolar error E(I, I ′), we define dense photometric error T(I, I ′) as:
T(I, I ′) = 1 V ∑ x∈Θ(I) v(x;φ, I, I ′)T(x) + 1 V ′ ∑ x′∈Θ(I′) v(x′;φ, I, I ′)T(x′), (14)
where T(x) is the expectation of photometric error of x similar to E(x), i.e. T(x) =∑ x′∈Θ(I′)T(x,x ′) where T(x,x′) = M(x,x′;φ, I, I ′)‖I(x)− I ′(x′)‖2.
Then we compute multiview photometric consistency loss as: LT = ∑
{I,I′}∈DU
T(I, I ′), (15)

heading: 3.3 Network Design
text: We design a new network architecture composed of twin networks to learn the dense keypoint detector by enforcing multiview consistency over dense keypoint fields as shown in Figure 4. Each network is made of a fully convolutional network that outputs the dense keypoint field per body part.
Given two dense keypoint fields, we compute the probabilistic epipolar error by constructing two affinity matrices: matchability matrix and epipolar matrix, M,E ∈ R|Θ(I)|×|Θ(I′)| where |Θ(I)| is the cardinality of the range of foreground pixels. These two matrices are defined by:
Mij = M(xi,xj ;φ, I, I ′), Eij = d(xi,xj ;F) (16) where Mij is the i, j entry of the matrix M. xi and xj are the ith and jth foreground pixels from the images I and I ′, respectively. We also compute the visibility maps Vi = v(x;φ, I, I ′) and V′i = v(x
′;φ, I, I ′). We design a new operation to measure the dense epipolar error in Equation (14):
E(I, I ′) = 1 V VT(M E)1|Θ(I′)| + 1 V ′ V′ T (M E)T1|Θ(I)|, (17)
where 1n is the n-dimensional vector of which entries are all one. is the element-wise multiplication of matrices. Note that dense epipolar error T(I, I ′) can be computed following the same operations.

heading: 4 Experiments
text: We perform experiments on human and monkey targets as two example applications to evaluate the effectiveness of our proposed semi-supervised learning pipeline.

heading: 4.1 Implementation Details
text: We use HRNet [18] as the backbone network followed by four head networks made up of convolutional layers to predict foreground mask, body part index, and UV coordinates on the canonical body surface, respectively. Each network takes as an input a 224×224 image and outputs 15-channel (for foreground mask head only) or 25-channel 56×56 feature maps [8]. We train the network in two stages. In the first stage, we train an initial model using the labeled data by the labeled data loss LL. Specifically, for the human dense keypoints, we use 48K human instances in DensePose-COCO [8] training set to train the initial model. For the monkey dense keypoints, we directly use the pretrained model for chimpanzees [36] as the initial model since we do not have an access to the labeled data. In the second stage, we train our model on unlabeled multiview data leveraging all losses. The initial model is used for two purposes: (1) the pre-trained model φ0 with weights fixed for distillation-based regularization and (2) the initial model to be refined via multiview supervision.

heading: 4.2 Evaluation Datasets
text: Human3.6M [11] is a large-scale indoor multiview dataset captured by 4 cameras for 3D human pose estimation. It contains 3.6 millions of images captured from 7 subjects performing 15 different daily activities, e.g. Walking, Greeting, and Discussion. Following common protocols, we use subject S1, S5, S6, S7 and S8 for training, and reserve subject S9 and S11 for testing. Following [54] and [53], we leverage SMPL [24] parameters generated by HMR [17] via applying MoSh [23] to the sparse 3D MoCap marker data to recover ground truth 3D human meshes. We further perform Procrustes analysis [39] to align them with ground truth 3D poses in global coordinate system and then render ground truth IUV maps using PyTorch3D [31]. Since it is the only multiview human dataset that we have access to ground truth 3D mesh / IUV maps, we use this dataset to perform comprehensive experiment and studies.
3DPW [47] is an in-the-wild dataset made of single view images with accurate 3D human pose and shape annotations. We use the test split of this dataset for dense keypoint accuracy evaluations (3D metrics do not apply). Ground truth IUV maps are rendered from 3D pose and shape annotations by leveraging the provided camera parameters.
Ski-Pose PTZ-Camera Dataset [40] is a multiview dataset capturing competitive skiers performing giant slalom runs. 6 synchronized and calibrated pant-tile-zoom-cameras (PTZ) cameras are used to track a single skier at a time. The global locations of the cameras were measured by a tachymeter
theodolite. It contains 8.5K training images and 1.7K testing images. We use this dataset to evaluate generalization towards in-the-wild multiview settings. We use its standard train/test split to train and evaluation our model. We select 6 adjacent view pairs to form training samples.
OpenMonkeyPose [3] is a large landmark dataset of rhesus macaques captured by 62 synchronized multiview cameras. It consists of nearly 200K labeled images with four macaque subjects that freely move in a large cage while performing foraging tasks. Each monkey instance is annotated with 13 2D and 3D joints. We use this dataset to show our model’s ability on transferring dense keypoints to monkey data. We split about 64K images for training and 12K images for testing. For training, we generate the densepose of monkey data using a pretrained model [36] as pseudo-labels. These pseudo-labels are then used for refining the pretrained model.

heading: 4.3 Baselines
text: In our experiments, we consider four baselines: (1) A model fully-supervised by labeled data, i.e. DensePose-COCO [8], which is also our initial model for semi-supervised learning. We refer to this as Supervised in the following. (2) A model trained using multiview bootstrap strategy [38, 22], where multiview triangulation results from previous stage are used as the pseudo ground truth. (3) A learnable 3D mesh estimation method where dense keypoint estimation can be acquired by reprojecting 3D mesh to image domain, e.g. HMR [17]. (4) A 3D mesh estimation framework similar to HMR but additionally incorporating Model-fitting in the Loop, e.g. SPIN [20]. Supervised is also used as an initial model for our approach.

heading: 4.4 Metrics
text: We evaluate the performance of our dense keypoint model using metrics from three aspects: (1) geometric consistency, (2) accuracy of dense keypoints, and (3) accuracy of 3D reconstruction from multiple views.
Geometric Consistency We use epipolar distance (unit: pixel) averaged over views and frames as the metric for evaluating multiview geometric consistency. Ideally, two dense keypoints corresponding to the same point on 3D surface should have epipolar distance equal to 0. This metric can be evaluated on any multiview dataset with ground truth camera parameters available.
Dense Keypoint Accuracy We evaluate the model’s performance on dense keypoint accuracy from two aspects: (1) Ratio of Correct Point (RCP) and (2) Ratio of Correct Instances (RCI). RCP evaluates correspondence accuracy over the whole image domain. Specifically, it records the ratio of foreground pixels on images with corresponding 3D body surface correctly predicted as a function of geodesic distance threshold, where the prediction is considered correct if its geodesic distance to the ground truth is below the threshold (10cm and 30cm). RCIs consider instance-wise accuracy where an instance is declared to be correct if its geodesic point similarity (GPS) [8] is above the threshold. We also report the mean RCI (mRCI) and the mean GPS for all instances (mGPS).
Reconstruction Accuracy Given dense keypoints, we measure 3D reconstruction error by triangulating them in 3D. We compute Mean Per mesh Vertex Position Error (MPVPE) as the metric for reconstruction accuracy, which is defined as the mean euclidean distance between triangulated vertices and corresponding ground truth ones. In addition, inspired by geodesic point similarity [8], we define vertex similarity as: VS = 1|V | ∑ vi∈V exp ( −−d(v̂i−vi) 2 2κ2 ) , where d(v̂i − vi) is the euclidean distance between triangulated vertex v̂i and corresponding ground truth one vi, and V is the set of visible ground truth vertices from both views. κ is a normalizing parameter. For vi that does not correspond to the triangulated vertex, it is set to infinity. Further, to account for false positives in triangulated vertices (vertices not visible from both views), we define masked vertex similarity (MVS) as MVS = √ VS · I , where I is the intersection over union between the set of triangulated vertices and V . We report mean MVS (mMVS) over all instances.

heading: 4.5 Evaluation on Human3.6M Dataset
text: We use Human3.6M dataset to perform (1) comprehensive cross-method evaluations, (2) study on model’s generalizability towards new views, and (3) ablation study on losses used for training. Results are summarized in Table 1 with all metrics reported.
Cross-method Evaluation We evaluate the performance of our model against other methods as shown in the comparison block in Table 1 and show qualitative results (1st column in Figure 5). For the metrics of keypoint accuracy and geometry consistency, ours outperforms all other methods
(except for AUC10 only second to HMR [17]). Note that HMR and SPIN are trained with the 3D ground truth (pre-estimated mesh) of Human3.6M while other algorithms including ours predict dense keypoints without the 3D ground truth. Having the 3D ground truth is a significant advantage while it requires a substantial additional amount of 3D annotation effort. It is expected to outperform the ones without it, in particular, on reconstruction accuracy metrics. We included these baselines to provide the upper bound of our semi-supervised method as a reference. Note that despite the lack of the 3D ground truth, ours still outperforms HMR and SPIN by a margin of up to 26.2% and 67.8% on keypoint accuracy metrics on three keypoint accuracy metrics. Compared to “Supervised” and “Bootstrapping” which are close to our setup, ours outperform on all metrics by a margin of up to 9.2% and 7.0% on keypoint accuracy metrics.
Generalizability towards new views We evaluate generalization by testing on different views: two views are used for training and other two views are used for testing. The results are summarized in the generalization block in Table 1. As can be seen, although a model trained purely on one camera pair does not use any sample captured by the other pair, its performance still improves on all metrics on top of Supervised model by a margin of 0.6%-12.8% on keypoint accuracy, 38.3% - 47.9% on geometric consistency, and 6.5% - 13.7% on reconstruction accuracy. This shows that model trained by our semi-supervised approach can be generalized to new views.
Ablation Study We conduct an ablation study to evaluate the impact of each loss. The results are reported in the ablation block in Table 1. Note that we propose semi-supervised learning thus LL is always used. The inferior performance of model trained with LL + LM or LL + LT or LL + LM + LT proves that refining initial model by multiview loss alone suffers from degeneration (Section 3.2). This limitation can be addressed by adding the regularization LR. Note that regularization itself does not add any value: it is identical to the supervised model in theory. This is empirically verified since LL + LR performs similarly to LL. Both multiview consistency losses show effectiveness: LL + LR + LM and LL + LR + LT outperform LL + LR on all metrics by a margin up to 6.5% / 2.7% for keypoint accuracy metrics and 8.7% / 9.8% for reconstruction accuracy metrics respectively. LL + LR + LM + LT further achieves better results on all metrics. Multiview geometric consistency loss mainly contributes to the improvement on keypoint accuracy and geometric consistency metrics, while multiview photometric consistency loss mainly contributes to reconstruction accruacy metrics.
4.6 Evalution on 3DPW Dataset Keypoint accuracy
To further validate the usefulness of our method in terms of dense keypoint accuracy, in addition to detection accuracy on multiview image data, we conduct another cross-method evaluation on 3DPW dataset (single-view dense keypoint detection) on the keypoint accuracy met-
rics, i.e., training on DensePose-COCO with Human3.6M multiview supervision and testing on 3DPW without an adaptation, as shown in Table 2. The results show that our trained model with geometric consistency is more generalizable than the baselines.

heading: 4.7 Evalution on Ski-Pose PTZ-Camera Dataset and OpenMonkeyPose Dataset
text: We evaluate our method on multiview in-the-wild datasets: Ski-pose and OpenMonkeyPose. Since no ground truth is available, we evaluate geometric consistency summarized in Table 3 and show qualitative results (2nd and 3rd columns in Figure 5). The results show that our model outperforms other baselines by a large margin: 37.2%-54.1% on Ski-Pose and 31.9%-47.5% on OpenMonkeyPose, which can be visually identified by qualitative results.

heading: 5 Conclusion
text: We present a novel end-to-end semi-supervised approach to learn a dense keypoint detector by leverage a large amount of unlabeled multiview images. Due to the nature of continuous keypoint representation, finding exact correspondences between views is challenging unlike sparse keypoints. We address this challenge by formulating a new dense epipolar constraint that allows measuring a field-to-field geometric error without knowing exact correspondences. Additionally, we proposed a distillation-based regularization to prevent degenerated cases. We design a new network architecture made of twin networks that can effectively measure the dense epipolar error by considering all possible correspondences using affinity matrices. We show that our method outperforms the baseline approaches in keypoint accuracy, multiview consistency, and reconstruction accuracy.

heading: Acknowledgement
text: This project is partially supported by NSF IIS 1846031 and NSF IIS 2022894.

Review summaries:
The paper presents a method for learning continuous dense keypoint fields with part-supervised/labelled data and part-unlabelled data. The authors derive a probabilistic matchability constraint and define a loss that includes multi-view consistency, photometric consistency and a supervised loss from the labelled data. The method is evaluated quantitatively by comparing to a few other baselines and doing ablation studies on various loss terms and train/test splits. Geometric consistency and reconstruction accuracy are used as metrics to measure the accuracy of the proposed method and a few qualitative results are provided. 

The paper proposes a semi-supervised approach for dense keypoints detection (assining a unique ID or a UV coordinate to an image of a human body). The key idea of the paper is an epipolar inconsistency loss function which can be computed and trained with multiview datasets. The approach has been evalauted on multi-view human datasets compared against single-view and multi-view techniques. Experimental results are OK, but not very great.

This paper proposes a novel method to learn dense keypoints in a semi-supervised manner using unlabeled multiview images. They propose to leverage the epipolar constraint of dense correspondences given a known fundamental matrix. In order to make epipolar constraint learnable, they propose a probabilistic epipolar constraint, incorporating uncertainty in correspondences using soft correspondences. The proposed method is proven to improve dense keypoint detectors in multiple datasets over reasonable baseline methods. 

This paper proposes a new method for the semi-supervised learning of keypoints using multiview images. The main challenge addressed is finding exact correspondences betweenkeypoints in multiple views, since the inverse of keypoint matching cannot be analytically derived or differentiated. The proposed probabilistic epipolar constraint encodes a soft correspondence and geometric consistency in the correspondence field. A distillation-based regularization is also proposed to avoid degenerative cases. Each contribution is ablated and show significant improvement over the baseline and relative to other methods.