Dense Keypoints via Multiview Supervision

1 Introduction . The spatial arrangement of keypoints of dynamic organisms characterizes their complex pose , providing a computational representation of the way they behave . Recently , computer vision models offer fine grained behavioral modeling through dense keypoints that establish an injective mapping from the image coordinates to the continuous body surface of humans [ 8 ] and chimpanzees [ 36 ] . These models predict the continuous keypoint field from an image , supervised by a set of densely annotated keypoints , which shows remarkable performance on real-world imagery and brings out a number of applications including 3D mesh reconstruction [ 54 , 53 , 35 , 50 , 7 , 21 ] , texture/style transfer [ 26 , 37 ] , and geometry learning [ 14 , 1 ] . Nonetheless , attaining such densely annotated data is labor intensive , and more importantly , the quality of the annotations is fundamentally bounded by the visual ambiguity of keypoints , e.g. , points on textureless shirt . This visual ambiguity leads to a suboptimal model when applying it to out-of-sample distributions . In this paper , we present a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . new semi-supervised method to learn a dense keypoint detection model from the unlabeled multiview images via the epipolar constraint as shown in Figure 1 . Our main conjecture is that the dense keypoint model is optimal only if it is geometrically consistent across views . That is , every pair of corresponding keypoints , independently predicted by two views , must satisfy the epipolar constraint [ 9 ] . However , enforcing the epipolar constraint to learn a dense keypoint model is challenging because ( 1 ) the ground truth 3D model is unknown and thus the projections of the 3D model can not be used as the ground truth dense keypoints ; ( 2 ) the predicted dense keypoints are inaccurate and continuous over the body surface , and therefore , existing multiview supervision approaches [ 51 , 38 , 45 ] for sparse keypoints are not applicable . In these previous methods , the epipolar constraint was enforced between two keypoints ( or features ) of which semantic meaning was explicitly defined by a finite set of joints ( e.g. , elbow channel in a network ) ; and ( 3 ) establishing correspondences across views requires knowing an inverse mapping from the body surface to the image that can be neither analytically derived nor differentiable . These challenges limit the performance of previous work [ 22 ] that relies on iterative offline bootstrapping , which is not end-to-end trainable , or requires additional parameters to learn for 3D reconstruction1 . We tackle these challenges through a probabilistic epipolar constraint by incorporating an uncertainty in correspondences . This new constraint encodes the two desired properties . ( 1 ) Soft correspondence : given a keypoint in one image , we define matchability—the likelihood of correspondence for all predicted keypoints in another image based on the distance in the canonical body surface coordinate ( e.g. , texture coordinate ) . This allows evaluating geometric consistency in the form of a weighted average of epipolar errors over continuous body surface coordinates , eliminating the requirement of exact correspondences . ( 2 ) Geometric consistency : we generalize symmetric Sampson distance [ 9 ] for all possible pairs of keypoints from two views to enforce the epipolar constraint , collectively . With these properties , we derive a new differentiable multiview consistency measure that is labelagnostic , allowing us to utilize a large number of the unlabeled multiview images without explicit 3D reconstruction . We design an end-to-end trainable twin network architecture that takes a pair of images as an input and outputs geometrically consistent dense keypoint fields . This network design builds the affinity maps between two keypoint fields based on the matchability and epipolar errors , which facilitates measuring the probabilistic epipolar errors for all possible correspondences . In addition , inspired by knowledge distillation , we use a pretrained model to regularize network learning , which can prevent degenerate cases . Our method shows superior performance compared to existing methods , including non-differentiable bootstrapping [ 22 ] in terms of keypoint accuracy , multiview consistency , and 3D reconstruction accuracy . Our contributions include : ( 1 ) a novel formulation of probabilistic epipolar constraint that can be used to enforce multiview consistency on continuous dense keypoint fields in a differentiable way ; ( 2 ) a new design of the neural network that enable to precisely measure the probabilistic epipolar error , which allows utilizing a large number of the unlabeled multiview images ; ( 3 ) a distillation-based regularization to prevent degenerate model learning ; ( 4 ) strong performance on real-world multiview image data , including Human3.6M [ 11 ] , Ski-Pose [ 40 ] , and OpenMonkeyPose [ 3 ] , outperforming existing methods including non-differentiable dense keypoint learning [ 22 ] . Broader Impact Statement The ability to understand animals ’ individual and social behaviors is of central importance to multiple disciplines such as biology , neuroscience , and behavioral science . 1An analogous insight has been used for fundamental matrix , directly computed from correspondences that does not require additional variables for 3D reconstruction . Measuring their behaviors has been extremely challenge due to limited annotated data . This approach offers a way to address this challenge with a limited number of annotated data , which will lead to a scalable behavioral analysis . The negative societal impact of this work is minimum . 2 Related Work . Our framework aims at training a dense keypoint field estimation model via multiview supervision . We briefly review the related works . Dense Keypoint Field Estimation Finding dense correspondence fields between two images is a challenging problem in computer vision . 3D measurements ( e.g. , depth and pointcloud ) can provide a strong geometric cues which enable matching of deformable shapes [ 42 , 29 , 48 ] . Similarly , in 2D , visual and geometric cues have been used to find the dense correspondence fields in an unsupervised learning [ 56 , 6 , 4 , 44 , 43 ] . Notably , for special foreground targets such as humans , the dense matching problem can be cast as finding a dense keypoint field that maps pixel coordinates to a canonical body surface coordinates [ 24 ] ( e.g. , DensePose [ 8 ] ) . These works were built upon a large amount of data labeled by crowd-workers and generalized to learn the correspondence fields for face [ 2 ] and chimpanzees [ 36 ] . Key limitations of these approaches are the inaccuracy of labeling and requirement of large labeled data . We address these limitations by formulating multiview supervision that can enforce geometric consistency , which allows utilizing a large amount of unlabeled multiview images . Multiview Feature Learning Epipolar geometry can be used to learn a visual representation by transferring visual information from one image to another via epipolar lines or 3D reconstruction . For instance , a fusion layer can be learned to fuse feature maps across views [ 30 ] . Such fusion models can be factored into shape and camera components to reduce the number of learnable parameters and improve generalizability [ 49 ] . Given the camera calibration , a light fusion module can be learned to directly fuse deep features [ 10 ] or heatmaps [ 55 ] from other view along corresponding epipolar line . Several works combine multiview image features to form 3D features [ 13 , 46 ] or view-invariant feature in 2D [ 32 ] . Multiview Supervision Synchronized multiview images [ 11 , 15 , 52 ] possess a unique geometric property : images are visually similar yet geometrically distinctive , provided by stereo parallax . Such property offers a new opportunity to learn a geometrically coherent representation without labels . Bootstrapping by 3D reconstruction [ 22 ] can be used to learn a keypoint detector supervised by the projection of the 3D reconstruction to enforce cross-view consistency . MONET [ 51 ] enables an end-to-end learning by eliminating the necessity of 3D reconstruction and directly minimizing epipolar error . Learning keypoints can be combined with 3D pose estimation [ 34 , 12 ] by enforcing predicting the same pose in all views while using a few labeled examples with 3D or 2D pose annotations to prevent degeneration . One can alleviate the need for large amounts of annotations by matching the predicted 3D pose with the triangulated pose [ 19 ] . Several works further learn a latent representation encoding 3D geometry from images [ 5 , 25 ] or 2D pose [ 33 , 41 ] by enforcing consistent embedding , texture [ 28 ] , and view synthesis [ 45 ] across views . Unlike these approaches designed for sparse keypoints where the geometric consistency is applied on finite points , we study geometric consistency on continuous dense keypoint fields . Capture Dense [ 22 ] is the closest work to ours , which uses bootstrapping through 3D reconstruction of human mesh model [ 16 ] . However , due to the non-differentiable nature of bootstrapping , it is not end-to-end trainable . A temporal consistency has been also used for self-supervise the dense keypoint detector [ 27 ] . 3 Method . We present a novel method to learn a dense keypoint detector by using unlabeled multiview images . We formulate the epipolar constraint for continuous correspondence fields , which allows us to enforce geometric consistency between views . 3.1 Dense Epipolar Geometry . Given a pair of corresponding images from two different views , a pair of corresponding points x↔ x′ , are related by a fundamental matrix given the calibrated cameras , i.e . x̃′TFx̃ = 0 , where F ∈ R3×3 is the fundamental matrix , and x̃ ∈ P2 is a homogeneous representation of x . The measure of geometric consistency between the two images can be written as [ 9 ] : d ( x , x′ ; F ) = |x̃′TFx̃|√ ( Fx̃ ) 21 + ( Fx̃ ) 2 2 , x↔ x′ , ( 1 ) where ( Fx ) i is the ith entry of Fx . This measures epipolar error that is the Euclidean distance between x′ and the epipolar line Fx̃ . Consider an injective dense keypoint mapping φ : R2 → R2 that maps a pixel coordinate to a canonical 2D body surface coordinate , i.e. , u = φ ( x ; I ) where x ∈ Θ ( I ) is the set of the foreground pixels in the image I , and u ∈ R2 is the 2D coordinate in the body surface as shown in Figure 2 . This dense keypoint mapping can be learned from annotations , e.g. , DensePose [ 8 ] for humans where it maps to the texture coordinate of 3D human body surfaces . This mapping is incomplete because there exists missing correspondences in an image due to occlusion . One can find a correspondence between the two images through u , i.e. , φ−1 ( u ; I ) ↔ φ−1 ( u ; I ′ ) where I and I ′ the two view images . However , φ is an injective mapping where the analytic inverse does not exist in general . Given a point x in the image I , one can measure the expectation of geometric error by a nearest neighbor search on the body surface space : E ( x ) = d ( x , x′ ; F ) , x′ = argmin x′∈Θ ( I′ ) ‖φ ( x ; I ) − φ ( x′ ; I ′ ) ‖ , ( 2 ) where E ( x ) is the expectation of geometric error at x . The expectation of the geometric error measures the epipolar error over all possible matches in the other view image , i.e. , ∀x′ ∈ Θ ( I ′ ) . There are two limitations in the nearest neighbor search : ( 1 ) The correspondences are not exact , leading to a biased estimate of the geometric error expectation . ( 2 ) The argmin operation is not differentiable , which can not be used to learn a dense keypoint detector in an end-to-end fashion . Instead , we address these limitations by making use of a soft correspondence , or matchability—a likelihood of a point being matched to another point as shown in Figure 2 : M ( x , x′ ; φ , I , I ′ ) = P ( φ ( x ; I ) , φ ( x′ ; I ′ ) ; { φ ( y ; I ′ ) } y∈Θ ( I′ ) ) , ( 3 ) where P ( u , u′ ; Ω ) is the probability of matching between u and u′ : P ( u , u′ ; Ω ) = exp ( −‖u− u ′‖2 2σ2 ) / ∑ v∈Ω exp ( −‖u− v‖ 2 2σ2 ) . ( 4 ) Ω is the domain of the canonical coordinates , σ is the standard deviation that controls the smoothness of matching , e.g. , when σ → 0 , it approximates the nearest neighbor search . We use the matchability to form a probabilistic epipolar error : E ( x , x′ ) = M ( x , x′ ; φ , I , I ′ ) d ( x , x′ ; F ) , ( 5 ) where E ( x , x′ ) is the epipolar error between x and x ’ weighted by the matchability . The error expectation of x can be computed by marginalizing over all x′ : E ( x ) = ∑ x′∈Θ ( I′ ) E ( x , x′ ) . ( 6 ) That is , the expectation of the geometric error of x measures a weighted average of epipolar errors for all possible correspondences . The higher matchability , the more contribution to the error expectation . Unlike Equation ( 2 ) , Equation ( 6 ) does not include the non-differentiable argmin operation . Further , it takes into account all possible pairs of correspondences between two images , which eliminates the bias introduced by the nearest neighbor search . Given the error expectations over the dense keypoint field , we derive a symmetric dense epipolar error that measures geometric consistency between two images : E ( I , I ′ ) = 1 V ∑ x∈Θ ( I ) v ( x ; φ , I , I ′ ) E ( x ) + 1 V ′ ∑ x′∈Θ ( I′ ) v ( x′ ; φ , I , I ′ ) E ( x′ ) , ( 7 ) where v ( x ; φ , I , I ′ ) ∈ { 0 , 1 } is a visibility indicator : v ( x ; φ , I , I ′ ) = { 1 , ∃x′ ∈ Θ ( I ′ ) , ‖φ ( x ; I ) − φ ( x′ ; I ′ ) ‖ < 0 , otherwise . ( 8 ) V = ∑ x∈Θ ( I ) v ( x ; φ , I , I ′ ) and V ′ = ∑ x′∈Θ ( I′ ) v ( x ′ ; φ , I , I ′ ) are the numbers of foreground pixels visible in I ′ and I , respectively . is a correspondence tolerance defined in the canonical surface coordinate . In fact , this dense epipolar error is a generalization of Sampson distance [ 9 ] that defines epipolar error between explicit point correspondences ( point-to-point ) . We generalize it to model probabilistic correspondence between two set of points ( field-to-field ) . Existing approaches such as bootstrapping [ 22 ] establish the matching through 3D reconstruction of mesh and enforce the geometric error in an alternating fashion due to the non-differentiability of matching . Our differentiable formulation allows learning the dense keypoint detector in an end-to-end manner , which is flexible and shows superior performance .

Summary: The paper presents a method for learning continuous dense keypoint fields with part-supervised/labelled data and part-unlabelled data. The authors derive a probabilistic matchability constraint and define a loss that includes multi-view consistency, photometric consistency and a supervised loss from the labelled data. The method is evaluated quantitatively by comparing to a few other baselines and doing ablation studies on various loss terms and train/test splits. Geometric consistency and reconstruction accuracy are used as metrics to measure the accuracy of the proposed method and a few qualitative results are provided.