Dense Keypoints via Multiview Supervision

1 Introduction . The spatial arrangement of keypoints of dynamic organisms characterizes their complex pose , providing a computational representation of the way they behave . Recently , computer vision models offer fine grained behavioral modeling through dense keypoints that establish an injective mapping from the image coordinates to the continuous body surface of humans [ 8 ] and chimpanzees [ 36 ] . These models predict the continuous keypoint field from an image , supervised by a set of densely annotated keypoints , which shows remarkable performance on real-world imagery and brings out a number of applications including 3D mesh reconstruction [ 54 , 53 , 35 , 50 , 7 , 21 ] , texture/style transfer [ 26 , 37 ] , and geometry learning [ 14 , 1 ] . Nonetheless , attaining such densely annotated data is labor intensive , and more importantly , the quality of the annotations is fundamentally bounded by the visual ambiguity of keypoints , e.g. , points on textureless shirt . This visual ambiguity leads to a suboptimal model when applying it to out-of-sample distributions . In this paper , we present a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . new semi-supervised method to learn a dense keypoint detection model from the unlabeled multiview images via the epipolar constraint as shown in Figure 1 . Our main conjecture is that the dense keypoint model is optimal only if it is geometrically consistent across views . That is , every pair of corresponding keypoints , independently predicted by two views , must satisfy the epipolar constraint [ 9 ] . However , enforcing the epipolar constraint to learn a dense keypoint model is challenging because ( 1 ) the ground truth 3D model is unknown and thus the projections of the 3D model can not be used as the ground truth dense keypoints ; ( 2 ) the predicted dense keypoints are inaccurate and continuous over the body surface , and therefore , existing multiview supervision approaches [ 51 , 38 , 45 ] for sparse keypoints are not applicable . In these previous methods , the epipolar constraint was enforced between two keypoints ( or features ) of which semantic meaning was explicitly defined by a finite set of joints ( e.g. , elbow channel in a network ) ; and ( 3 ) establishing correspondences across views requires knowing an inverse mapping from the body surface to the image that can be neither analytically derived nor differentiable . These challenges limit the performance of previous work [ 22 ] that relies on iterative offline bootstrapping , which is not end-to-end trainable , or requires additional parameters to learn for 3D reconstruction1 . We tackle these challenges through a probabilistic epipolar constraint by incorporating an uncertainty in correspondences . This new constraint encodes the two desired properties . ( 1 ) Soft correspondence : given a keypoint in one image , we define matchability—the likelihood of correspondence for all predicted keypoints in another image based on the distance in the canonical body surface coordinate ( e.g. , texture coordinate ) . This allows evaluating geometric consistency in the form of a weighted average of epipolar errors over continuous body surface coordinates , eliminating the requirement of exact correspondences . ( 2 ) Geometric consistency : we generalize symmetric Sampson distance [ 9 ] for all possible pairs of keypoints from two views to enforce the epipolar constraint , collectively . With these properties , we derive a new differentiable multiview consistency measure that is labelagnostic , allowing us to utilize a large number of the unlabeled multiview images without explicit 3D reconstruction . We design an end-to-end trainable twin network architecture that takes a pair of images as an input and outputs geometrically consistent dense keypoint fields . This network design builds the affinity maps between two keypoint fields based on the matchability and epipolar errors , which facilitates measuring the probabilistic epipolar errors for all possible correspondences . In addition , inspired by knowledge distillation , we use a pretrained model to regularize network learning , which can prevent degenerate cases . Our method shows superior performance compared to existing methods , including non-differentiable bootstrapping [ 22 ] in terms of keypoint accuracy , multiview consistency , and 3D reconstruction accuracy . Our contributions include : ( 1 ) a novel formulation of probabilistic epipolar constraint that can be used to enforce multiview consistency on continuous dense keypoint fields in a differentiable way ; ( 2 ) a new design of the neural network that enable to precisely measure the probabilistic epipolar error , which allows utilizing a large number of the unlabeled multiview images ; ( 3 ) a distillation-based regularization to prevent degenerate model learning ; ( 4 ) strong performance on real-world multiview image data , including Human3.6M [ 11 ] , Ski-Pose [ 40 ] , and OpenMonkeyPose [ 3 ] , outperforming existing methods including non-differentiable dense keypoint learning [ 22 ] . Broader Impact Statement The ability to understand animals ’ individual and social behaviors is of central importance to multiple disciplines such as biology , neuroscience , and behavioral science . 1An analogous insight has been used for fundamental matrix , directly computed from correspondences that does not require additional variables for 3D reconstruction . Measuring their behaviors has been extremely challenge due to limited annotated data . This approach offers a way to address this challenge with a limited number of annotated data , which will lead to a scalable behavioral analysis . The negative societal impact of this work is minimum . 2 Related Work . Our framework aims at training a dense keypoint field estimation model via multiview supervision . We briefly review the related works . Dense Keypoint Field Estimation Finding dense correspondence fields between two images is a challenging problem in computer vision . 3D measurements ( e.g. , depth and pointcloud ) can provide a strong geometric cues which enable matching of deformable shapes [ 42 , 29 , 48 ] . Similarly , in 2D , visual and geometric cues have been used to find the dense correspondence fields in an unsupervised learning [ 56 , 6 , 4 , 44 , 43 ] . Notably , for special foreground targets such as humans , the dense matching problem can be cast as finding a dense keypoint field that maps pixel coordinates to a canonical body surface coordinates [ 24 ] ( e.g. , DensePose [ 8 ] ) . These works were built upon a large amount of data labeled by crowd-workers and generalized to learn the correspondence fields for face [ 2 ] and chimpanzees [ 36 ] . Key limitations of these approaches are the inaccuracy of labeling and requirement of large labeled data . We address these limitations by formulating multiview supervision that can enforce geometric consistency , which allows utilizing a large amount of unlabeled multiview images . Multiview Feature Learning Epipolar geometry can be used to learn a visual representation by transferring visual information from one image to another via epipolar lines or 3D reconstruction . For instance , a fusion layer can be learned to fuse feature maps across views [ 30 ] . Such fusion models can be factored into shape and camera components to reduce the number of learnable parameters and improve generalizability [ 49 ] . Given the camera calibration , a light fusion module can be learned to directly fuse deep features [ 10 ] or heatmaps [ 55 ] from other view along corresponding epipolar line . Several works combine multiview image features to form 3D features [ 13 , 46 ] or view-invariant feature in 2D [ 32 ] . Multiview Supervision Synchronized multiview images [ 11 , 15 , 52 ] possess a unique geometric property : images are visually similar yet geometrically distinctive , provided by stereo parallax . Such property offers a new opportunity to learn a geometrically coherent representation without labels . Bootstrapping by 3D reconstruction [ 22 ] can be used to learn a keypoint detector supervised by the projection of the 3D reconstruction to enforce cross-view consistency . MONET [ 51 ] enables an end-to-end learning by eliminating the necessity of 3D reconstruction and directly minimizing epipolar error . Learning keypoints can be combined with 3D pose estimation [ 34 , 12 ] by enforcing predicting the same pose in all views while using a few labeled examples with 3D or 2D pose annotations to prevent degeneration . One can alleviate the need for large amounts of annotations by matching the predicted 3D pose with the triangulated pose [ 19 ] . Several works further learn a latent representation encoding 3D geometry from images [ 5 , 25 ] or 2D pose [ 33 , 41 ] by enforcing consistent embedding , texture [ 28 ] , and view synthesis [ 45 ] across views . Unlike these approaches designed for sparse keypoints where the geometric consistency is applied on finite points , we study geometric consistency on continuous dense keypoint fields . Capture Dense [ 22 ] is the closest work to ours , which uses bootstrapping through 3D reconstruction of human mesh model [ 16 ] . However , due to the non-differentiable nature of bootstrapping , it is not end-to-end trainable . A temporal consistency has been also used for self-supervise the dense keypoint detector [ 27 ] . 3 Method . We present a novel method to learn a dense keypoint detector by using unlabeled multiview images . We formulate the epipolar constraint for continuous correspondence fields , which allows us to enforce geometric consistency between views . 3.1 Dense Epipolar Geometry . Given a pair of corresponding images from two different views , a pair of corresponding points x↔ x′ , are related by a fundamental matrix given the calibrated cameras , i.e . x̃′TFx̃ = 0 , where F ∈ R3×3 is the fundamental matrix , and x̃ ∈ P2 is a homogeneous representation of x . The measure of geometric consistency between the two images can be written as [ 9 ] : d ( x , x′ ; F ) = |x̃′TFx̃|√ ( Fx̃ ) 21 + ( Fx̃ ) 2 2 , x↔ x′ , ( 1 ) where ( Fx ) i is the ith entry of Fx . This measures epipolar error that is the Euclidean distance between x′ and the epipolar line Fx̃ . Consider an injective dense keypoint mapping φ : R2 → R2 that maps a pixel coordinate to a canonical 2D body surface coordinate , i.e. , u = φ ( x ; I ) where x ∈ Θ ( I ) is the set of the foreground pixels in the image I , and u ∈ R2 is the 2D coordinate in the body surface as shown in Figure 2 . This dense keypoint mapping can be learned from annotations , e.g. , DensePose [ 8 ] for humans where it maps to the texture coordinate of 3D human body surfaces . This mapping is incomplete because there exists missing correspondences in an image due to occlusion . One can find a correspondence between the two images through u , i.e. , φ−1 ( u ; I ) ↔ φ−1 ( u ; I ′ ) where I and I ′ the two view images . However , φ is an injective mapping where the analytic inverse does not exist in general . Given a point x in the image I , one can measure the expectation of geometric error by a nearest neighbor search on the body surface space : E ( x ) = d ( x , x′ ; F ) , x′ = argmin x′∈Θ ( I′ ) ‖φ ( x ; I ) − φ ( x′ ; I ′ ) ‖ , ( 2 ) where E ( x ) is the expectation of geometric error at x . The expectation of the geometric error measures the epipolar error over all possible matches in the other view image , i.e. , ∀x′ ∈ Θ ( I ′ ) . There are two limitations in the nearest neighbor search : ( 1 ) The correspondences are not exact , leading to a biased estimate of the geometric error expectation . ( 2 ) The argmin operation is not differentiable , which can not be used to learn a dense keypoint detector in an end-to-end fashion . Instead , we address these limitations by making use of a soft correspondence , or matchability—a likelihood of a point being matched to another point as shown in Figure 2 : M ( x , x′ ; φ , I , I ′ ) = P ( φ ( x ; I ) , φ ( x′ ; I ′ ) ; { φ ( y ; I ′ ) } y∈Θ ( I′ ) ) , ( 3 ) where P ( u , u′ ; Ω ) is the probability of matching between u and u′ : P ( u , u′ ; Ω ) = exp ( −‖u− u ′‖2 2σ2 ) / ∑ v∈Ω exp ( −‖u− v‖ 2 2σ2 ) . ( 4 ) Ω is the domain of the canonical coordinates , σ is the standard deviation that controls the smoothness of matching , e.g. , when σ → 0 , it approximates the nearest neighbor search . We use the matchability to form a probabilistic epipolar error : E ( x , x′ ) = M ( x , x′ ; φ , I , I ′ ) d ( x , x′ ; F ) , ( 5 ) where E ( x , x′ ) is the epipolar error between x and x ’ weighted by the matchability . The error expectation of x can be computed by marginalizing over all x′ : E ( x ) = ∑ x′∈Θ ( I′ ) E ( x , x′ ) . ( 6 ) That is , the expectation of the geometric error of x measures a weighted average of epipolar errors for all possible correspondences . The higher matchability , the more contribution to the error expectation . Unlike Equation ( 2 ) , Equation ( 6 ) does not include the non-differentiable argmin operation . Further , it takes into account all possible pairs of correspondences between two images , which eliminates the bias introduced by the nearest neighbor search . Given the error expectations over the dense keypoint field , we derive a symmetric dense epipolar error that measures geometric consistency between two images : E ( I , I ′ ) = 1 V ∑ x∈Θ ( I ) v ( x ; φ , I , I ′ ) E ( x ) + 1 V ′ ∑ x′∈Θ ( I′ ) v ( x′ ; φ , I , I ′ ) E ( x′ ) , ( 7 ) where v ( x ; φ , I , I ′ ) ∈ { 0 , 1 } is a visibility indicator : v ( x ; φ , I , I ′ ) = { 1 , ∃x′ ∈ Θ ( I ′ ) , ‖φ ( x ; I ) − φ ( x′ ; I ′ ) ‖ < 0 , otherwise . ( 8 ) V = ∑ x∈Θ ( I ) v ( x ; φ , I , I ′ ) and V ′ = ∑ x′∈Θ ( I′ ) v ( x ′ ; φ , I , I ′ ) are the numbers of foreground pixels visible in I ′ and I , respectively . is a correspondence tolerance defined in the canonical surface coordinate . In fact , this dense epipolar error is a generalization of Sampson distance [ 9 ] that defines epipolar error between explicit point correspondences ( point-to-point ) . We generalize it to model probabilistic correspondence between two set of points ( field-to-field ) . Existing approaches such as bootstrapping [ 22 ] establish the matching through 3D reconstruction of mesh and enforce the geometric error in an alternating fashion due to the non-differentiability of matching . Our differentiable formulation allows learning the dense keypoint detector in an end-to-end manner , which is flexible and shows superior performance .

Summary: This paper proposes a new method for the semi-supervised learning of keypoints using multiview images. The main challenge addressed is finding exact correspondences betweenkeypoints in multiple views, since the inverse of keypoint matching cannot be analytically derived or differentiated. The proposed probabilistic epipolar constraint encodes a soft correspondence and geometric consistency in the correspondence field. A distillation-based regularization is also proposed to avoid degenerative cases. Each contribution is ablated and show significant improvement over the baseline and relative to other methods.