Holographic and other Point Set Distances for Machine Learning

We introduce an analytic distance function for moderately sized point sets of known cardinality that is shown to have very desirable properties , both as a loss function as well as a regularizer for machine learning applications . We compare our novel construction to other point set distance functions and show proof of concept experiments for training neural networks end-to-end on point set prediction tasks such as object detection . 1 INTRODUCTION . Parametric machine learning models , like artificial neural networks , are routinely trained by empirical risk minimization . If we aim to predict an output y ∈ Y from an input x ∈ X , we collect a training setD of ( x , y ) pairs and train a parametrized prediction function hθ : X → Y by minimizing R ( θ ) : = 1 |D| ∑ ( x , y ) ∈D L ( hθ ( x ) , y ) . ( 1 ) Here , L : Y × Y → R is a loss function that assigns a scalar loss value to the prediction ŷ = hθ ( x ) for the true value y . Classical machine learning problems allow for standard choices for the loss function . E.g. , for regression problems , where y , ŷ ∈ R , it is common to use the squared loss L ( ŷ , y ) = ( ŷ − y ) 2 . Assume , however , that we want to train a machine learning system which—given some input— predicts a set of points . In particular , the ordering of the points is neither semantically meaningful nor in any way consistent across data instances . As an example , one may consider an object detection task such as finding the positions of the black stones on an image of a real-world game of Nine Men ’ s Morris . What should the loss function be in such a case ? Naively , we might just impose an arbitrary ordering to the sets and treat them as ordered tuples . However , this conceals an important property of the task , the permutation invariance , from our machine learning system . This property might in fact be crucial to learn such a task efficiently . Instead , we would like our loss function to define a meaningful distance between two sets , which requires such a loss to be permutation-invariant . 1.1 PERMUTATION-INVARIANT LOSS FUNCTIONS . We consider distance functions a for pair of finite sets of points from Rd and assume the cardinality N to be the same for both sets and known in advance . While we are concerned with unordered sets of points , representing them in a machine-learning context will require us to impose some ordering to its elements . Hence , we will represent a set { z1 , . . . , zN } ⊂ Rd of points as an ordered tuple Z = ( z1 , . . . , zN ) and define the shorthand ZN : = ( Rd ) N for the space of such tuples . If a loss function L ( Z , Ẑ ) operating on such ordered tuples is supposed to define a meaningful distance between the underlying sets , it will have to be invariant to their arbitrary ordering . Hence , for Z = ( z1 , . . . , zN ) ∈ ZN and a permutation σ ∈ SN , we define σ ( Z ) : = ( zσ ( 1 ) , . . . , zσ ( N ) ) and demand the following property . Definition 1 ( Permutation-invariant loss ) . We call a function L : ZN × ZN → R permutationinvariant if and only if for any Ẑ , Z ∈ ZN and permutations π , σ ∈ SN , we have L ( π ( Ẑ ) , σ ( Z ) ) = L ( Ẑ , Z ) . ( 2 ) 1.2 RELATED WORK . While distance measures for sets have been studied in Mathematics and theoretical computer science , they have rarely been investigated in the context of machine learning . A natural way to define a point set distance is to explicitly find a “ best ” matching between target and prediction points and take the distance between the re-ordered tuples , e.g. , LHung ( Ẑ , Z ) = min σ∈SN ∑ i∈ [ N ] ‖zi − ẑσ ( i ) ‖2 . ( 3 ) This has been proposed under the name Hungarian loss by Stewart et al . ( 2016 ) for object detection . The name refers to the Hungarian algorithm ( Kuhn , 1955 ) , which can be used to solve the assignment problem . To our knowledge , that is the first work to propose a permutation-invariant loss function for training a machine learning model . This loss function is intuitive and is a simple , well-conditioned quadratic function of Ẑ in regions where the optimal permutation is constant . But it also has its drawbacks . Its gradient is undefined at “ transition points ” with more than one optimal permutation . More importantly , having to solve the assignment problem ( Eq . 3 ) for each single evaluation of the loss function is problematic in a machine learning context , where we nowadays rely on simple , highly-efficient computational modules in frameworks such as TensorFlow ( Abadi et al. , 2016 ) . It can also be computationally slow , since the Hungarian algorithm is O ( N3 ) and does not lend itself to a highly-parallelized implementation on GPUs . Rezatofighi et al . ( 2018 ) propose to use a neural network to jointly model the elements of a set as well as the permutation in which they appear . This approach gives rise to an alternating optimization procedure , where the first step solves for an optimal permutation using the Hungarian algorithm and the second step updates the weights of the neural network . We note that their probabilistic formulation provides an elegant framework to learn the cardinality of the set alongside its elements . While this is certainly important in practice , in this paper we focus on investigating properties of some point set distance functions , assuming the cardinality to be known . Future work could integrate these loss functions with the cardinality learning procedure of Rezatofighi et al . ( 2018 ) to replace the cumbersome inner-loop solution of an optimal assignment problem . Zaheer et al . ( 2017 ) discuss neural network architectures that take sets as inputs . They find a general characterization of permutation-invariant functions and use that insight to devise neural network architectures which operate on sets in a permutation-invariant fashion . 1.3 SCOPE OF THIS WORK . In this work , we consider some alternative permutation-invariant point set distance functions , all of which have running time O ( N2 ) and can be implemented with operations that are readily available in automatic differentiation frameworks like TensorFlow . In particular , they do not rely on the solution of an assignment problem akin to Eq . ( 3 ) . We pay specific attention to properties relevant to machine learning , in particular , how amenable such distances are to gradient-based numerical optimization . As one option , we propose a novel distance function for point sets , which we name “ holographic loss ” , since it is a metric distance on fingerprints of point sets that “ holographically ” encodes their structure , i.e . moving any point in the set will collectively ( and analytically ) change all entries in the fingerprint-vector . We show that it has favorable properties in addition to being analytic everywhere , such as having a diagonal Hessian at the minima , which furthermore all are global and correspond to exactly matched up point sets . Also , this loss function has a natural generalization to multi-sets . We present proof of concept experiments on a simple object detection task based on MNIST digits , where we train a convolutional neural network to directly predict the locations of digits in an image . These experiments show that end-to-end training with a simple permutation-invariant loss function is a viable approach for problems that can be formulated as a point set prediction task . 2 HAUSDORFF DISTANCE AND SUM OF MINIMAL DISTANCES-SQUARED . Distance functions for sets have been studied at least since the days of Hausdorff ’ s 1914 book on set theory Hausdorff ( 1914 ) . Considering Hausdorff ’ s construction in our context , i.e . as a ( 2nd order ) loss function for finite point sets , the one-sided Hausdorff distance is given by LOSH ( Ẑ , Z ) 2 = maxi∈ [ N ] minj∈ [ N ] ‖zi − ẑj‖2 , which is the maximum squared distance of any target point to its closest prediction point . The symmetric Hausdorff distance is LH ( Ẑ , Z ) 2 = max ( LOSH ( Ẑ , Z ) 2 , LOSH ( Z , Ẑ ) 2 ) . ( 4 ) LH,2 is easily computed with O ( N2 ) effort , almost everywhere differentiable , and makes intuitive sense as a machine learning loss function . However , it has some properties that might not be desirable in a machine learning context . Since the Hausdorff distance is determined by a single “ worstoffender ” pair of target and prediction points , each gradient descent step will only adjust the position of a single prediction point , until there is an almost-tie between different point-pairs , and henceforth subsequent gradient steps will jump around and keep almost-tied point pairs almost-tied . It stands to reason that this might , especially with some optimization strategies , be undesirable compared to loss functions whose gradient collectively moves all points . A possible remedy to that problem is to sum the minimal distances-squared instead of taking their maximum . The one-sided sum of minimal distances-squared is LOSSMD ( Ẑ , Z ) 2 = 1 N ∑N i=1 minj∈ [ N ] ‖zi − ẑj‖2 and the ( symmetric ) sum of minimal distances-squared is LSMD ( Ẑ , Z ) 2 = 1 2 ( LOSSMD ( Ẑ , Z ) 2 + LOSSMD ( Z , Ẑ ) 2 ) . ( 5 ) We do not consider the one-sided variants of these two distance functions in our comparison . The one-sided variants centered on the prediction points will not make sense as a machine learning loss function , since they become zero when matching all predicted points to any one of the target points , e.g. , ẑj = z1 for all j . The one-sided variants centered on the target points , while more sensible , have simple failure cases as well , such as the one depicted in Figure 4 . We thus restrict our attention to the symmetric loss functions LH and LSMD . Figures 1a and 1b show contour plots for these two functions on a simple problem with two one-dimensional target points , while 1c shows our construction , described in the next section . 3 “ HOLOGRAPHIC ” LOSS . 3.1 SET FINGERPRINTS . A general strategy to devise permutation-invariant loss functions is to ( differentiably ) map tuples Z representing a point set ( or multi-set ) to a “ fingerprint ” vector f ( Z ) in some fingerprint-space F such that f ( Ẑ ) = f ( Z ) if and only if Ẑ = σ ( Z ) for some permutation σ . A permutation-invariant loss function can then be defined via ( some ) metric distance in F. Any such loss function will inherit properties of the distance in F , such as positive definiteness , symmetry , or the triangle inequality . In one dimension , a suggestive choice for a fingerprint function on tuples of N points would be a vector of moments Mk ( Z ) : = ∑N i=1 z k i . The mathematical appendix of Zaheer et al . ( 2017 ) identifies the moments of points in a set as an universal permutation-invariant fingerprint , in the sense that every other such fingerprint can be defined in terms of these quantities . This suggest using squared-euclidean-distance-of-moments-vectors as a loss function . However , this has some fundamental problems , such as poor conditioning , and it is non-trivial to generalize to higher-dimensional points . See Appendix B for a detailed discussion . 3.2 EXPLOITING THE 2-D { POINT SET } ⇔ { POLYNOMIAL } CORRESPONDENCE In this section , we want to focus our attention to two-dimensional point sets ( or multi-sets ) , and consider one-dimensional sets as a special case with all points having a 2nd coordinate of zero . We will subsequently consider generalizations to higher dimensions . We start by identifying a two-dimensional point z = ( x , y ) ∈ R2 with the complex number z = x + iy ∈ C. With this , we can map any tuple Z = ( z1 , . . . , zN ) that represents a ( multi- ) set to a complex polynomial PZ ( u ) = ( u− z1 ) ( u− z2 ) · · · ( u− zN ) . ( 6 ) Note that PZ = PẐ if and only if Ẑ = σ ( Z ) for some permutation σ ∈ SN . Furthermore , such a monic ( i.e . having leading coefficient 1 ) polynomial of degree N is uniquely determined by its values atN distinct points . This suggests to choose a set U = ( u1 , . . . , uN ) ofN distinct evaluation points and to define a fingerprint as f ( Z ) = ( PZ ( u1 ) , . . . , PZ ( uN ) ) . ( 7 ) Taking the real-valued L2 distance in CN gives us the “ holographic ” loss function LHolU ( Ẑ , Z ) 2 = 1 N N∑ i=1 |PZ ( ui ) − PẐ ( ui ) | 2 = 1 N N∑ i=1 ( PZ ( ui ) − PẐ ( ui ) ) ∗ · ( PZ ( ui ) − PẐ ( ui ) ) . ( 8 ) While the underlying fingerprint is holomorphic , i.e . complex differentiable , the distance-squared function is not , due to anti-linearity of the underlying complex scalar product in the first argument . The adjective ‘ holographic ’ here refers to this function depending collectively on the locations of all points , unlike Hausdorff distance , which for a generic configuration will not change when slightly changing the positions of any points other than the specific pair that determines its value . Also , for U 6= Z , correctly matching up the candidate and target points means matching up both the complex amplitude as well as the complex phase at the evaluation-points . However , the analogy with holography ends here , there is no deeper correspondence beyond that . Proposition 1 . The Holographic loss trivially has the following properties : • Positive definiteness : LHolU ( Ẑ , Z ) 2 ≥ 0 and LHolU ( Ẑ , Z ) 2 = 0 if and only if Ẑ = σ ( Z ) . • Symmetry : LHolU ( Ẑ , Z ) 2 = LHolU ( Z , Ẑ ) 2 • Triangle inequality : √ LHolU ( Ẑ , Z ) 2 ≤ √ LHolU ( Ẑ , Z ′ ) 2 + √ LHolU ( Z ′ , Z ) 2 . • LHolU ( Ẑ , Z ) 2 is quadratic in each individual ẑi ’ s real and also imaginary part when keeping all other point-coordinates fixed .

Summary: This paper proposes a new loss for points registration (aligning two point sets) with preferable permutation invariant property. For a 2D point set, the idea is to define a complex polynomial with the points (interpreted as complex numbers) as roots. To compare two point sets, the loss eventually boils down to evaluating the complex polynomial at all target points. The loss can be extended to high-dimensional point sets without the same theoretical framework. It is a sum over each target point the product of the distance of between the target point and all points of the source set. The claimed advantage of the proposed distance is it is more efficient compare to other metric such as Hausdorff distance or the sum of squared distance between matched point pairs (SMD). Experiments show that the proposed loss can be used in training.