The magnitude vector of images

1 INTRODUCTION . The topology community has recently invested much effort in studying a newly introduced quantity called magnitude ( Leinster , 2010 ) . While it originates from category theory , where it can be seen as a generalization of Euler characteristics to metric spaces , the magnitude of a metric space is most intuitively understood as an attempt to measure the effective size of a metric space . As a descriptive scalar , this quantity extends the set of other well known descriptors such as the rank , diameter or dimension . However , unlike those descriptors , the properties of magnitude are not yet well understood . Even though the metric space structure of dataset is of the utmost importance to understand e.g . the regularisation behaviour of classifiers , magnitude has received little attention by the machine learning community so far . However , it turns out that one can instead investigate the magnitude of each data point separately , considering each data instance as its own metric space . Following this line of thought , magnitude vectors were introduced as a way to characterise the contribution of each data sample to the overall magnitude , such that the sum of the elements of the magnitude vector amounts to the magnitude . As shown in previous works , the magnitude vectors can detect boundaries of a metric space , with boundary points having a larger contribution to magnitude ( Bunch et al. , 2021 ) . In this work , we seek to advance the research about magnitude in machine learning by addressing these challenges . Since the metric space structure of an entire dataset may not be the most useful application in machine learning , we instead therefore consider the magnitude of each individual data point by endowing each of them with a metric space structure and explore its properties . In particular , because of their ubiquity and the large dimensionality , we focus our analysis on image datasets , where each individual image is seen as a metric space . We then extend previous results by investigating the theoretical properties associated with magnitude vector in such a context . In addition , we study the properties of the magnitude vector for improving the adversarial robustness of existing neural network architectures for image classification . We propose a novel , fully differentiable , magnitude layer , which can serve as an input layer for any deep learning architecture . We show that it results in more robustness for several types of adversarial attacks , with an acceptable reduction in classification performance , paving the way for a new exciting direction in the creation of robust neural architectures . Moreover , since naive magnitude calculations may be computationally prohibitive for large images , we introduce a new algorithm that dramatically speeds up the computation of the magnitude vector . Leveraging the regular structure of images , this allows to conveniently approximate magnitude vectors for large images with minimal error . Intractable computational runtime often stymies the applicability of magnitude in machine learning and therefore hinders further the research of it ; our algorithm opens the door to using magnitude efficiently in machine learning research . Equipped with the speed up algorithm , we showcase possible use cases of magnitude vectors in machine learning in the realm of in edge detection and adversarial robustness . Our contributions are summarized as follows : • We formalize the notion of magnitude vectors for images and investigate the impact of the choice of different metrics . • We introduce an algorithm that dramatically speeds up the computation of magnitude with little loss , which removes a critical roadblock in using magnitude vectors in machine learning applications . • We provide a theoretical framework to understand the edge detection capability of magnitude vectors and report empirical supporting evidence . • We demonstrate the capabilities of a novel , fully differentiable , magnitude layer for improving adversarial robustness of computer vision architectures . 2 THEORY . In this section , we introduce essential notions of the theory of magnitude and magnitude vectors and develop the theoretical background that will help interpreting the empirical results . 2.1 MATHEMATICAL BACKGROUND . We start by formally introducing the notion of a finite metric space . Definition 1 . A metric space is an ordered pair ( B , d ) , where B is a finite set and d is a metric on B . We denote the cardinality of B by |B| . In our application the set B is a set of vectors B ⊂ Rn and the metric considered will be the ` p norm . In order to define the magnitude of such a space we first define the similarity matrix . Definition 2 . Given a finite metric space M = ( B , d ) , its similarity matrix is ζM with entries ζM ( i , j ) = e −d ( Bi , Bj ) for Bi , Bj ∈ B . The similarity matrix can be seen as the weights arising from an exponential kernel . We are now in a position to define the magnitude vector and the magnitude of a finite vector space . Definition 3 . Given a finite metric space M = ( B , d ) with |B| = n and similarity matrix ζM with inverse ζ−1M , the magnitude vector of element Bi is given by wi = ∑n j=1 ζ −1 M ( i , j ) . Moreover , the magnitude of M , magM is ∑n i , j=1 ζ −1 M ( i , j ) = ∑n i=1 wi . Not every finite metric space has a magnitude . In particular , the magnitude is not defined when the similarity matrix is not invertible ; the magnitude therefore characterizes the structure of a metric space to some extent . It should be also noted that the definition of the magnitude vector is reminiscent of optimising a support vector machine . This connection has been pointed out for the Euclidean norm by Bunch et al . ( 2021 ) . 2.2 THEORETICAL RESULTS . 2.2.1 THE MAGNITUDE OF AN IMAGE . This work focuses on the analysis of magnitude on images , by considering each individual image as its own metric space . We then first define how we build such a metric space from an image and then prove the existence of a magnitude on images . We refer to the metric spaces on images as image metric spaces and define them as follows . Definition 4 . Let I ∈ Rc×n×m be an image with c channels , height n and width m. Let { V : vij , i ∈ 1 , . . . , m ; j ∈ 1 , . . . , n } be the set of c-dimensional vectors of pixel values in the image . Then the image metric space M ( B , d ) is given by a set of vectors B ⊂ Rc+2 of the form B = { ( i , j , v ( 1 ) ij , . . . , v ( c ) ij ) T : ∀vij ∈ V } together with a metric d on Rc+2 . Informally , we put all vectors corresponding to pixel values on a grid , concatenating the position vector with the pixel value vector and use the resulting vectors as the ground set B for our finite metric space . Let us note that |B| = m× n and therefore the number of points in the metric space can be quite large even for moderately-sized images , a potential limitation that we address in Section 2.3 . We now turn to investigating when an image metric space has magnitude and , by extension , a magnitude vector . This is a priori not clear since , as we saw , the existence of magnitude depends on the properties of the metric space . From the definition of magnitude ( Definition 3 ) , it is readily seen that an image metric space M has magnitude if and only if its similarity matrix ξM is invertible . As generic n × n matrices are invertible ( i.e . subjecting any non-invertible matrix to a random perturbation will almost certainly result in an invertible matrix ) , we conclude that generic image metric spaces have magnitude . In fact , since the vectors b ∈ B of the image metric space by a factor t > 0 can be rescaled , we can define scaled metrics d ( · , · ) 7→ td ( · , · ) . This rescaled metric space has magnitude except for finitely many t > 0 ( Leinster et al. , 2017 , Proposition 2.8 ) . In other words , we can always find a scaling which gives rise to a magnitude vector . The univariate function defined by this scaling is called the magnitude function . In general , we are interested in computing the magnitude of all images in a whole dataset , and not only of a single image , such that we can compare their magnitude ( vectors ) . Therefore , we would like theoretical guarantees that there exists a scaling such that every image in a dataset has a magnitude vector . Proposition 5 . Let M ( B , d ) be an image metric space . If d is an ` p norm with 0 < p < 2 , then every image metric space M has magnitude . Proof . This is a special case of ( Meckes , 2013 , Theorem 3.6 ) . The above proposition theoretically only applies to ` p norm with 0 < p < 2 . However , in practice , we find that on the various datasets we considered in this work , an ` p norm with p = 4 , 10 and the Hamming distance also lead to invertible similarity matrices , and thus to the existence of magnitude . The image metric space exhibits substantial structure ; in particular , there is an underlying regular subspace ( the grid ) . To quantify this further , we use the notion of a product space . Lemma 6 . Let M1 ( B1 , d1 ) and M2 ( B2 , d2 ) be two finite metric spaces with d1 , d2 being ` p norms . Its product metric spaceM = M1×M2 is a metric space with a metric ρp : ( B1×B2 ) × ( B1×B2 ) → R such that ρp ( ( x1 × y1 ) , ( x2 × y2 ) ) = p √ ( d1 ( x1 , x2 ) p + d2 ( y1 , y2 ) p , 1 ≤ p ≤ ∞ . Proof . This is a special case of ( Dovgoshey & Martio , 2009 ) . Note that the product space formulation gives a lot of freedom , since every positively weighted combination of d1 and d2 is also a metric on the product space . Magnitude vector on images based on harmonic analysis We now present our main result of this subsection which is an interpretation of the magnitude vector for images based on harmonic analysis . For this we consider grey scale images , however , our reasoning generalises to colour images . First , notice that a grey scale image can be seen as discretisation of a surface in R3 , i.e . z = f ( x , y ) , where x , y are the pixel positions and z is the brightness . In general , it is not clear what an “ outlier ” on a continuous surface or curve is . In this paper we define an outlier as a point on the surface where the gradient is large , i.e . neighbouring points are further away w.r.t . some distance measure . In the image case , outliers are points where the brightness value is substantially different between neighbouring pixels . We can use this reasoning to define a filtration F on the vectors of B , bij = ( i , j , v ( 1 ) ij ) T , namely ∅ ⊆ B ( 1 ) ⊆ B ( 2 ) ⊆ · · · ⊆ B ( K ) = B such that v ( 1 ) ij < δ ( k ) for k = 1 , . . . , K , where the δ ( k ) are different brightness thresholds . Due to symmetry in the problem we also define F ′ with criterion v ( 1 ) ij ≥ δ ( k ) . Further , we define the projection onto R2 , p : R3 → R2 , p ( bij ) 7→ ( i , j ) . By considering the projections of each subset of F ( F ′ ) we break the problem down into successive boundary detection of compact subsets of R2 . To extend this reasoning to colour images we can either consider each channel as a grey scale image or define multi-dimensional filtrations . A visual description of our reasoning is found in Figure 1 . To investigate the effect of different metrics on boundary detection , we consider the behaviour of the weighting vector on the 2-dimensional grid . We closely follow the argument of Bunch et al . ( 2021 ) , extending their results to the product space metric in the case when di are ` p norms with p = 1 , 2 and ρ ( ( x1 × z1 ) , ( x2 × z2 ) ) = α1d1 ( x1 , x2 ) + α2d2 ( z1 , z2 ) ( 1 ) for some positive weights α1 , α2 . For image we can consider the xi as two-dimensional vectors indicating the position on the unit square and zi as the corresponding brightness value . Consider a regular 2d grid and the equation defining a weighting on the grid points ζMv = ( 1 , . . . , 1 ) T . Using a continuous analogue , this can be written as a convolution ( Bunch et al. , 2021 ) f ? v ( x ) = ∫ R2 f ( x− y ) v ( y ) = ∫ R2 e−αd ( x , y ) v ( y ) = I [ 0,1 ] 2 ( x ) , ( 2 ) where I [ 0,1 ] 2 is the indicator function and d is any translation invariant metric . Using the Fourier transform , F ( f ) ( ξ ) = ∫ R2 e−i2πx·ξf ( x ) dx , ( 3 ) its well-known properties Folland ( 1999 ) and the convolution theorem , we can derive an intuitive understanding of the magnitude vector and the effects of specific metrics . In the case of d ( · , · ) being the Euclidean ( ` 2 ) norm , it has been shown in ( Bunch et al. , 2021 ) that Γ ( 32 ) απ 3 2 ( α2 − 2∑ i=1 ∂2i ) 3 2 I [ 0,1 ] 2 = v , ( 4 ) where we generalised the result of Bunch et al . ( 2021 ) to the unit square . We can interpret equation 4 as the weighting v being constant in the interior of the unit square ( ∂2i is zero in interior ) and different to this constant on the boundary . In fact , the weighting will also be constant on the edges and corners . In the Euclidean case it has been shown ( Bunch et al. , 2021 ) that the intuitive understanding translates rigorously to compact subsets of Rn with n odd using distribution theory . Guided by this , we continue our argument using the Fourier transform . If d ( · , · ) is the ` 1 norm , i.e . the Manhattan or Cityblock distance , we obtain a similar result to equation 4 , [ 2∏ i=1 ( α2 − ∂2i ) ] I [ 0,1 ] 2 = v , ( 5 ) in other words , the ` 1 norm admits the same interpretation as ` 2 , however , this time the differential operators acts multiplicatively on each dimension . For the product space metric ρ ( · , · ) = α1d1 ( · , · ) + α2d2 ( · , · ) and di are ` p with p = 1 , 2 it follows that the Fourier transform is a product of the single-metric results . Equipped with this insight , we can now explain the edge detection capabilities of the magnitude vector . Consider a filtration F ( and F ′ ) on a grey scale image and choose a subset of vectors B ( i ) from the filtration . Apply the projection map p to every vector in this subset . The transformed set is a grid with potentially “ missing ” grid points on the domain [ 0 , n ] × [ 0 , m ] . From the results on the unit square and the discretising of the ∂ operator , we expect a constant weighting vector except on the boundaries of the grid , i.e . on points adjacent to the missing grid points and on the boundaries of the domain . This procedure can be performed on every B ( i ) ∈ F and the final magnitude vector can be seen as a a combination of the weightings of each step ( see Figure 1 ) . We find that our expectations from the theoretical result agree well with empirical results ( see Appendix D ) . Moreover , we empirically find that Hamming distance and ` p norms for p 6= 1 , 2 have similar properties .

Summary: This paper applies the machinery of magnitude vectors of metric spaces to extract information from individual images (which can be themselves be seen as finite metric spaces having one point for each pixel). After some theoretical considerations, the paper proposes a divide-and-conquer algorithm to approximate the magnitude vector of an image which is significantly faster than the naive inversion of a $N\times N$ matrix (where $N$ is the number of pixels in the image). Finally, the magnitude vector is shown to correlate with the result of traditional edge detectors (after hyper-parameter tuning) and an application to adversarial robustness is described.