Limitations for Learning from Point Clouds

In this paper we prove new universal approximation theorems for deep learning on point clouds that do not assume fixed cardinality . We do this by first generalizing the classical universal approximation theorem to general compact Hausdorff spaces and then applying this to the permutation-invariant architectures presented in PointNet ( Qi et al ) and Deep Sets ( Zaheer et al ) . Moreover , though both architectures operate on the same domain , we show that the constant functions are the only functions they can mutually uniformly approximate . In particular , DeepSets architectures can not uniformly approximate the diameter function but can uniformly approximate the center-of-mass function but it is the other way around for PointNet . Additionally , even when the point clouds are limited to at most k points , PointNet can not uniformly approximate center-of-mass and we obtain explicit error bounds and a method to produce geometrically derived adversarial examples . 1 INTRODUCTION . Recently , architectures proposed in PointNet ( Qi et al. , 2017 ) and Deep Sets ( Zaheer et al. , 2017 ) have allowed for the direct processing of point clouds within a deep learning framework . These methods produce outputs that are permutation-invariant with respect to the member points and work for point clouds of arbitrarily large cardinality . A common source of such data is LIDAR measurements from autonomous vehicles . Zaheer et al . ( 2017 ) also presents a permutation-equivariant architecture which we do not discuss here . Each of of these works provide their own universal approximation theorem ( UAT ) to support the empirical success of their architectures . However , both results assume the cardinality of the point cloud is fixed to some size n. In this work we refine these results , remove the cardinality limitation , use weaker architecture assumptions , and arrive at three main results which can be summarized roughly as follows ( assuming unrestricted finite cardinality for the input point clouds ) : 1 ) PointNet ( DeepSets ) architectures can uniformly approximate real-valued functions that are uniformly continuous with respect to the Hausdorff ( Wasserstein ) metric and nothing else ( Theorem 3.4 ) . 2 ) Only the constant functions can be uniformly approximated by both architectures . In particular , PointNet architectures can uniformly approximate the diameter function but DeepSets architectures can not . Conversely , DeepSets architectures can uniformly approximate the center-of-mass function but PointNet architectures can not ( Theorem 4.1 ) . 3 ) We prove explicit error lower bounds and produce adversarial examples to show that even when limited to point clouds of size k , PointNet can not uniformly approximate center-ofmass ( Theorem 4.2 ) . To do this we extend the many universal approximation results for feed-forward networks ( Cybenko , 1989 ; Hornik et al. , 1989 ; Leshno et al. , 1993 ; Stinchcombe , 1999 ) to the abstract setting of general compact Hausdorff spaces . We then find appropriate compact metric spaces over which PointNet and DeepSets architectures can be easily analyzed and then finally we observe the resulting consequences in the original setting of interest , i.e . point clouds . 2 PRELIMINARIES . 2.1 POINTNET AND DEEPSETS ARCHITECTURES . In practice , the implementations of the architectures presented in PointNet and Deep Sets can involve many additional tricks , but the essential ideas are quite simple . We do however make a small modification to the Deep Sets model . For A ⊆ Rn of cardinality |A| < ∞ , we have Qi et al . ( 2017 ) and Zaheer et al . ( 2017 ) suggesting scalar-output neural networks of the form FPN ( A ) = ρ ( max a∈A ϕ ( a ) ) , and FDS ( A ) = ρ ( b+ 1 |A| ∑ a∈A ϕ ( a ) ) , respectively . Here ϕ : Rn → Rm creates features for each point in A , then a symmetric operation is applied , and then ρ : Rm → R combines these features into a scalar output ( here max is the component-wise maximum ) . In practice , we need both ρ and ϕ to be neural networks . Note that because we use a symmetric operation before ρ , the output will not depend on the ordering of points in the point cloud , and because the max and sum operations scale to arbitrary finite cardinalities the size of the point cloud is not an issue . The original model in Deep Sets did not have a bias term b and used a sum instead of the averaging we use here . This change will help us later in our theoretical analysis . It will help to introduce some simplifying notation . Let F ( Ω ) denote the set of all nonempty finite subsets of a set Ω ( i.e . point clouds in Ω ) , F≤k ( Ω ) the set of nonepmty subsets of size ≤ k , and Fk ( Ω ) the set of k-point subsets . Now consider Ω ⊆ RN and define maxf , avef , b : F ( Ω ) → R which are given by maxf ( A ) = maxa∈A f ( a ) and avef , b ( A ) = b + 1|A| ∑ a∈A f ( a ) respectively . We make sense of this in the natural way if we use vector-valued f and b by operating componentwise . We call these operations max neurons and biased-averaging neurons respectively . Once again letting ρ and ϕ be neural networks , FPN = ρ ◦ maxϕ and FDS = ρ ◦ aveϕ , b will be the general form of what we call the PointNet and DeepSets architectures ( resp . ) in this paper . Some natural questions are 1 ) is there a topology forF ( Ω ) that makes these architectures continuous , 2 ) how expressive are these approaches , and 3 ) how deep is deep enough for function approximation ? 2.2 FUNCTION SPACES AND UNIFORM APPROXIMATION . From now on , we only consider R-valued functions unless otherwise stated . Let B ( A ) be the set of bounded functions on a set A , let C ( X ) and Cb ( X ) be the set of continuous and bounded continuous functions on a topological space X ( respectively ) , and let U ( M ) and Ub ( M ) be the uniformly continuous and bounded uniformly continuous functions on a metric space ( M , d ) ( respectively ) . We equip all of these with the uniform norm i.e . |||f |||A = supa∈A |f ( a ) | – we reserve ‖∗‖ for the Euclidean norm . This makes them all normed spaces , with B ( A ) , Cb ( X ) and Ub ( M ) additionally being Banach spaces . Moreover , if X is compact and ( M , d ) has compact metric completion , then C ( X ) = Cb ( X ) and U ( M ) = Ub ( M ) and hence are also Banach spaces . For background see Rudin ( 2006 ) . If given an injective map i : A→ X , then we say that ϕ : A→ R ( uniquely ) continuously extends to X if there is a ( unique ) ϕ̃ ∈ C ( X ) such that ϕ̃ ◦ i = ϕ . We say a family of funcntions N on A ( uniquely ) continuously extends to X if every ϕ ∈ N ( uniquely ) continuously extends to X . We will make use of the following lemma which is proved in the appendix . Lemma 2.1 . LetN ⊆ B ( D ) where D is a dense subset of a compact metric space ( X , d ) . Suppose N has a continuous extension to X denoted by N ′ ⊆ C ( X ) which is dense . Then the uniform closure ofN in B ( D ) is r ( C ( X ) ) = U ( D ) where r : C ( X ) → Cb ( D ) is the domain restriction map . Letting D = F ( Ω ) , this lemma suggest the following plan of attack : find a compact metric space ( X , d ) in which we can realize F ( Ω ) as a dense subset and hope that our class of neural networks N continuously extends to a dense subset of C ( X ) . If we can do that , then we know the uniform closure of our class of neural networks are precisely the uniformly continuous functions on F ( Ω ) with respect to the metric inherited from X . This motivates the next subsection . 2.3 METRICS ON THE SPACE OF POINT CLOUDS . From now on we will assume ( Ω , d ) is a compact metric space and when Ω ⊆ Rn it will be compact and equipped with the Euclidean metric . Let K ( Ω ) denote the set of all compact subsets of Ω and P ( Ω ) denote the set of all Borel probability measures on Ω . The Hausdorff metric dH ( Munkres , 2000 ) is a natural metric for K ( Ω ) and 1-Wasserstein metric dW ( Villani , 2009 ) ( also called the Earth-mover distance ) is a natural metric for P ( Ω ) . With these metrics , K ( Ω ) and P ( Ω ) become compact metric spaces of their own . From now on we will assume these two spaces are always equipped with the aforementioned metrics . We also briefly mention M ( Ω ) the Banach space of finite signed regular Borel measures on Ω . By the Riesz-Markov theorem it is the topological dual space of C ( Ω ) . Of interest to us is that P ( Ω ) ⊆M ( Ω ) and that the weak- * topology on P ( Ω ) coincides with the topology induced by dW . This means that dW ( µn , µ ) → 0 iff ∫ f dµn → ∫ f dµ for all f ∈ C ( Ω ) . Next , note that F ( Ω ) ⊆ K ( Ω ) and let iK denote the natural inclusion map . We can also define an injective map iP : F ( Ω ) → P ( Ω ) by mapping A ∈ F ( Ω ) to its associated empirical measure iP ( A ) = µA = 1 |A| ∑ a∈A δa ∈ P ( Ω ) where δa is the Dirac delta measure supported at a . The injective maps iK and iP allow us to induce the dH and dW metrics on F ( Ω ) . We will denote the metrized versions by FH ( Ω ) and FW ( Ω ) respectively and use the same convention for FkH ( Ω ) and FkW ( Ω ) . Another important fact to know is that iK and iP embed F ( Ω ) as dense subset of K ( Ω ) and P ( Ω ) . The former follows from compactness of the members of K ( Ω ) and to see why the latter is true see Fournier & Guillin ( 2015 ) ; Villani ( 2009 ) For f ∈ C ( Ω ) and b ∈ R , define Maxf : K ( Ω ) → R and Avef , b : P ( Ω ) → R as the functions Maxf ( K ) = maxx∈K f ( x ) and Avef , b ( µ ) = b+ ∫ Ω f dµ . Lemma 2.2 . Let ( Ω , d ) be compact , f ∈ C ( Ω ) , and b ∈ R. Then Maxf ∈ C ( K ( Ω ) ) and Avef , b ∈ C ( P ( Ω ) ) and Maxf ◦iK = maxf and Avef , b ◦iP = avef , b . As a consequence , PointNet and DeepSets are uniformly continuous on FH ( Ω ) and FW ( Ω ) respectively . This lemma ( proved in the appendix ) tells us that the max neurons and biased-averaging neurons continuously extend to K ( Ω ) and P ( Ω ) and hence so do PointNet and DeepSets architectures ( since we merely compose with the continuous ρ after ) . Thus , we wil be able analyze such architectures as continuous functions on compact metric spaces , which is mathematically a much nicer problem than studying them as set-theoretic functions on an un-metrized F ( Ω ) .

Summary: This work examines the fundamental properties of two popular architectures -- PointNet and DeepSets -- for processing point clouds (and other unordered sets). The authors provide a new universal approximation theorem on real-valued functions that doesn't require the assumption of a fixed cardinality of the input set. They further provide examples of functions that can't be mutually approximated by PointNets and DeepSets.