Limitations for Learning from Point Clouds

In this paper we prove new universal approximation theorems for deep learning on point clouds that do not assume fixed cardinality . We do this by first generalizing the classical universal approximation theorem to general compact Hausdorff spaces and then applying this to the permutation-invariant architectures presented in PointNet ( Qi et al ) and Deep Sets ( Zaheer et al ) . Moreover , though both architectures operate on the same domain , we show that the constant functions are the only functions they can mutually uniformly approximate . In particular , DeepSets architectures can not uniformly approximate the diameter function but can uniformly approximate the center-of-mass function but it is the other way around for PointNet . Additionally , even when the point clouds are limited to at most k points , PointNet can not uniformly approximate center-of-mass and we obtain explicit error bounds and a method to produce geometrically derived adversarial examples . 1 INTRODUCTION . Recently , architectures proposed in PointNet ( Qi et al. , 2017 ) and Deep Sets ( Zaheer et al. , 2017 ) have allowed for the direct processing of point clouds within a deep learning framework . These methods produce outputs that are permutation-invariant with respect to the member points and work for point clouds of arbitrarily large cardinality . A common source of such data is LIDAR measurements from autonomous vehicles . Zaheer et al . ( 2017 ) also presents a permutation-equivariant architecture which we do not discuss here . Each of of these works provide their own universal approximation theorem ( UAT ) to support the empirical success of their architectures . However , both results assume the cardinality of the point cloud is fixed to some size n. In this work we refine these results , remove the cardinality limitation , use weaker architecture assumptions , and arrive at three main results which can be summarized roughly as follows ( assuming unrestricted finite cardinality for the input point clouds ) : 1 ) PointNet ( DeepSets ) architectures can uniformly approximate real-valued functions that are uniformly continuous with respect to the Hausdorff ( Wasserstein ) metric and nothing else ( Theorem 3.4 ) . 2 ) Only the constant functions can be uniformly approximated by both architectures . In particular , PointNet architectures can uniformly approximate the diameter function but DeepSets architectures can not . Conversely , DeepSets architectures can uniformly approximate the center-of-mass function but PointNet architectures can not ( Theorem 4.1 ) . 3 ) We prove explicit error lower bounds and produce adversarial examples to show that even when limited to point clouds of size k , PointNet can not uniformly approximate center-ofmass ( Theorem 4.2 ) . To do this we extend the many universal approximation results for feed-forward networks ( Cybenko , 1989 ; Hornik et al. , 1989 ; Leshno et al. , 1993 ; Stinchcombe , 1999 ) to the abstract setting of general compact Hausdorff spaces . We then find appropriate compact metric spaces over which PointNet and DeepSets architectures can be easily analyzed and then finally we observe the resulting consequences in the original setting of interest , i.e . point clouds . 2 PRELIMINARIES . 2.1 POINTNET AND DEEPSETS ARCHITECTURES . In practice , the implementations of the architectures presented in PointNet and Deep Sets can involve many additional tricks , but the essential ideas are quite simple . We do however make a small modification to the Deep Sets model . For A ⊆ Rn of cardinality |A| < ∞ , we have Qi et al . ( 2017 ) and Zaheer et al . ( 2017 ) suggesting scalar-output neural networks of the form FPN ( A ) = ρ ( max a∈A ϕ ( a ) ) , and FDS ( A ) = ρ ( b+ 1 |A| ∑ a∈A ϕ ( a ) ) , respectively . Here ϕ : Rn → Rm creates features for each point in A , then a symmetric operation is applied , and then ρ : Rm → R combines these features into a scalar output ( here max is the component-wise maximum ) . In practice , we need both ρ and ϕ to be neural networks . Note that because we use a symmetric operation before ρ , the output will not depend on the ordering of points in the point cloud , and because the max and sum operations scale to arbitrary finite cardinalities the size of the point cloud is not an issue . The original model in Deep Sets did not have a bias term b and used a sum instead of the averaging we use here . This change will help us later in our theoretical analysis . It will help to introduce some simplifying notation . Let F ( Ω ) denote the set of all nonempty finite subsets of a set Ω ( i.e . point clouds in Ω ) , F≤k ( Ω ) the set of nonepmty subsets of size ≤ k , and Fk ( Ω ) the set of k-point subsets . Now consider Ω ⊆ RN and define maxf , avef , b : F ( Ω ) → R which are given by maxf ( A ) = maxa∈A f ( a ) and avef , b ( A ) = b + 1|A| ∑ a∈A f ( a ) respectively . We make sense of this in the natural way if we use vector-valued f and b by operating componentwise . We call these operations max neurons and biased-averaging neurons respectively . Once again letting ρ and ϕ be neural networks , FPN = ρ ◦ maxϕ and FDS = ρ ◦ aveϕ , b will be the general form of what we call the PointNet and DeepSets architectures ( resp . ) in this paper . Some natural questions are 1 ) is there a topology forF ( Ω ) that makes these architectures continuous , 2 ) how expressive are these approaches , and 3 ) how deep is deep enough for function approximation ? 2.2 FUNCTION SPACES AND UNIFORM APPROXIMATION . From now on , we only consider R-valued functions unless otherwise stated . Let B ( A ) be the set of bounded functions on a set A , let C ( X ) and Cb ( X ) be the set of continuous and bounded continuous functions on a topological space X ( respectively ) , and let U ( M ) and Ub ( M ) be the uniformly continuous and bounded uniformly continuous functions on a metric space ( M , d ) ( respectively ) . We equip all of these with the uniform norm i.e . |||f |||A = supa∈A |f ( a ) | – we reserve ‖∗‖ for the Euclidean norm . This makes them all normed spaces , with B ( A ) , Cb ( X ) and Ub ( M ) additionally being Banach spaces . Moreover , if X is compact and ( M , d ) has compact metric completion , then C ( X ) = Cb ( X ) and U ( M ) = Ub ( M ) and hence are also Banach spaces . For background see Rudin ( 2006 ) . If given an injective map i : A→ X , then we say that ϕ : A→ R ( uniquely ) continuously extends to X if there is a ( unique ) ϕ̃ ∈ C ( X ) such that ϕ̃ ◦ i = ϕ . We say a family of funcntions N on A ( uniquely ) continuously extends to X if every ϕ ∈ N ( uniquely ) continuously extends to X . We will make use of the following lemma which is proved in the appendix . Lemma 2.1 . LetN ⊆ B ( D ) where D is a dense subset of a compact metric space ( X , d ) . Suppose N has a continuous extension to X denoted by N ′ ⊆ C ( X ) which is dense . Then the uniform closure ofN in B ( D ) is r ( C ( X ) ) = U ( D ) where r : C ( X ) → Cb ( D ) is the domain restriction map . Letting D = F ( Ω ) , this lemma suggest the following plan of attack : find a compact metric space ( X , d ) in which we can realize F ( Ω ) as a dense subset and hope that our class of neural networks N continuously extends to a dense subset of C ( X ) . If we can do that , then we know the uniform closure of our class of neural networks are precisely the uniformly continuous functions on F ( Ω ) with respect to the metric inherited from X . This motivates the next subsection . 2.3 METRICS ON THE SPACE OF POINT CLOUDS . From now on we will assume ( Ω , d ) is a compact metric space and when Ω ⊆ Rn it will be compact and equipped with the Euclidean metric . Let K ( Ω ) denote the set of all compact subsets of Ω and P ( Ω ) denote the set of all Borel probability measures on Ω . The Hausdorff metric dH ( Munkres , 2000 ) is a natural metric for K ( Ω ) and 1-Wasserstein metric dW ( Villani , 2009 ) ( also called the Earth-mover distance ) is a natural metric for P ( Ω ) . With these metrics , K ( Ω ) and P ( Ω ) become compact metric spaces of their own . From now on we will assume these two spaces are always equipped with the aforementioned metrics . We also briefly mention M ( Ω ) the Banach space of finite signed regular Borel measures on Ω . By the Riesz-Markov theorem it is the topological dual space of C ( Ω ) . Of interest to us is that P ( Ω ) ⊆M ( Ω ) and that the weak- * topology on P ( Ω ) coincides with the topology induced by dW . This means that dW ( µn , µ ) → 0 iff ∫ f dµn → ∫ f dµ for all f ∈ C ( Ω ) . Next , note that F ( Ω ) ⊆ K ( Ω ) and let iK denote the natural inclusion map . We can also define an injective map iP : F ( Ω ) → P ( Ω ) by mapping A ∈ F ( Ω ) to its associated empirical measure iP ( A ) = µA = 1 |A| ∑ a∈A δa ∈ P ( Ω ) where δa is the Dirac delta measure supported at a . The injective maps iK and iP allow us to induce the dH and dW metrics on F ( Ω ) . We will denote the metrized versions by FH ( Ω ) and FW ( Ω ) respectively and use the same convention for FkH ( Ω ) and FkW ( Ω ) . Another important fact to know is that iK and iP embed F ( Ω ) as dense subset of K ( Ω ) and P ( Ω ) . The former follows from compactness of the members of K ( Ω ) and to see why the latter is true see Fournier & Guillin ( 2015 ) ; Villani ( 2009 ) For f ∈ C ( Ω ) and b ∈ R , define Maxf : K ( Ω ) → R and Avef , b : P ( Ω ) → R as the functions Maxf ( K ) = maxx∈K f ( x ) and Avef , b ( µ ) = b+ ∫ Ω f dµ . Lemma 2.2 . Let ( Ω , d ) be compact , f ∈ C ( Ω ) , and b ∈ R. Then Maxf ∈ C ( K ( Ω ) ) and Avef , b ∈ C ( P ( Ω ) ) and Maxf ◦iK = maxf and Avef , b ◦iP = avef , b . As a consequence , PointNet and DeepSets are uniformly continuous on FH ( Ω ) and FW ( Ω ) respectively . This lemma ( proved in the appendix ) tells us that the max neurons and biased-averaging neurons continuously extend to K ( Ω ) and P ( Ω ) and hence so do PointNet and DeepSets architectures ( since we merely compose with the continuous ρ after ) . Thus , we wil be able analyze such architectures as continuous functions on compact metric spaces , which is mathematically a much nicer problem than studying them as set-theoretic functions on an un-metrized F ( Ω ) .

Summary: PointNet (Qi et al, 2017) and Deep sets (Zaheer et al, 2017) have allowed to use deep architectures that deal with point clouds as inputs, taking into account the invariance in the ordering of points. However, existing results on their approximation abilities are limited to fixed cardinalities. This paper removes the cardinality limitation and gives two kinds of results: