Active Learning of Convex Halfspaces on Graphs

1 Introduction . We systematically characterise active learning of geodesically convex halfspaces on graphs . While unlabelled graphs such as social networks are readily accessible , obtaining many labels is tedious and labour intensive . Active learning reduces the labelling effort by iteratively selecting informative data points to be labelled . Graph-based active learning , in particular , has been successfully applied on communication [ Regol et al. , 2020 ] and protein-protein-interaction networks [ Vazquez et al. , 2003 , Xiong et al. , 2014 ] . While most previous approaches assume that the number of edges with differently labelled vertices , the cut size , is small [ Afshani et al. , 2007 , Guillory and Bilmes , 2009 , Dasarathy et al. , 2015 ] , we take a different approach and instead assume geodesic convexity : For every pair of vertices with the same label it holds that every vertex on every shortest path between them also has that label . This assumption is used , for example , by biologists on gene similarity networks [ Zhou et al. , 2002 ] and cancer-related protein-protein-interaction networks [ Li et al. , 2012 , 2013 ] . It also typically holds for connected subgraphs of collaboration networks [ Marc and Šubelj , 2018 , Šubelj et al. , 2019 ] and our preliminary experiments confirmed that many communities in community detection datasets are convex . We derive bounds on the number of queries required to determine all labels , that is , the query complexity , using concepts from convexity theory and geodesic convexity spaces , which naturally generalise the regular Euclidean convexity . While learning convex sets and halfspaces in Euclidean spaces has been intensively investigated in the machine learning community for over half a century , geodesic convexity on graphs is understudied . For a concise presentation of our theoretical results in this paper , we concentrate on active learning of halfspaces on graphs , that is , the vertex set is partitioned into two classes which are both geodesically convex . After introducing the concepts of graph and abstract convexity theory in Section 2 , we derive general label-independent upper and lower bounds on the query complexity . We discuss that such general bounds can be loose on specific graphs and derive tighter lower bounds along the separation axioms of abstract convexity theory in Section 3 . This allows us to substantially reduce the gap between our upper and lower bounds . Note that Euclidean spaces can not be structured in a similar way : while the separation axioms partition the space of all graphs into non-empty sets , all separation axioms 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hold in all real vector spaces including the Euclidean spaces . We discuss related topics like previous active learning bounds that depend on the cut size between the classes , active learning of halfspaces in Euclidean spaces , non-active learning of geodesically convex halfspaces , and other related learning theoretic concepts in Section 4 . Notably , we provide a bound on the VC dimension of geodesically convex halfspaces given by the Radon number . After presenting an empirical validation of our assumptions and approach in Section 5 , we conclude with a discussion of future work . 2 Convexity spaces . In this section , we introduce the required concepts of convexity theory . For a more thorough introduction on convexity theory we refer the reader to van de Vel [ 1993 ] and Pelayo [ 2013 ] . For a set X and a family C ⊆ 2X of subsets , the pair ( X , C ) is a convexity space if ( i ) ∅ , X ∈ C , ( ii ) C is closed under intersection , and ( iii ) C is closed under unions of sets totally ordered by inclusion . For finite set systems , property ( iii ) always holds . Any set in C is called convex . If a set C and its complement X \ C are convex , both are called halfspaces . Two disjoint sets A , B are halfspace separable if there exists a halfspace C such that A ⊆ C and B ⊆ X \C . A mapping σ : 2X → 2X is a convex hull ( or closure ) operator if for all A , B ⊆ X with A ⊆ B ( i ) σ ( ∅ ) = ∅ , ( ii ) σ ( A ) ⊆ σ ( B ) , ( iii ) A ⊆ σ ( A ) , and ( iv ) σ ( σ ( A ) ) = σ ( A ) . Any convexity space ( X , C ) induces a convex hull operator by σ ( A ) = ⋂ { C | A ⊆ C ∈ C } . A set A ⊆ X is convex , that is A ∈ C , if and only if is equal to its convex hull , A = σ ( A ) . A set H ⊆ X is a hull set if its convex hull is the whole space , σ ( H ) = X . For A , B ⊆ X , the set A/B = { x ∈ X | A ∩ σ ( B ∪ { x } ) 6= ∅ } is the extension of A away from B . For a , b ∈ X , the extension { a } / { b } is also called a ray a/b . Two disjoint sets A1 , A2 form a partition of A ⊆ X if A1 ∪ A2 = A . The partition A1 , A2 of A is a Radon partition if σ ( A1 ) ∩ σ ( A2 ) 6= ∅ . The Radon number is the minimum number r such that any subset of X of size r has a Radon partition . A particular type of convexity is an interval convexity . Apart from the convex hull σ ( · ) it has an interval mapping I : X×X → 2X such that for all x , y ∈ X , ( i ) x , y ∈ I ( x , y ) and ( ii ) I ( x , y ) = I ( y , x ) . I ( x , y ) is the interval between x and y . We denote I ( A ) = ⋃ a , b∈A I ( a , b ) . A set C in an interval convexity space is convex if and only if C = I ( C ) . The convex hull is given by σ ( A ) = ⋃∞ k=1 I k ( A ) , where I1 ( · ) = I ( · ) and Ik+1 ( · ) = I ( Ik ( · ) ) , A well-known instance of interval convexity spaces are metric spaces ( X , d ) . There , the interval contains all the points for which the triangle inequality holds with equality : Id ( x , y ) = { z ∈ X | d ( x , y ) = d ( x , z ) + d ( z , y ) } . In Euclidean space this corresponds to all points on a line segment and leads to the classical notion of convex sets . We study metric spaces induced by graphs . The geodesic convexity ( or shortest path convexity ) of a connected graph G = ( V , E ) is given by the interval mapping Id , where d : V 2 → R is the shortest path distance in G. Let x , y ∈ V . For unweighted graphs d ( x , y ) is the minimum number of edges on any x-y-path and for graphs with edge weights , w : E → R > 0 , it is the minimum total edge weight of any x-y-path . A set of vertices C ⊆ V is , thus , ( geodesically ) convex if and only if C contains every vertex on every shortest path joining vertices in C , corresponding again to the Euclidean case . We denote by V ( G ) the vertex set of a graph G , the Radon number of the induced geodesic convexity space of a graph as r ( G ) , and the size of the minimum hull set as h ( G ) . Rays a/b in graphs can be defined using the shortest path distance d : In unweighted graphs a/b = { v ∈ V | d ( a , v ) < d ( b , v ) } and in weighted graphs a/b = { v ∈ V | d ( b , v ) = d ( b , a ) + d ( a , v ) } . A vertex v is extreme if { v } forms a halfspace , that is , V \ { v } is convex . We denote the set of extreme vertices of G as Ext ( G ) . In unweighted graphs , v is extreme if and only if its neighbours form a clique . The diameter d ( G ) of a weighted or unweighted graph G is the maximum number of edges in any shortest path in G . 3 Active learning halfspaces on graphs . Having introduced the necessary concepts from convexity theory , we now present the main theoretical results of the paper . In active node classification , a graph G = ( V , E ) with unknown vertex labels λ : V → { 0 , 1 } is given and the goal is to accurately predict all labels using as few as possible vertex queries . Edges can be weighted or unweighted . The learner queries the vertices one by one : it iteratively selects vertices v ∈ V and receives their label λ ( v ) . In this paper , we consider active learning on an undirected connected graph where the vertices with same label form geodesically convex sets . The classes are , thus , halfspaces and we call the labelling halfspace separable . Our results in this section are upper and lower bounds on the number of queries required to deduce all labels λ of the graph . We denote the number of queries required to identify any halfspace separable labelling in the worst-case , the query complexity , by qc ( G ) . We derive two general upper bounds ( Proposition 1 , Theorem 5 ) and one lower bound ( Theorem 7 ) on qc ( G ) that hold for any halfspace separable labelling . To reduce the gap between the upper and lower bounds ( Proposition 8 ) , we derive increasingly tighter lower bounds along separation axioms ( Theorem 10 ) . In this section , we consider only undirected and connected graphs . Full proofs and a discussion of the multiclass , directed , and disconnected setting are in the supplementary material . 3.1 Upper bounds on the query complexity . To derive a simple upper bound , we note that one immediate consequence of the halfspace assumption is that any shortest path P can have at most one cut edge , that is , an edge with differently labelled endpoints . To deduce all labels of P , we first query its endpoints . If they have the same label , we know that all vertices on P have this label . Otherwise , we find the cut edge by binary search with at most dlog d ( G ) e queries , as the length of P is at most the diameter |V ( P ) | − 1 ≤ d ( G ) . Here , log is the base 2 logarithm . We can generalise this approach to the whole graph using shortest path covers [ Thiessen and Gärtner , 2020 ] , which is a set S of shortest paths whose vertices cover the graph : ⋃ P∈S V ( P ) = V ( G ) . Performing binary search on each path in S gives our first upper bound . Proposition 1 . For any weighted graph G with minimum shortest path cover S∗ the query complexity can be bounded as qc ( G ) ≤ |S∗| ( 2 + dlog d ( G ) e ) . By considering a complete graph with edge weights such that the position of the cut edges on all shortest path in the minimum shortest path cover can be chosen independently of each other , we can show that this bound is tight : Proposition 2 . For any ` , s ∈ N , there exists a weighted graph G with diameter d ( G ) = ` and minimum shortest path cover S∗ of size s such that qc ( G ) ≥ |S∗| log d ( G ) . However , the graph in Proposition 2 is an artificial worst-case example . For most graphs , we can do much better as the labels of vertices on different paths in the cover can typically not be chosen independently of each other . Consider a cubic hypergrid graph G , that is , the Cartesian product of k paths with ` vertices each . More than |V ( G ) |/d ( G ) + 1 ≥ ` k/k ` shortest paths are needed to cover G but 2 + dlog ( k ` ) e queries suffice to identify all labels , as fixing one cut edge determines all others . Our main idea to improve the bound and derive a better algorithm for active learning of geodesically convex halfspaces is to deduce additional labels using convex hulls and extensions after each query . Algorithm 1 realises this idea . It first queries the vertices of a hull set ( line 1 ) . If all labels in the hull set are the same , we conclude that all vertices in the graph have the same label ( line 2-3 ) . Otherwise , it performs binary search on a shortest path between two vertices with different labels to identify a cut edge { a , b } ( line 4-5 ) . Finally , Algorithm 1 queries the remaining vertices . It initialises the sets A and B with the convex hull of the rays a/b and b/a , respectively , corresponding to vertices with already known labels ( line 6 ) . In each iteration of the main loop , the algorithm queries a vertex v in the set Ŵ=ab = V ( G ) \ ( σ ( a/b ) ∪ σ ( b/a ) ) , consisting of all vertices whose labels we can not directly deduce through convex hulls and extensions , and updates the sets A and B using the new vertex ( line 7-12 ) . The number of queries spend in the first five lines are summarised in Lemma 3 . Lemma 3 . Let G be a weighted graph with halfspace separable labels . Using h ( G ) + dlog d ( G ) e queries , we can either find a cut edge or determine that all vertices of the graph have the same label . Proof . The convex hull of any minimum hull set is the whole graph . If all vertices in a hull set have the same label , then all vertices in the graph have this label . If not , we can take two vertices with different labels and find a cut edge on a shortest path between them with dlog d ( G ) e queries . Lemma 4 shows that the initialisation and updates of A and B in Algorithm 1 are indeed valid . Lemma 4 . Let G be a weighted graph with halfspace separable labels given by a halfspace C and let A ⊆ C and B ⊆ V ( G ) \ C. It holds that σ ( A/B ) ⊆ C. Proof sketch . A/B ⊆ C as A ∩ σ ( B ∪ { x } ) ⊆ C ∩ ( V ( G ) \ C ) = ∅ for all x ∈ V ( G ) \ C. This implies σ ( A/B ) = ⋂ { C ′ | A/B ⊆ C ′ , C ′ convex } ⊆ C by definition . Algorithm 1 : Ray-based Active Halfspace Learning on Graphs Input : graph G , oracle access to labels λ Output : halfspaces corresponding to both classes 1 Compute a hull set H and query its vertices 2 if ∀h , h′ ∈ H : λ ( h ) = λ ( h′ ) then 3 return ( ∅ , V ( G ) ) 4 Choose h , h′ ∈ H such that λ ( h ) 6= λ ( h′ ) 5 Perform binary search queries on any shortest h-h′-path to find a cut edge { a , b } 6 A : = σ ( a/b ) , B : = σ ( b/a ) 7 while A ∪B 6= V ( G ) do 8 query any vertex v ∈ V ( G ) \ ( A ∪B ) 9 if λ ( v ) = λ ( a ) then 10 A : = σ ( ( A ∪ { v } ) /B ) , B : = σ ( B/ ( A ∪ { v } ) ) 11 else 12 A : = σ ( A/ ( B ∪ { v } ) ) , B : = σ ( ( B ∪ { v } ) /A ) 13 return ( A , B ) We can bound the remaining number of vertices queried by Algorithm 1 using the following set W ∗=ab = { v ∈ Ŵ=ab | @ w ∈ Ŵ=ab \ { v } such that v ∈ ( w/a ) ∩ ( w/b ) } as it is enough to know the labels of the vertices in W ∗=ab to deduce all labels of Ŵ=ab . Theorem 5 shows that Algorithm 1 is correct and summarises our main upper bound . Theorem 5 . Let G be a weighted graph . For the query complexity it holds that : qc ( G ) ≤ h ( G ) + dlog d ( G ) e+ max { a , b } ∈E ( G ) |W ∗=ab| . Proof sketch . By Lemma 3 , we can use h ( G ) + dlog d ( G ) e queries to find the first cut edge { a , b } . Every time Algorithm 1 queries a vertex v ∈ Ŵ=ab \W ∗=ab there is a w ∈W ∗=ab by definition such that v ∈ ( w/a ) ∩ ( w/b ) . It holds that w /∈ A∪B , as otherwise v ∈ ( w/a ) ∩ ( w/b ) ⊆ A∪B , which contradicts line 8 of Algorithm 1 , v ∈ V ( G ) \ ( A ∪ B ) . Thus , in each iteration at least one new vertex from W ∗=ab will be added to A or B . This bounds the number of iterations by |W ∗=ab| . Comparing our two upper bounds , we see that Theorem 5 is preferable to Proposition 1 , as long as |W ∗=ab| is small . To see this , note that the endpoints of paths in a minimum shortest path cover S∗ form a hull set , implying h ( G ) ≤ 2|S∗| . If |W ∗=ab| is small , the bound of Theorem 5 is additive in the dominating terms h ( G ) + dlog d ( G ) e ≤ 2|S∗|+ dlog d ( G ) e , not multiplicative as in Proposition 1 . To state the bound for the unweighted case in more common parameters , we relate |W ∗=ab| to the well-studied treewidth tw ( G ) [ Bodlaender , 1996 ] , which measures the ‘ tree-likeness ’ of a graph . It is known , that the largest k ∈ N , such that the complete bipartite graph K2 , k is a minor of a graph G , is at most twice the treewidth [ Bodlaender et al. , 1997 ] : k ≤ 2 tw ( G ) . This bounds the size |W ∗=ab| ≤ 2 tw ( G ) for any edge { a , b } , as the vertices { a , b } ∪W ∗=ab form a minor K2 , |W∗=ab| . Consequently , graphs with small treewidth like molecules [ Horváth and Ramon , 2010 ] and infrastructure-based networks [ Maniu et al. , 2019 ] have small W ∗=ab . Corollary 6 summarises this result . Corollary 6 . Let G be an unweighted graph . For the query complexity it holds that qc ( G ) ≤ h ( G ) + dlog d ( G ) e+ 2 tw ( G ) .

Summary: The paper studies the problem of active learning on graphs to identify two classes. The main assumption is that the vertices of the same class form a geodesically convex set, and the two classes are halfspace separable.  The authors propose a simple halfspace querying algorithm, and analyzes the upper and lower bounds on the number of queries required to deduce all labels of the graph for binary classification on connected undirected graphs. In particular this number of queries is related to the graph properties and the separation condition between two classes. Furthermore the VC dimension of the hypothesis class is related to the aforementioned properties.