Directed Probabilistic Watershed

1 Introduction . In the last years , machine learning has experienced a great boost thanks to the large quantity of available data and increasing accessibility of less expensive and more powerful processing capacities . However , the acquisition of labeled data is still laborious in some applications like computer-aided diagnosis or drug discovery . Semi-supervised learning is the subfield of machine learning that utilizes both labeled and unlabeled data . It permits exploiting the large amount of unlabeled data available in many use cases jointly with frequently smaller sets of labeled data . When data is encoded as a network , graph-based semi-supervised learning aims to assign a label or class to each of the nodes of a network based on its topology and a given set of labeled nodes / seeds . Graph-based semi-supervised learning has been applied in multiple domains like computer vision [ 6 , 15 , 17 ] , NLP [ 2 , 28 , 31 ] , social networks [ 1 , 45 ] and biomedical science [ 24 ] . Most graph-based semi-supervised learning algorithms focus on undirected weighted graphs [ 5 , 36 ] , though directed graphs appear naturally in many cases like in k-Nearest Neighbors graphs or citation and recommendation networks among others . Instances of methods proposed for directed graphs are [ 9 , 11 , 42 , 48 , 49 ] . Recently , [ 12 ] proposed the transductive graph-based semi-supervised learning algorithm Probabilistic Watershed ( ProbWS ) . It envisages a Gibbs distribution over the labeled nodes separating forests ( forests whose trees contain a unique labeled node ) that span a weighted undirected graph . The probability of each forest is proportional to its weight . By means of the Matrix Tree Theorem [ 19 , 40 ] , the ProbWS computes the probability of sampling a seed separating forest such that a query node is connected to a single labeled node / seed . This approach turns out to be equivalent computationally and by result to the Random Walker / Harmonic energy minimization [ 3 , 15 , 47 , 50 ] . The Matrix Tree Theorem [ 19 , 40 ] has been generalized to directed graphs [ 25 ] . We propose an extension of the ProbWS , the “ Directed Probabilistic Watershed '' ( DProbWS ) , that can be applied 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to directed graphs . Instead of considering seed separating spanning forests , we deal with incoming directed forests rooted at the seed nodes ( see Figure 1 ) . We analyze the DProbWS from a theoretical point of view and generalize the results presented in [ 12 ] to directed graphs . Building on the arguments used in the original ProbWS paper , we • analytically compute the probability that a graph node is assigned to a particular seed in an ensemble of Gibbs distributed incoming directed spanning forests rooted at the seeds , • demonstrate the equivalence of the Directed Probabilistic Watershed and the directed version of the Random Walker presented in [ 15 ] , • propose a natural extension of the Power Watershed [ 6 ] to directed graphs , • run an illustrative experiment to show the empirical performance of the Directed Probabilistic Watershed and compare it with other semi-supervised transductive methods . 1.1 Related work . Semi-supervised learning is an important subfield of machine learning due to is ability to exploit both labeled and unlabeled data [ 41 ] . If data is represented as a graph , graph based semi-supervised learning exploits , in addition to the labeled samples , the topology of the graph to infer the class of the unlabeled nodes . Undirected graphs have received more attention in this field than the directed ones [ 5 , 36 ] . Nonetheless , there have been methods proposed for directed graphs . In [ 48 , 49 ] the labels are inferred by minimizing a regularized function that forces highly connected subgraphs to have the same label . The paper [ 42 ] builds a symmetric pairwise similarity via co-linkage analysis from the directed graph in order to apply algorithms for undirected graphs . The work presented in [ 11 ] proposes a method based on game theory , where the game ’ s Nash equilibrium defines the labeling of the data . Related to our work by the use of absorbing random walks is the one proposed in [ 9 ] . Their algorithm assigns the label of the seed that maximizes the accumulated expected number of visits from the unlabeled query node before being absorbed by an artificially added metanode . We , instead , prove that the DProbWS turns out to assign the label of the seed that maximizes the probability of absorption of a random walker . Our method extends the Probabilistic Watershed ( ProbWS ) [ 12 ] , which , in turn , is based on the Watershed algorithm [ 7 ] . The ProbWS paper established a link between the Watershed [ 7 ] and the Random Walker [ 3 , 15 , 47 , 50 ] . Another such connection was made by the Power Watershed paper [ 6 ] . We extend these relations to the directed graph framework . We smooth the combinatorial minimum spanning in-forest via entropic regularization . More specifically , we consider a Gibbs distribution over all spanning in-forests , as was proposed in the original ProbWS paper [ 12 ] . Entropic regularization has been applied in other situations like the randomized shortest path framework [ 14 , 20 ] or optimal transport [ 8 ] . If applied in conjunction with deep networks , it allows end-to-end training [ 32 ] . The Matrix Tree Theorem ( MTT ) is key to the theory developed in our work . The directed version of the MTT [ 25 ] allows us to compute in closed form the weight of a set of incoming forests . The MTT has been also applied in NLP [ 22 ] , biology [ 38 ] and network analysis [ 37 , 39 ] . A generalization of the MTT , the Matrix Forest Theorem ( MFT ) , was used by [ 4 ] to define a distance between the nodes of a graph . Similar to our approach , [ 34 ] defines a Gibbs distribution over the forests using the MFT . 2 Background . 2.1 Notation and terminology . In this section , we introduce the main definitions and notation used in the paper . Most of them have been borrowed from [ 12 , 25 ] . Let G = ( V , E , w , c ) be a directed graph where V denotes the set of nodes , E the set of edges and w and c are functions that assign a weight w ( e ) ∈ R≥0 and a cost c ( e ) ∈ R to each edge e ∈ E. The edge e = ( i , j ) indicates that there is a directed edge from i to j . If ( i , j ) /∈ E , we set c ( ( i , j ) ) =∞ and w ( ( i , j ) ) = 0 . The set S = { s1 , s2 } ⊂ V will denote the set of seed / labeled nodes and U = V \S the unlabeled ones . In order to ease the exposition we only consider the 2-seed scenario , although the method can be generalized to the n-seed scenario . We will use the terms seed and labeled node interchangeably . Definition 1 . Let t = ( Vt , Et ) be a graph then we say that it is an incoming directed spanning tree rooted at r ∈ Vt ( in-tree for short ) if : 1 ) every vertex u 6= r has one and only one outgoing edge ; 2 ) the root node r does not have any outgoing edges ; 3 ) t does not have directed cycles , i.e. , there does not exist a directed path in t such that the initial and final vertex are the same . Note that condition 3 ) , in conjunction with the other conditions , implies that no cycles ( directed or undirected ) will be formed . Equivalently , one can define an in-tree as a graph in which , for any vertex u ∈ Vt there is exactly one directed path from u to the root r , and the root does not have any out-going edges.1 Note that an in-tree becomes a tree in the classical sense if the directions of the edges are ignored . We say that an in-tree is spanning on G = ( V , E ) if t is a subgraph of G and Vt = V . The set of spanning in-trees of G rooted at r will be denoted by T −→r . Analogously , the set F−→v−→u represents the set of 2-in-trees spanning forests rooted at u and v , i.e. , the spanning graphs of G consisting of two disjoint in-trees , such that u and v are the respective roots of the in-trees . Furthermore , if we consider a third node q , we define F−→v−→u , q ⊆ F −→v−→u , as the set of all 2-in-trees spanning forests where q is connected to u by a directed path . In order to shorten the notation we will refer to 2-in-trees spanning forests simply as 2-in-forests or in-forests . We define the weight of an arbitrary graph ( e.g . in-forests ) ) as the product of the weights of all its edges , w ( G ) = ∏ e∈E w ( e ) . The weight of a set of graphs , w ( { Gi } ni=0 ) is the sum of the weights of the graphsGi . In a similar manner , we define the cost of a graph as the sum of the costs of all its edges , c ( G ) = ∑ e∈E c ( e ) . As in the Probabilistic Watershed [ 12 ] , we consider w ( e ) = exp ( −µc ( e ) ) , µ ≥ 0 , which will be a consequence of the definition of a Gibbs distribution over the 2-in-forests in F−→v−→u . Thus , a low edge-cost corresponds to a large edge-weight , and a minimum edge-cost spanning 1One can define analogously an outgoing tree ( out-tree ) rooted at r as a tree where there is exactly one directed path from the root r to any other node , and the root does not have any incoming edges . in-forest ( mSF ) is equivalent to a maximum edge-weight spanning in-forest ( MSF ) . Depending on the context , the abbreviations mSF and MSF will be also used for the undirected versions of the minimum/maximum spanning forests . Via the Directed Matrix Tree Theorem [ 25 ] , we aim to compute the weight of the set of the 2-in-forests rooted at the seeds . The theorem makes use of the out-Laplacian matrix which we define next . Definition 2 . Given a weighted directed graph G = ( V , E , w ) we define the out-Laplacian of G as L : = D −A > , ( 1 ) where A ∈ R|V |×|V | is the vertex-adjacency matrix of G represented entry-wise as Aij = w ( ( i , j ) ) and D denotes the diagonal matrix defined as Dii = ∑ j∈V Aij , i.e. , Dii is the out-degree of vertex i.2 For brevity , we will refer to the out-Laplacian just as Laplacian . For any v ∈ V , L [ v ] will stand for the Laplacian after removing the row and column indexed by v .

Summary: This paper proposes an extension of a previous graph-based method for semi-supervised learning. Specifically, the proposed method builds upon the probabilistic watershed algorithm, which is a transductive graph-based algorithm for semi-supervised classification, based on building "separating forests" (i.e., in which each tree contains a single labeled node) and defining a Gibbs distribution on this set of forests.  The main contribution of the paper is a generalization of this method to directed graphs, and showing a couple of theoretical results related to the proposed generalization.