Neural Message Passing for Multi-Label Classification

1 INTRODUCTION . Multi-label classification ( MLC ) is receiving increasing attention in tasks such as text categorization and image classification . Accurate and scalable MLC methods are in urgent need for applications like assigning topics to web articles , classifying objects in an image , or identifying binding proteins on DNA . The most common and straightforward MLC method is the binary relevance ( BR ) approach that considers multiple target labels independently ( Tsoumakas and Katakis , 2006 ) . However , in many MLC tasks there is a clear dependency structure among labels , which BR methods ignore . Accordingly , probabilistic classifier chain ( PCC ) models were proposed to model label dependencies and formulate MLC in an autoregressive sequential prediction manner ( Read et al. , 2009 ) . One notable work in the PCC category was from Nam et al . ( 2017 ) which implemented a classifier chain using a recurrent neural network ( RNN ) based sequence to sequence ( Seq2Seq ) architecture , Seq2Seq MLC . This model uses an encoder RNN encoding elements of an input sequence , a decoder RNN predicting output labels one after another , and beam search that computes the probability of the next T predictions of labels and then chooses the proposal with the max combined probability . However , the main drawback of classifier chain models is that their inherently sequential nature precludes parallelization during training and inference . This can be detrimental when there are a large number of positive labels as the classifier chain has to sequentially predict each label , and often requires beam search to obtain the optimal set . Aside from time-cost disadvantages , PCC methods have several other drawbacks . First , PCC methods require a defined ordering of labels for the sequential prediction , but MLC output labels are an unordered set , and the chosen order can lead to prediction instability ( Nam et al. , 2017 ) . Secondly , even if the optimal ordering is known , PCC methods struggle to accurately capture long-range dependencies among labels in cases where the number of positive labels is large ( i.e. , dense labels ) . For example , the Delicious dataset has a median of 19 positive labels per sample , so it can be difficult to correctly predict the labels at the end of the prediction chain . Lastly , many real-world applications prefer interpretable predictors . For instance , in the task of predicting which proteins ( labels ) will bind to a DNA sequence based binding site , users care about how a prediction is made and how the interactions among labels influence the predictions1 . 1An important task is modelling what is known as “ co-binding ” effects , where one protein will only bind if there is another specific protein already binding , or similarly will not bind if there is another already binding . Message Passing Neural Networks ( MPNNs ) ( Gilmer et al. , 2017 ) introduce a class of methods that model joint dependencies of variables using neural message passing rather than an explicit representation such as a probabilistic classifier chain . Message passing allows for efficient inference by modelling conditional independence where the same local update procedure is applied iteratively to propagate information across variables . MPNNs provide a flexible method for modeling multiple variables jointly which have no explicit ordering ( and can be modified to incorporate an order , as explained in section 3 ) . To handle the drawbacks of BR and PCC methods , we propose a modified version of MPNNs for MLC by modeling interactions between labels using neural message passing . We introduce Message Passing Encoder-Decoder ( MPED ) Networks aiming to provide fast , accurate , and interpretable multi-label predictions . The key idea is to replace RNNs and to rely on neural message passing entirely to draw global dependencies between input components , between labels and input components , and between labels . The proposed MPED networks allow for significantly more parallelization in training and testing . The main contributions of this paper are : • Novel approach for MLC . To the authors ’ best knowledge , MPED is the first work using neural message passing for MLC . • Accurate MLC . Our model achieves similar , or better performance compared to the previous state of the art across five different MLC metrics . We validate our model on seven MLC datasets which cover a wide spectrum of input data structure : sequences ( English text , DNA ) , tabular ( bag-of-words ) , and graph ( drug molecules ) , as well as output label structure : unknown and graph . • Fast . Empirically our model achieves an average 1.7x speedup over the autoregressive seq2seq MLC at training time and an average 5x speedup over its testing time . • Interpretable . Although deep-learning based systems have widely been viewed as “ black boxes ” due to their complexity , our attention based MPED models provide a straightforward way to explain label to label , input to label , and feature to feature dependencies . 2 METHOD : MPED NETWORKS . 2.1 BACKGROUND : MESSAGE PASSING NEURAL NETWORKS . Message Passing Neural Networks ( MPNNs ) ( Gilmer et al. , 2017 ) are a generalization of graph neural networks ( GNNs ) ( Gori et al. , 2005 ; Scarselli et al. , 2009 ) , where variables are represented as nodes on a graph G and joint dependencies are modelled using message passing rather than explicit representations , which allows for efficient inference . MPNNs model the joint dependencies using message function M t and node update function U t for T time steps , where t is the current time step . The hidden state vti ∈ Rd of node i ∈ G is updated based on messages mti from its neighboring nodes { vtj∈N ( i ) } defined by neighborhood N ( i ) : mti = ∑ j∈N ( i ) M t ( vti , v t j ) , ( 1 ) vt+1i = U t ( vti , m t i ) . ( 2 ) After T updates , a readout function R is used on the updated nodes for a prediction ( e.g. , node classification or graph classification ) on the graph G. Many possibilities exist for functions M t and U t. For example , one can pass messages using neural attention in which nodes are able to attend over their neighborhoods differentially ( Veličković et al. , 2017 ) . This allows for the network to learn different weights for different nodes in a neighborhood , without depending on knowing the graph structure a priori . In this formulation , messages for node vti are obtained by a weighted sum of all neighboring nodes { vtj∈N ( i ) } where the weights are obtained by attention ( Bahdanau et al. , 2014 ) . In our implementation , we implement neural message passing with attention . In the rest of the paper , we use “ graph attention ” and “ neural message passing ” interchangeably . Neural message passing with attention works as follows . Attention weights αtij for pair of nodes ( v t i , v t j ) are computed using some attention function a ( · ) : etij = a ( v t i , v t j ) , ( 3 ) where etij represents the importance of node j for node i . For our attention function , we used a scaled dot product with node-wise linear transformations Wq ∈ Rd×d on node vti and Wu ∈ Rd×d on node vtj . Scaling by √ d is used to mitigate training issues ( Vaswani et al. , 2017 ) : a ( vti , v t j ) = ( Wqvti ) > ( Wuvtj ) √ d . ( 4 ) Attention coefficients etij are then normalized across all neighboring nodes of node i using a softmax function : αtij = softmaxj ( e t ij ) = α t ij = exp ( etij ) ∑ k∈N ( i ) exp ( e t ik ) . ( 5 ) In our method , we use a so called attention message function M tatn to produce the message from node j to node i using the learned attention weights αtij and another transformation matrix W v ∈ Rd×d . Then we compute the full message mti by linearly combining messages from all nodes j ∈ N ( i ) with a residual connection on the current vti : Matn ( v t i , v t j ; W ) = α t ijW vvtj , ( 6 ) mti = v t i + ∑ j∈N ( i ) Matn ( v t i , v t j ; W ) . ( 7 ) Lastly , node vti is updated to state v t+1 i using message m t i by a multi-layer perceptron ( MLP ) update function Umlp with matrices Wr ∈ R2d×d , Wb ∈ R2d×d , and a residual connection on mti : Umlp ( m t i ; W ) = ReLU ( W rmti + b1 ) > Wb + b2 , ( 8 ) vt+1i = m t i + Umlp ( m t i ; W ) . ( 9 ) It is important to note that matrices W are shared ( i.e. , separately applied ) across all nodes . This can be viewed as 1-dimensional convolution with kernel and stride sizes of 1 . Weight sharing across nodes is a key aspect of MPNNs , where node dependencies are learned in an order-invariant manner . 2.2 MPED : MESSAGE PASSING ENCODER-DECODER NETWORK FOR MLC . Notations : We define the following notations , used throughout the paper . Let D = { ( xn , yn ) } Nn=1 be the set of data samples with inputs x ∈ X and outputs y ∈ Y . Inputs x are a ( possibly ordered ) set of S components { x1 , x2 , ... , xS } , and outputs y are a set of L labels { y1 , y2 , ... , yL } . MLC involves predicting the set of binary labels { y1 , y2 , ... , yL } , yi ∈ { 0 , 1 } given input x . Input features are represented as embedded vectors { ct=01 , ct=02 , ... , ct=0S } , cti ∈ Rd , using embedding matrix Wx ∈ Rδ×d , where d is the embedding size , δ is the vocabulary size , and t represents the ‘ state ’ of the embedding after t updates . Similarly , labels are represented as an embedded vectors { ht=01 , ht=02 , ... , ht=0L } , hti ∈ Rd , using embedding matrix Wy ∈ RL×d , where L is the number of labels , and t represents the ‘ state ’ of the embedding after t updates . Setup : In MLC , each output label is determined by a joint probability of other labels and the input features . Our goal is to achieve the performance of explicit joint probability methods such as PCCs ( Eqs . 22 and 23 ) , at the test speed of BR methods ( Eq . 21 ) . We introduce Message Passing Encoder-Decoder ( MPED ) networks , where we formulate MLC using an encoder-decoder architecture . In MPED Networks , input components are represented as nodes in encoder graph GENC using embedding vectors { ct1 : S } , and labels are represented as nodes in decoder graph GDEC using embedding vectors { ht1 : L } . MPED networks use three MPNN modules with attention to pass messages within GENC , from GENC to GDEC , and within GDEC to model the joint prediction of labels . The first module , MPNNxx , is used to update input component nodes { ct1 : S } by passing messages within the encoder ( between input nodes ) . The second module , MPNNxy , is used to update output label nodes { ht1 : L } by passing messages from the encoder to decoder ( from input nodes { ct1 : S } to output nodes { ht1 : L } ) . The third module , MPNNyy , is used to update output label nodes { ht1 : L } by passing messages within the decoder ( between label nodes ) . Once messages have been passed to update input and label nodes , a readout function R is then used on the label nodes to make a binary classification prediction for each label , { ŷ1 , ŷ2 , ... , ŷL } . An overview of our model is shown in Fig . 1 .

Summary: The paper describes an approach for using graph neural networks (GNN) to perform multi-label classification (MLC). The main idea is to use attentional pooling to project an input graph into a "label graph", whose nodes correspond to labels on some MLC problem. Multiple rounds of self-attention/message-passing hops can be performed on the input graph and label graph. Each output label is binary-valued, and is predicted from its corresponding node in the label graph. They evaluate on 6 multi-label sequence classification datasets, and report strong perform over baselines.