On the Importance of Looking at the Manifold

1 INTRODUCTION . The ability to recognize relational information between or even within individual percepts is one of the fundamental differences between human and artificial learning systems . For example , the feature-binding problem ( Roskies , 1999 ) , i.e . the mechanism governing the visual system to represent hierarchical relationships between features in an image , is still largely unsolved by neuroscientists , exacerbating the development of bio-inspired statistical learning systems . Traditional relational learning approaches mostly sort into either learning internal or external relational structure between samples and rely heavily on crafting domain-specific expert knowledge that is engineered into the model ( Struyf & Blockeel , 2010 ) . Consequently , these models have yet to prove their usability in real applications and , although some neurocomputational frameworks for relational learning were proposed ( Isbister et al. , 2018 ) , building statistical models that explore higher-order dependencies between samples remains a key challenge for computer vision and robotics application . Consequently , relational reasoning has been advocated a pivotal role for the future of artificial intelligence ( Battaglia et al. , 2018 ) . On the very contrary , deep learning as a purely data-driven approach has enjoyed remarkable success in recent years by learning complex non-linear functions mapping raw inputs to outputs without explicit dependency modelling . Fields like relational reinforcement learning ( Džeroski et al. , 2001 ) and statistical relational learning ( Koller et al. , 2007 ) aimed to fill this gap ; but recently augmenting deep ( reinforcement ) learning models toward relational reasoning emerged as a promising approach ( Zambaldi et al. , 2018 ; Zhang et al. , 2016 ) . Many successful contributions for relational modelling in images however largely rely on Euclidean spaces ( Dai et al. , 2017 ; Yao et al. , 2018 ) . It is widely agreed that graphs are the ideal structure to enable relational deep learning ( Hamilton et al. , 2017 ) . Prior work has shown that metagraphs incorporating relational information about the dataset can improve unsupervised representation learning in finding less complex models that preserve relational information without loosing expressivity on the original representation ( Dumancic & Blockeel , 2017 ) . In terms of predictive modelling , the relational representations can be superior to ordinary ones ( Dumancic & Blockeel , 2017 ) and graph-induced kernels can aid in improving phenotype prediction compared to non-topological kernels ( Manica et al. , 2019 ) . In generative modelling , relational distribution comparison was demonstrated to facilitate the learning of generative models across incomparable spaces ( Bunne et al. , 2019 ) . Here , we perform an extensive study on the impact of the topological information in learning data representations . Specifically , we focus on the trade-off between leveraging data point features and relational information . We consider a selection of unsupervised models for learning representations lying in different areas of the spectrum . Ranging from Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) to node embedding techniques based on random walks on graphs ( Grover & Leskovec , 2016 ) , passing through graph neural networks ( Veličković et al. , 2018 ) and the proposed Graph-Regularized Variational Autoencoders ( GR-VAE ) , our adaptation of VAEs where the latent space is regularized through a metagraph representing relations between samples of the dataset that we introduce in this work . The methods considered are evaluated on different datasets and downstream tasks where the impact of the topology can be appropriately assessed . Initially , we examine the impact of implicitly accounting for the topology to validate the GR-VAE in two synthetic studies based on topologically connected 4D point clouds and MNIST ( LeCun et al. , 2010 ) with added relational information based on the labels . After this initial validation , we move to evaluating all the methods in the case of text representations and chemical reactions . In the case of text representations , we analyze the methods performance on Cora , CiteSeer , and PubMed ( Sen et al. , 2008 ) , using their citation networks , and evaluating the representations learned in a downstream classification task . Finally , we study the impact of the topology in molecule representations using a chemical reaction dataset ( Jin et al. , 2017 ) , where the downstream task consists in predicting the reactivity of reactant/reagent-product pairs . 2 METHODS . In this section we present the different models compared in this study . Our approach is to explore a spectrum of models with varying availability of features and topology ( see Figure 1 ) . 2.1 IMPLICIT TOPOLOGICAL LEARNING . We first explore VAEs ( Kingma & Welling , 2013 ) which only intake features from the nodes , thus serving as a baseline model agnostic to topological information . Graph Regularized VAE . We then introduce a variation of VAEs ( Kingma & Welling , 2013 ) , defined as Graph-Regularized VAEs ( GR-VAE ) , that augments the trade-off between reconstruction error and KL divergence by a topological constraint ( see Figure 2 for an overview ) . In GR-VAE , the latent space is regularized through a metagraph present in the data . We suggest that accounting for available information on how different samples relate ( henceforth simply referred as the graph ) may help at obtaining more powerful representations , especially where this relationships are directly involved with a downstream task of interest . Our approach adds a term to the loss defined by the set of constraints to the samples ’ representation in the latent space given the distances of the samples ’ metagraph . For a given set of samples , S , we can compute their distances in the latent space D , as well as over the graph G. For each node , ν , we expect the distances to the other nodes , once embedded in D , to resemble the distances over G. Thus , we enforce a constraint aimed to preserve the relative distances in the two spaces . Formally , fixing a node ν and considering any pair of nodes ( i , j ) , we can define the following penalty term : φ ( dD , dG , ν , i , j ) = ( dD ( ν , j ) − dD ( ν , i ) ) + if dG ( ν , i ) > dG ( ν , j ) ( dD ( ν , i ) − dD ( ν , j ) ) 2 if dG ( ν , i ) = dG ( ν , j ) ( dD ( ν , i ) − dD ( ν , j ) ) + if dG ( ν , i ) < dG ( ν , j ) ( 1 ) where dD and dG are metrics defined in the latent space and over the graph respectively . In the following , we select the Euclidean distance as dD and the geodesic distance ( Floyd , 1962 ) as dG . The overall loss function of the GR-VAE thus becomes : LGR-VAE ( X ; θ ) = LVAE ( X ; θ ) + γ ∑ ν∈ S ∑ ( i , j ) ∈ S× S φ ( dD , dG , ν , i , j ) ( 2 ) where X are the features of the samples in S , θ the network parameters and γ ≥ 0 regulates the strength of the penalty . 2.2 EXPLICIT TOPOLOGICAL LEARNING . Notably , GR-VAE is devised to infer topological information solely from a soft constraint , without any architectural requirements such as graph convolutions . On the other side of the spectrum , graph neural networks ( GNN ) instead model topology explicitly . Deep Graph Infomax . Here , we consider to a Deep Graph Infomax ( DGI ) , a state-of-the-art GNN for unsupervised representation learning ( Veličković et al. , 2018 ) . DGI relies on maximizing mutual information between subgraphs ( themselves derived with GCNs ) yielding representations that facilitate downstream node-wise classification tasks . These two models used as a comparison for the text classification task . GAE and VGAE . We consider both non-probabilistic Graph Autoencoders ( GAE ) and Variational Graph Autoencoders ( VGAE ) ( Kipf & Welling , 2016 ) which are models for unsupervised-graph data inspired on the VAE . GraphSAGE . We consider GraphSAGE , an inductive framework that generates node embeddings by sampling and aggregating the features for the local neighborhood of a node ( Hamilton et al. , 2018 ) . GraphSAGE is used as a comparison model for the text classification task . node2vec . Finally , we utilize node2vec ( Grover & Leskovec , 2016 ) , which only consumes topological information but no node-specific features . The node2vec algorithm learns a compressed feature space that maximizes the probability to preserve local neighborhoods . With the exception of node2vec , the specific details for the configuration of each model will depend on the dataset we are evaluating on , thus will be detailed in each of the datasets ’ results . 2.3 DATASETS . In the following we describe the datasets used and the experimental setup for the downstream tasks . 2.3.1 SYNTHETIC DATA : A QUALITATIVE ASSESSMENT . First , we consider a synthetic dataset with arbitrarily generated graphs on a plane . Each node ’ s features will be composed by the combination of the first two edges directions ’ ( in the case of nodes with a single edge the feature vector is padded with zeros ) , resulting in a feature vector of 4 dimensions . Thus , each node holds partial , yet insufficient topological information about the graph . As described above , the entire graph is then used to regularize the latent space . 2.3.2 MNIST . On a similar line we expanded this experiments by taking MNIST ( LeCun et al. , 2010 ) and generating a topology across the different labels by chaining the samples from 0 all the way to 9 . We use this dataset to further test the model ’ s capability of affecting the topology of the latent where the individual node features are of higher complexity , at least when compared to the synthetic data , while maintaining comparable reconstruction performance to the non-constrained scenario . 2.3.3 TEXT REPRESENTATION . We evaluate three classification datasets : Cora , CiteSeer , and PubMed ( Sen et al. , 2008 ) . These datasets contain networks of documents linked by the citation links between documents . The text of the document is represented as a bag-of-words , which we take as a feature vector for each of the documents . Furthermore , each document corresponds to a particular task . We divide each dataset , and use a part of it to train the embedding and the other part on a downstream class prediction task , using the embedding model mentioned above . 2.3.4 CHEMICAL REACTION REPRESENTATION . Finally , we analyze the influence of the topology in learning effective representations for molecules in the context of chemical reactions , a topic that has testified a surge in popularity in the recent past as a field for deep learning applications ( Schwaller et al. , 2019 ; 2020 ) . To this end , we adopt the dataset compiled by Jin et al . ( 2017 ) where we represent reagents , reactants and products using SMILES representations ( Weininger , 1988 ) , using the splits provided . For each molecule we extract features using the encoder of a VAE based on stack-augmented GRU layers ( Joulin & Mikolov , 2015 ) , as proposed in Born et al . ( 2020 ) , pretrained on PubChem ( Kim et al. , 2015 ) ( more details can be found in the Appendix A.2 ) . As for the topological reaction representation we consider a bipartite graph connecting the products to all the reactants and reagents . Each reaction bipartite graph is then used to generate the resulting final graph connecting all the nodes that are shared between different reactions . Using the training split provided by Jin et al . ( 2017 ) , the models are finetuned as follows : VAE at molecule level , GR-VAE and DGI at reaction level ( GR-VAE in an implicit form through the loss regularizer ) , node2vec on the aggregated graph . Furthermore , DGI uses the different VAEs and GR-VAEs as part of its encoder . To evaluate the quality of the representations learned and the impact of the topology , we consider the task of predicting whether two molecules are respectively reactant/reagent and products of a valid chemical reaction . The resulting binary classification task has an inherent relation with the underlying reaction network . For VAE , GR-VAE and node2vec we represent a pair of molecules as the concatenation of the encoded molecules/nodes in the respective latent spaces . In the DGI case , we represent the pair as the embedding of a graph connecting the molecules . These representations are then trained on the validation split and later evaluated on the test split as defined by Jin et al . ( 2017 ) .

Summary: The paper focuses on studying the importance of utilising manifold/topology information for machine learning tasks. To this end, the authors benchmark four different approaches, including VAE, GR-VAE (using graph distances to regularise  embedding distances (as shown in Eq. 1)).  The paper performs experiments on four tasks, including synthetic data, MNIST, text representation, and chemical reactions. As conclusion, the paper demonstrates that in some cases, adding relational information is beneficial, while in other cases, the effect is subtle. Thus, the paper aims to provide a metric for understanding when and how manifold/topology information is needed.