On the Importance of Looking at the Manifold
Authors: Nil Adell Mill
Jannis Born
Nathaniel Park
James Hedrick
María Rodríguez Martínez
Matteo Manica
Venue:  (2021)
Keywords: Topological Learning
GNN
VAE

Abstract: Data rarely lies on uniquely Euclidean spaces. Even data typically represented in regular domains, such as images, can have a higher level of relational information, either between data samples or even relations within samples, e.g., how the objects in an image are linked. With this perspective our data points can be enriched by explicitly accounting for this connectivity and analyzing them as a graph. Herein, we analyze various approaches for unsupervised representation learning and investigate the importance of considering topological information and its impact when learning representations. We explore a spectrum of models, ranging from uniquely learning representations based on the isolated features of the nodes (focusing on Variational Autoencoders), to uniquely learning representations based on the topology (using node2vec) passing through models that integrate both node features and topological information in a hybrid fashion. For the latter we use Graph Neural Networks, precisely Deep Graph Infomax (DGI), and an extension of the typical formulation of the VAE where the topological structure is accounted for via an explicit regularization of the loss (Graph-Regularized VAEs, introduced in this work). To extensively investigate these methodologies, we consider a wide variety of data types: synthetic data point clouds, MNIST, citation networks, and chemical reactions. We show that each of the representations learned by these models may have critical importance for further downstream tasks, and that accounting for the topological features can greatly improve the modeling capabilities for certain problems. We further provide a framework to analyze these, and future models under different scenarios and types of data.

Under review as a conference paper at ICLR 2021
ON THE IMPORTANCE OF LOOKING AT THE MANIFOLD
Anonymous authors Paper under double-blind review
ABSTRACT
Data rarely lies on uniquely Fuclidean spaces. Even data typically represented regular domains, such as images, can have higher level of relational information. either between data samples or even relations within samples, e.g., how the objects in an image are linked. With this perspective our data points can be enriched by explicitly accounting for this connectivity and analyzing them as graph. Herein, we analyze various approaches for unsupervised representation learning and in- vestigate the importance of considering topological information and its impact when learning representations. We explore a spectrum of models, ranging from uniquely learning representations based on the isolated features of the nodes (fo- cusing on Variational Autoencoders), to uniquely learning representations based the topology (using node2vec) passing through models that integrate both node features and topological information in a hybrid fashion. For the latter we use Graph Neural Networks, precisely Deep Graph Infomax (DGI), and an extension of the typical formulation of the VAE where the topological structure is accounted for via an explicit regularization of the loss Reau--Regglrrized VAEs, introduced in this work). To extensively investigate these methodologies. we consider a wide variety of data types: synthetic data point clouds, MNIST, citation networks, and chemical reactions. We show that each of the representations learned by these models may have critical importance for further downstream tasks, and that ac- counting for the topological features can greatly improve the modeling capabili- ties for certain problems. We further provide a framework to analyze these, and future models under different scenarios and types of data.
INTRODUCTION
The ability to recognize relational information between or even within individual percepts is one of the fundamental differences between human and artificial learning systems. For example, the feature-binding problem (Roskies) 1999), i.e. the mechanism governing the visual system repre- sent hierarchical relationships between Teatures an image, is still largely unsolved by neuroscien- tists, exacerbating the development of bio-inspired statistical learning systems. Traditional relational learning approaches mostly sort into either learning internal or external relational structure between samples and rely heavily crafting domain-specific expert knowledge that is engineered into the model (Struyf & Blockeel| 2010). Consequently, these models have yet to prove their usability in real applications and, although some neurocomputational frameworks for relational learning were proposed (Isbister et al.] 2018). building statistical models that explore higher-order dependencies between samples remains a key challenge for computer vision and robotics application. Conse- quently, relational reasoning has been advocated a pivotal role for the future of artificial intelli- gence (Battaglia al. 2018). On the very contrary, deep learning as purely data-driven approach has enjoyed remarkable success in recent years by learning complex non-linear functions mapping raw inputs to outputs without explicit dependency modelling. Fields like relational reinforcement learning (Džeroski et al., (001)) and statistical relational learning (Koller et al.) (007] aimed to fill this gap; but recently augmenting deep (reinforcement) learning models toward relational reasoning emerged promising approach (Zambaldi al. 2018 Zhang al., (016). Many successful con- tributions for relational modelling in images however largely rely on Euclidean spaces (Dai et al,
2017 Yao al. 2018).
is widely agreed that graphs are the ideal structure to enable relational deep learning (Hamilton
et al. 2017). Prior work has shown that metagraphs incorporating relational information about the

---

Under review as a conference paper at ICLR 2021
dataset improve unsupervised representation learning in finding less complex models that pre- serve relational information without loosing expressivity on the original representation (Dumancic & Blockeel 2017). In terms of predictive modelling. the relational representations can be superior to ordinary ones (Dumancic & Blockeel 2017) and graph-induced kernels can aid improving pheno- type prediction compared non-topological kernels (Manica et al. 2019). In generative modelling. relational distribution comparison was demonstrated to facilitate the learning of generative models across incomparable spaces (Bunne al. 2019).
Here, we perform an extensive study on the impact of the topological information in learning data representations. Specifically, we focus on the trade-off between leveraging data point features and relational information. We consider a selection of unsupervised models for learning representations lying in different areas of the spectrum. Ranging from Variational Autoencoders (VAEs) (Kingma & Welling 2013) to node embedding techniques based on random walks on graphs (Grovvr & Leskovee 2016). passing through graph neural networks (Veličković et al.] 2018) and the proposed Graph-Regularizer Variational Autoencoders (GR-VAE), our adaptation of VAEs where the latent space regularized through a metagraph representing relations between samples of the dataset that introduce in this work.
The methods considered are evaluated on different datasets and downstream tasks where the impact of the topology can be appropriately assessed. Initially, examine the impact of implicitly ac- counting for the topology to validate the GR- VAE in two synthetic studies based on topologically connected 4D point clouds and MNIST (LeCun et al. (2000) with added relational information based on the labels. After this initial validation, we move to evaluating all the methods in the case of text representations and chemical reactions. In the case of text representations, performance on Cora, CiteSeer, and PubMed (Sen et al.) 2008), using their citation networks, and evaluating the representations learned in a downstream classification task. Finally, we study the im- pact of the topology in molecule representations using a chemical reaction dataset (Jin et al. (2017), where the downstream task consists predicting the reactivity reactant/reagent-praat pairs.
analyze the methods
2
METHODS
In this section we present the different models compared in this study. Our approach is to explore spectrum models with varying availability of features and topology (see (see FFiuuee I).
VAE
GR-VAE
DGI
node2vec
Topology influence spectrum
Figure 1: Topology influence spectrum in the light of the model considered. From left to right we selected the models in order to smoothly transition from a case where only the point/node features are relevant (left, standard VAE) to the opposite end of the spectrum where only the topological properties are considered (right, node2vec). In the middle we find the cases where the point node features and the topology are blended, either implicitly via regularizer in the GR-VA CR-VAE case or explicitly the DGI case.
2

---

Under review as a conference paper at ICLR 2021
2.1 IMPLICIT TOPOLOGICAL LEARNING
We first explore VAES (Kingma & Welling 2013) which only intake features from the nodes, thus serving as baseline model agnostic topological information.
Graph Regularized VAE. We then introduce a variation of VAES (Kingma & Welling 2013 defined as Graph-Regularized VAEs (GR-VAE), that augments the trade-off between reconstruction error and KL divergence by topological constraint (see Figure 2 for an overview). In RAE, the latent space is regularized through metagraph present the data. We suggest that accounting for available information on how different samples relate (henceforth simply referred as the graph) may help at obtaining more powerful representations, especially where this relationships are directly involved with downstream task of interest.
Encode
A
O
0
0 2
d12
khs
d14
0
1
1
d21
d3
d2
C
1
0
1
1
dg
dg2
d34
D
2
d4
dep
dg.
Adjacency Matrix
Distance Matrix
Distance
Matrix
Figure 2: Definition of the graph used to build the regularization term in the GR-VAE for- mulation. Relational information between the data points (A-D) considered to define a dataset metagraph that can be used constraint distances in the learned latent space. The notation dy, in the distance matrix abbreviates dn(v, i) from Equation
Our approach adds a term to the loss defined by the set of constraints to the samples' representation in the latent space given the distances of the samples' metagraph. For given set of samples, S, can compute their distances in the latent space D, as well as over the graph g. For each node, v, embedded in D, to resemble the distances over g.
expect the distances to the other nodes, Thus, we enforce constraint aimed to preserve the relative distances in the two spaces. Formally, fixing node and considering any pair of nodes j), we can define the following penalty term:
(do(v, dp(x,
Adn, u,,), =
(dp(v, do(v.)))²
(do(v, (db(v.))-d(vv.))" if dglv,
d dz(
where do and dg are metrics defined in the latent space and over the graph respectively. In the following, we select the Euclidean distance as dp and the geodesic distance (Floyd 1962) as dg. The overall loss function of the GR-VAE thus becomes:
CGR-V = LVAE X;0) + Σ Σ
b(dn, v, v v,i,))
DE S
where X are the features of the samples in S, 0 the network parameters and y 2 0 regulates the strength of the penalty.

---

Under review as a conference paper at ICLR 2021
2.2 EXPLICIT TOPOLOGICAL LEARNING
Notably, GR-VAE is devised to infer topological information solely from a soft constraint, without any architectural requirements such graph convolutions. On the other side of the spectrum, graph neural networks (GNN) instead model topology explicitly.
Deep Graph Infomax. Here, we consider a Deep Graph Infomax (DGI), a state-of-the-art GNN for unsupervised representation learning (Veličković et al., 2018). DGI relies on maximizing mu- tual information between subgraphs (themselves derived with GCNs) yielding representations that facilitate downstream node-wise classification tasks. These two models used as comparison for the text classification task.
GAE and VGAE.
We consider both prnbbbbbaiiilittcc Graph Autoencoders (GAE) and Variational Graph Autoencoders (VGAB) (Kipf& & Welling 2016) which are models for unsupervised-grapp data inspired on the VAB.
GraphSAGE. We consider GraphSAGE, an inductive framework that generates node embeddings by sampling and aggregating the features for the local neighborhood of node (Hamilton et al, 2018). GraphSAGE is used as comparison model for the text classification task.
node2vec. Finally, we utilize node2vec (Grover & Leskovec, 2016), which only consumes topo- logical information but node-specific features. The node2vec algorithm learns compressed feature space that maximizes the probability to preserve local neighborhoods. With the exception of node2vec, the specific details for the configuration of each model will depend on the dataset evaluating on, thus will be detailed in each of the datasets' results.
2.3 DATASETS
In the following we describe the datasets used and the experimental setup for the downstream tasks.
2.3.1 SYNTHETIC DATA: A QUALITATIVE ASSESSMENT
First, consider synthetic dataset with arbitrarily generated graphs on plane. Each node's features will be composed by the combination of the first two edges directions' (in the case of nodes with single edge the feature vector is padded with zeros), resulting in feature vector of dimensions. Thus, each node holds partial, yet insufficient topological information about the graph. As described above, the entire graph is then used regularize the latent space.
2.3.2 MNIST
On similar line expanded this experiments by taking MNIST (LeCun al., 2010 and gener- ating a topology across the different labels by chaining the samples from 0 all the way to 9. We use this dataset further test the model's capability of affecting the topology of the latent where the individual node features are of higher complexity, at least when compared to the synthetic data, while maintaining comparable reconstruction performance to the consttratti scenario.
2.3.3 TEXT REPRESENTATION
We evaluate three classification datasets: Cora, CiteSeer, and PubMed (Sen et al. 2008). These datasets contain networks of documents linked by the citation links between documents. The text bag-of-words, which take feature vector for each of the
of the document is represented documents. Furthermore, each document corresponds to particular task. We divide each dataset, and use part of train the embedding and the other part on downstream class prediction task, using the embedding model mentioned above.
2.3.4 CHEMICAL REACTION REPRESENTATION
Finally, we analyze the influence of the topology in learning effective representations for molecules in the context of chemical reactions, topic that has testified surge in popularity in the recent past as
4

---

Under review as a conference paper at ICLR 2021
VAE
GR-VAE
Latent
13-10-11 DE
Fenture
Figure 3: Learned representations on two synthetic datasets with diferent topology. This figure displays qualitatively how VAE affects the latent space topology under different conditions, specifically when compared to its non-constrained counterpart. The plots on top show the latent space, the ones in the bottom show the feature space, for the first two features. The two datasets (left and right) had different topologies, shown as graph, next the color bar (all colors across plots correspond to the nodes' ids). On the features plots both the original data (round marker) and the reconstructed data (cross marker) are shown.
field for deep learning applications (Schwaller al. 2019 2020). To this end, we adopt the dataset compiled by Jin et al. (2017) where we represent reagents, reactants and products using SMILES representations (Weininger 1988), using the splits provided. For each molecule we extract features using the encoder of a VAE based on stack-augmented GRU layers (Joulin & Mikolov, 2015), as proposed Born et al. (2020), pretrained on PubChem (Kim et al.] (003)) (more details can be consider a bipartite graph connecting the products to all the reactants and reagents. Each reaction bipartite graph is then used generate the resulting final graph connecting all the nodes that shared between different reactions. Using the training split provided by JJi et al. (2017), the models are finetuned as follows: VAE molecule level, GR-VAE and DGI at reaction level (GR-VAE in implicit form through the loss regularizer), node2vec on the aggregated graph. Furthermore, DGI uses the different VAEs and GR-VAEs as part of its encoder.
found in the Appendix A.2). As for the topological reaction representation
To evaluate the quality of the representations learned and the impact of the topology, we consider the task of predicting whether two molecules are respectively reactant/reagent and products of valid chemical reaction. The resulting binary classification task has an inherent relation with the underlying reaction network. For VAE, GR-VAL and node2vec we represent a pair of molecules as the concatenation of the encoded molecules/nodes in the respective latent spaces. In the DGI case, we represent the pair as the embedding of a graph connecting the molecules. These representations are then trained on the validation split and later evaluated on the test split as defined by Jiin et al. 2017).
RESULTS
We break our experimental results in two parts based on the division previously made (in Section 2 First, analyze the validity of implicitly learning the topology through the proposed extended VAE formulation. Secondly, we compare all the different modalities, including implicit and explicit topology set of different task.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Quantitative results for the MNIST experiment. We report results for three differ- ent models with varying number of dimensions in the latent space: 3, 16, and 64. For each one we explore four training setups, regular VAE (y 0) and three intensities of GR-VAE (Y E [1, 10, 100)). We then report the reconstruction loss, the silhouette score of the test sam- ples in the latent space, and the Hamiltonian. Furthermore we train two downstream model: k-NN and classification tree, and report their average Fl scores over 5-fold cross-validation.
Latent dimensions
3
16
64
GR y
0
10
100
10
100
100
Reconstruction loss
141
143
147
172
83
82
84
105
81
79
81
95
Silhouctte score
.052
.092
.216
.195
.074
.096
.141
.178
.055
.060
.112
.168
K-NN Tree
.634
.711
.766
.816
.928
.933
.946
.940
.938
.941
.947
.937
.544
.643
.700
.753
.737
.728
.808
.818
.692
.755
.777
.819
3.1 A VALIDATION ON THE IMPLICIT TOPOLOGICAL LEARNING
In this section present the results for the described experiments on implicit topological learning using VAE and GR-VAE.
3.1.1 SYNTHETIC DATA
We perform different analysis to better understand how the addition topological information may influence the representations that model will learn. For that generate synthetic dataset (de- scribed in Section (2.3II) and train both a regular VAE and GR-VAR VAE. Qualitatively the results of this experiment (Figure 3) show us the stark influence that accounting for the structure has in the final learnt embedding. We can see that, while both models are able to achieve a comparable re- construction, the latent space in the GR-VAE still preserves the topology of the metagraph in the data, and one is able to reconstruct this from the different clusters in the latent space. This simple experiment already shows how one may achieve scenarios where the metagraph topology does in fact determine the latent space topology, and motivates further the rest of the experiments in this study.
3.1.2 MNIST
Here we discuss the of experiments conducted the MNIST dataset where impose a syn- thetic topological structure in the form of a chain following the numerical ordering of the labels (from 0 9). In Table we show how the model performance, Silhouette score (Rousseeuw) 1987) and Fl score in downstream task (i.e. label prediction) are affected by changing the latent dimensions and the weight of the graph regularizer. Interestingly, we can see how the topological information, even if in the form of a simple chain, is helping in improv- ing the quality of the learned image representations. Qualitatively, we can also observe in Figure 4 how the latent space can benefit from including topological information indirectly regularizing the loss, that becomes quite evident when looking at the distance matrix the of the centroids.
terms of reconstruction error,
3.2 EXPERIMENTS ON THE FULL TOPOLOGICAL SPECTRUM
In this section present the results for the described experiments explicit topological learning using all the models introduced in the the methods (Section 2)
3.3 TEXT REPRESENTATIONS
For the text datasets (Section 2.3.3) we adapted the models such that would be fitting with the data and structure. For the VAΣ we decided to use relatively simple model composed of two fully connected layers with ReLU activation, for both the encoder and the decoder. For DGI we followed the setup described in the original paper (Veličković et al.) 2018). where the encoder is composed by a one-layer Graph Convolutional Network (GCN) (Kipf & Welling 2017) with PReLu (He et al. 2015) activation. On all the models, the learned representations then used to train logistic

---

Under review as a conference paper at ICLR 2021
A.
B.
7 2 0 4
5 7
7 2 7
5 7
2
4 4
7 2 0 4
7 2
7
c.
10
123
100
Distance
75
matrix
65
Figure 4: Qualitative analysis of the latent representations learned in the MNIST case. A. PCA projection of the samples in the latent space under different training regimes. The original latent space has 16 dimensions. The metagraph is chain connecting each class from 0 9 in order (a representation can be seen on the top right). The samples can be seen coloured by class pertinence. B. Display of a reduced set of the original samples (bottom row) and their reconstructions (top row). C. Distance matrix between the centroids of each label's point cloud.
regression on of train nodes and predict each document's (i.e. node) class. We, then, report the prediction accuracy on the test nodes. The splits were reused from Yang et al. (2016).
Furthermore, order be able to train the ERRVAE we create batches by sub-sampling the graph around each of the nodes in the graph, instance of the DGI model was also trained under the same circumstances in order to fairly compare both setups, where to be the case that the sub-sampling affected the performance. The results for the text task can be seen in Table 2
order to train the GR-VAE we create batches by sub-sampling the graph around each of the nodes in the graph. In order to generate each of these batches, for each node in the graph sample 20 nodes in its neighborhood (up to hops away from the node). This setup. where at a particular step the model only has access as a separate setup. Furthermore, we explore, under this condition, how pre-encoding the node information (using either VAE GR-VAE) affects the performance of DGI when compared to using the raw node features directly. The results for this experimental setup can be seen in Table 3)
partial observation of the entire graph, is also applied DGI
Our results show clearly the strong performance of DGI in the three datasets. Interestingly DGI's performance drops when only obtaining batched information. As the authors point out when com- paring GCN, DGI seems benefit from the fact that has access the entire graph (Veličkovid et al. 2018). Node2vec outperforms both VAE and GR- VAE in Cora and PubMed, however it falls behind in CiteSeer. We assume that that difference arises due to the relative importance of the graph topology in the different datasets. In CiteSeer, it is likely that, features alone hold more informa- tion of the underlying class than the citation and references network around each document. The relationship between VAE and GR-VAE also reflects this balance, whereas in Cora the addition of topological information seems to aid to the structuring of the latent space and help in creating a more useful representation, the oposite happens in CiteSeer. That duality shows how this infor- mation may aid in cases where it's more relevant for the downstream task. but it may hinder in cases where the direct link between topology and class (or downstream task) is weaker straight

---

Under review as a conference paper at ICLR 2021
Table Results on the text representations. Accuracy results for the classification task in the Cora, CiteSeer, and PubMed dataset. In this particular experiment GR-Ve VAE model was trained with equally weighted factors (y = 1) of the loss components (reconstruction, Kl-ddiveeeecce and graph regularization).
Model
Cora
CiteSeer
PubMed
Input data
Random
0.152
0.152
0.322
VAE
0.530
0.531
0.525
V
GR- VAE
0.607
0.492
0.32
V,
DGI
0.819
0.684
0.736
V. E
GAE
0.779
0.644
0.774
V,
VGAE
0.759
0.569
0.718
V,
GraphSAGE
0.718
0.538
0.765
V, E
node2vec
0.719
0.464
0.676
E
Table 3: Results text representations with partial graph observability. Accuracy results for the text classification task in the Cora, CiteSeer, and PubMed dataset. In this particular experiment DGI was trained using the same batching procedure GR-VAE model trained with different values for the graph regularizer weight, 7.
Cora
CiteSeer
PubMed
Latent space size
Model
GR y
20
100
250
20
100
250
100
250
DGI (VAE)
0.738
0.723
0.746
0.611
0.650
0.635
0.457
0.558
0.620
DGI (GR-VAR
0.5
0.701
0.732
0.744
0.663
0.616
0.612
0.384
0.418
0.448
DGI (GR-VAE)
0.73
0.756
0.761
0.612
0.644
0.646
0.429
0.431
0,410
DGI (GR-VAE)
0.705
0.734
0.764
0.624
0.626
0.630
0.522
0.477
0.312
DGI VAE)
10
0.527
0.7
0.576
0.597
0.635
0.654
0,594
0.499
0.377
DGI VAE)
100
0.326
0.49
0.361
0.301
0.426
0.355
0.442
0.541
0.524
DGI
0.738
0.611
0.722
non-existent. Ultimately, this regularizer can lower signal-to-noise ratio in cases where particular topology is irrelevant.
The results (Table 3) with partial observability reflect the same observations stated previously. DGI performs worse than in the previous setup (Table 2), which was an expected result. We see that under some cireumstances pre-encoding the node features with either VAE GR-VAE may improve the performance. In particular the observation about the relationship between VAE and GR-VAE (made the previous results) is also reflected in these results. In spite of this, the results appear to show high variation and not clear pattern of what parameter combination increases performance. reason for that could be the unaccounted randomness introduced by the sampling method used in the batching procedure, which may provoke variation in accuracy from model to model.
3.4 CHEMICAL REACTIONS
The results of the experiments run on the chemical reactions dataset can be seen in Table 4 Similarly the text dataset, using DGI gave the best performance. Although the results are more nuanced, ul- timately the encoder used was critical. For instance, when using encoder pretrained in a different dataset we could find situations where the DGI performs worse than the VAE encoder alone. The opposite end is when combining all the methods used in the study, where DGI using node embed- dings finetuned with a GR- RRR achieves the highest accuracy. is extremely interesting to observe such behavior, where topological information is present. This seems to suggest that the quality of the SMILES embedding key in the task considered.
that among the three top performing models plain VAE with
8

---

Under review as a conference paper at ICLR 2021
Table 4: Results for chemical reactions experiment. We report the accuracy on the downstream reaction task. The annotations in parenthesis specify details about the encoder: Finetumed denotes that the VAE or GR-VAE has been finetuned on chemical reaction data (on a different split from the downstream reactions). in the case of the DGI the annotation references to which VAE model was used for encoding the SMILES. For each instance of the GR-VAE we display which y we used in training.
Model
GR 7 Accuracy
Random
0.5
VAE
0.5740
VAE (finetuned)
0,5613
GR-VAE (finetuned)
0.5
0.5631
GR- VAE (finetuned)
1
0.5470
GR-VAE (finetuned)
0.5624 2
GR-VAE (finetuned)
0.5543 5
DGI (VAE)
0,5003
DGI (VAE finetuned)
0.6248
DGI (GR-VAL finetuned)
0.5
0.6602
DGI (GR-VAE finetuned)
1
0.5617
DGI (GR-VAI finetuned)
2
0,5507
DGI (GR-VAE finetuned)
0.5 5
0.5321
node2vec
4 DISCUSSION
In this study we explored the importance of topological information in learning data representa- tions. We demonstrated the addition of inter-sample relational information means improve these representations, and stressed the trade-off between leveraging data point features and relational information.
We have described a novel loss that expands the VAE by leveraging a relational metagraph and described under which circumstances this added factor becomes a support for further downstream tasks. Most evident are our MNIST results, where adding data that is directly linked to the down- stream task of interest creates more useful arrangement of the latent space, resulting in improve- ments of the downstream prediction using these embeddings in all the explored setups. It is worth emphasizing that the regularized introduced in the GR-VAE, can not only inject topological aware- ness into non-topological models, but also be combined with them to achieve superior performance in downstream prediction tasks as we see in the chemical reaction case. Furthermore, we explore scenarios where the metagraph is less obviously linked to the end prediction. In those, the benefit of adding graph regularizer (i.e. GR-VAE vs. VAE) is more subtle. Our work opens the door to further exploring ways to evaluate which representation are more useful for given downstream tasks as well as, creating metrics to quantitatively evaluate so. In short, this work aims to be a motiva- tion for looking at the manifold and a small step towards understanding how inter-sample relational information can be beneficial, even in those cases where this data is not explicitly ingested by the model or where the link to a particular end goal may not be obvious.
REFERENCES
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
Jannis Born, Matteo Manica, Ali Oskooei, Joris Cadow, and María Rodríguez Martínez. Paccmann rl: Designing anticancer drugs from transcriptomic data via reinforcement learning. In Inter- national Conference on Research in Computational Molecular Biology, pp. 231-233. Springer, 2020.

---

Under review as a conference paper at ICLR 2021
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv:1571.06349, 2015.
Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In ICML, 2019.
Bo Dai, Yugi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In Proceedings of the IEEE conference on computer vision and Pattern recognition, pp. 3076-3086, 2017.
Sebastijan Dumancic and Hendrik Blockeel. Clustering-based relational unsupervised representa- tion learning with an explicit distributed representation. International Joint Conference on Artifi- cial Intelligence (IJCAI), 2017.
Saão Džeroski, Luc De Raedt, and Kurt Driessens. learning, 43(1-2):7-52, 2001.
Relational reinforcement learning. Machine
Robert W Floyd. Algorithm 97: shortest path. Communications of the ACM, 5(6):345, 1962.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks, 2016.
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning graphs: Methods and applications. arXiv preprint (0090057009 2017.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
Michael Held and Richard M Karp. A dynamic programming approach to sequencing problems. Journal of the Society for Industrial and Applied mathematics, 10(1):196-210, 1962.
James Isbister, Akihiro Eguchi, Nasir Ahmad, Juan M Galeazzi, Mark Buckley, and Simon Stringer. A new approach to solving the feature-binding problem in primate vision. Interface focus, 8(4):201 80021, 2018.
Wengong Jin, Connor Coley, Regina Barzilay, and Tommi Jaakkola. Predicting organic reaction out- comes with weisfeiler-lehmaal network. Advances in Neural Information Processing Systems, pp. 2607-2616, 2017.
Armand Joulin and Tomas Mikolov.
Inferring algorithmic patterns with stack-augmentt recurrent nets. In Advances nearal information processing systems, pp. 190-198, 2015.
Sunghwan Kim, Paul A Thiessen, Evan Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 440D1).D1222-11213 2015.
Diederík P Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114 2013.
Thomas N. Kipf and Max Welling. Variational graph auto-encoders, 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works, 2017.
Daphne Koller, Nir Friedman, Saão Džeroski, Charles Sutton, Andrew McCallum, Avi Pfeffer, Pieter Abbeel, Ming-Fai Wong, David Heckerman, Chris Meek, et al. Introduction statistical rela- tional learning. MIT press, 2007.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
Available: http://yannllecum. wwwp/xdd//mmiist..mmmmmmmmmmm http://yaaa.eeuu..om/eeddmmmmii
Matteo Manica, Joris Cadow, Roland Mathis, and María Rodríguez Martínez. Pimkl: Pathway-
induced multiple kernel learning. NPJ Systems Biology and Applications, 5(1):1-8, 2019.
10

---

Under review as a conference paper at ICLR 2021
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84. 2007.
Adina L Roskies. The binding problem. Newron, 24(1):7-9, 1999.
Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53-65, 1987.
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: A model for uncertainty--aliibateed chemical reaction prediction. ACS central science, 5(9):1572-188,, 2019.
Philippe Schwaller, Daniel Probst, Alain C Vaucher, Vishnu H Nair, David Kreutter, Teodoro Laino, and Jean-Louis Reymond. Mapping the space of chemical reactions using attention-based neural networks. ChemRxiv preprint, 2020.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Radd Collective classification in network data, Al magazine, 29(3):93-92 93, 2008
Jan Struyf and Hendrik Blockeel. Relational Learning. pp. 851-857. Springer US, Boston, MA, 2010. ISBN 978-0-387-301644 doi: 10. 1011007/978---387-3016-- 3rg/10.1007/00
URhhtps:://doi
0 177-30164488_117
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax, 2018.
David Weininger. Smiles, chemical language and information system. introduction method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Ronald Williams and David Zipser. learning algorithm continually running fully recurrent neural networks. Neural computation, 1(2):270-280. 1989.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. pp. 40-48. PMLR, 2016.
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pp. 684-699, 2018.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.
Xiao Zhang, Maria Leonor Pacheco, Chang Li, and Dan Goldwasser. Introducing drail-a step towards declarative deep relational learning. In Proceedings of the Workshop on Structured Pre- diction for NLP, pp. 54-62, 2016.
A APPENDIX
A.1 GRAPH REGULARIZED VAE DETAILS
In Figure 5 a detailed description of the GR-VAL architecture is depicted.
A.2 SMILES EMBEDDING
The SMILES VAE used for the chemical reaction dataset was implemented following the description
in (Bor et al. 2020). It consists of two layers of stack-augmented GRUs (Joulin & Mikolov 2015)
in both encooder and decoder and trained with teacher forcing (Williams & Zipser 1989 token dropout (Bowman et al. 2015) and one-hot encodings.
The dataset consisted of 500,000 molecules represented as canonical SMILES strings from Pub- Chem (Kim al. 2015)..
11

---

Under review as a conference paper at ICLR 2021
Per node reconstruction loss
Encoded
A
O
0 1
0
12 2
du dd du
B
0
1 0
2 0
10 d2
C
0
0
dg
d5t
D
0
0
1
dg
de
dg
Adjacency
Distance Matrix
Distance
Matrix
Matrix
Graph constraint
Figure 5: Complete overview of the GR-VAE approach. Notice that, the notation dys in the distance matrix abbreviates (D, from Equation
A.3 EXTENDED IMPLICIT LEARNING
Here we show the extended results for the MNIST implicit learning tasks.
Figure 6 shows expanded results for setup where the VAE was mapping latent space of dimensions. In that case we can see that with a strong regularizer we still accomplish our desired objective of organizing the point clouds the points, challenges the model and makes it more difficult to obtain reconstructions as faithful to the original images as those we saw with models with more dimensions (Figure 4p. However comes useful to display how the embeddings done using the graph regularizer can help at creating clear distinctions between sample groups. For instance, the non-regularized VAE mixes number of digits (see Figure 6B), while the models that were regularized manage to reconstruct the same digit (i.e class), usually at the expense generating reconstructions that are less faithful the original image in term of details or style
chain. This setup, with only 3 dimensions where to map
To validate the qualitative assessment on the model's ability to restore the original chain as shown in Figure 6C, we computed the Shortest Hamiltonian Paths (SHP) (Held & Karp, 1962) fully connected graph of 10 nodes (representing the centroids of the labels in the latent space) where the network topology (i.e. edge weights) was given by the pairwise distances of the centroids. If the it proves that the topology is preserved perfectly in the
SHP of such graph is a chain from latent space. To compare the different chains used Dynamic Time Warping Müller (2007), distance measure based on time series alignment computed with FastDTW An optimal topology corresponds to DTW distance of O. The results for the later can be seen in Table 5 the full chains can be seen in Table 6
12

---

Under review as a conference paper at ICLR 2021
-0
42 22003
12
72104
7 72104
7270 9
9
5
72. 0
7
72/04
4
7 72102
10
15
15
10
23
10. 8.
6.
9,
[0. 6,
9.
5.
10. 6,
4,
10. 3.
2, 3.
6,
7. 91
Figure 6: Qualitative analysis of the latent representations learned in the MNIST case (with latent space size set of experiments reason A. directly displays all the latent space dimensions (not PCA projection). also includes an extra setting (y 10) for the regularizer.
3). Figure with extended information about the MNIST results, displays same in Figure but using VAE with latent space of 3 dimensions. For that
Table 5: Shortest Hamiltonian Path (SHP) distance to ordered chain. This table displays the distance from each SHP to an ordered chain (0 9) using FastDTW. For reference, the average distance of a random connected path is 30.03 + 6.5 (computed with 1000 random sequences).
Latent space dimensions
3
16
64
23
23
27
31
27
10
0
31
3
100
0
0
0
Table 6: Shortest Hamiltonian Paths. Full chains obtained when running SHP over the class
centroids samples the latent space. We can see that with the biggest value of the regularizer (7 100) SHPs recover the original chain used for the constraint.
Latent space dimensions
Y
3
16
64
no regularizer
0 15833
7 4
06
8
9
06497
0649
5
321
06
7 21
0649715 82
10
0649
23
01
4 6
89
012345 9
100
012344
512346677
89
0
8
0 01234 3 89
13


Reviews:
[{"review_id": "gj1W5-VatE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\nThis paper investigates different ways of incorporating topological information about the data in the machine learning models. The paper introduces a novel loss that aims to enforce the relational information between data points into the embedding space learned by a Vae on the node features. The experiments demonstrate that for data with a certain topology type, the introduced loss can provide performance when used together with existing methods. The paper opens up possibilities of further investigation into incorporating topological information (if available) into the learning procedure.\n\n**Pros**:\n1. The paper is very well-written and easy to follow. The illustrations also present the idea clearly.\n2. The problem of understanding the importance of topological information is interesting, and could lead to future works.\n\n**Cons**:\n1. I am not sure if there is enough novelty for acceptance to ICLR, especially when the proposed method does not provide obvious benefit over existing methods, both in theory and practice. Specifically, the GR-VAE is a simple extension of VAE which does not yield good performance, unless it’s combined with DGI. Even in that case, it is not obvious to me the benefit is significant except in one case (namely for text representations), but when combined with DGI, it becomes unclear whether the performance boost is actually coming from the proposed loss or some other unintended regularization effect since DGI already uses message passing to incorporate the (local) topological structure. Am I missing something here?\n2. Since the paper is positioned to be an experimental study, it is perhaps acceptable to have limited novelty or improvement. However, the findings in the papers are somewhat expected. For example, we already know that GNN [1] based methods are superior to other methods on citation benchmarks since they account for both features and topology. In this light, I feel there is not too much new insight in the paper to warrant a publication at ICLR.\n3. For an empirical study, considering only four models may not be enough (e.g. different GNNs / graph models encode different topological information and that should be taken into account for a full spectrum). The same holds for feature-based methods. Some examples of this are Deep walk [3], GNN architecture or different VAE architecture and loss (especially there are so many variations of VAE). Most importantly, a comparison to [2] is missing.\n\n**Comment**:\n1. “Regularized” -> “regularizer” in conclusion line 6 paragraph 2?\n2. For a double-blind reviewing process, the funding information should perhaps be removed, although the one in this paper does not expose the authors' identity.\n\n**Conclusion**:\nWhile the work has interesting motivations and is well-written, it has not done a convincing job at demonstrating the effectiveness of their proposed method or shown a thorough experimental analysis. As such, I am inclined to reject the paper in its current form.\n\n**Reference**:\n\n[1] Semi-Supervised Classification with Graph Convolutional Networks (https://arxiv.org/abs/1609.02907)\n\n[2] Variational Graph Auto-Encoder (https://arxiv.org/abs/1611.07308)\n\n[3] Deepwalk: Online learning of social representations (https://arxiv.org/abs/1403.6652)\n\n\n**================================== Update after rebuttal ==================================**\n\nIt seems that the authors have only provided a general comment for all reviewers, which is understandable since most criticisms from all reviewers are on the same weaknesses of the paper.\nWhile I appreciate the author's effort in adding more experiments, I do not think the added experiments and reply properly addressed the concerns shared by other reviewers and myself. For example, it is still unclear what the advantage of the proposed method is or what insights we could gain from this study.\nI think this paper is not ready for publication. As today is the last day of discussion period, I will maintain my original assessment.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "caC1m8LsR8P", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on studying the importance of utilising manifold/topology information for machine learning tasks. To this end, the authors benchmark four different approaches, including VAE, GR-VAE (using graph distances to regularise  embedding distances (as shown in Eq. 1)).  The paper performs experiments on four tasks, including synthetic data, MNIST, text representation, and chemical reactions. As conclusion, the paper demonstrates that in some cases, adding relational information is beneficial, while in other cases, the effect is subtle. Thus, the paper aims to provide a metric for understanding when and how manifold/topology information is needed. \n\nPros:\n\n1. Instead explicitly learning a graph, this paper proposes an implicit method with graph regularisation.\n\n2. The related work is well-explained. The paper did a well summarization of previous methods.\n\n3. The paper performs extensive experiments to study the importance of manifold for prediction tasks.\n\nCons:\n\n1. The latent graph method is not novel enough, since the method can be categorised as graph regularisation, which is a widely used method in recommendation and information retrieval. Could the author explain why this regularisation is picked from a spectrum of graph regularisation algorithms?\n\n2. The comparison of the paper is problematic. First, the methods (DGI, node2vec, GR-VAE, VAE) compared are quite different methods. Can the authors confirm that the comparison is fair and meaningful (e.g. eliminating other confounding factors like controlling the number of parameters)? Second, I am not sure whether this comparison is optimal. In particular, to study the importance of relational information, other method can be used. For example, we can control adjacency matrix received by graph neural networks. We can totally ignore the edge information (like VAE in the paper) or use a predefined graph (e.g. a fully connected graph like in Transformer). In between, we can corrupt input graphs (e.g. randomly adding or deleting some edges) before feeding it to graph neural networks. This approach seems more reasonable to me for studying the importance of manifold. It is difficult to control these in this paper because the methods used in this paper are totally different  (e.g. DGI and GR-VAE differs in both loss function and input format). So, the conclusion of the paper is skeptical. It mainly justifies which method can perform better in downstream tasks instead of justifying the importance of manifold. \n\n3. The introduction is lengthy and should be more focused on the contribution of this paper. Similarly, the other sections need a major revision to highlight the contribution, as the main contribution of the paper lies in the implicit graph regularisation and a comparison of a series of methods with/without relational information.\n\n4. Some baseline methods are not considered, for example the methods learning latent graphs: Semi-supervised classification with graph convolutional networks and Glomo: Unsupervised learning of transferable relational graphs.\n\n5. The acknowledgement of the paper reveals location information, which may be a violation of anonymity. \n\nBased on these cons, I think a more rigorous comparison is needed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "cQ9A4lkx6AM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors present a regularisation term for Variational Auotencoders that forces the distance of mapped points in the embedding space to be similar to the distance of those points in the metagraph of the data derived from relational information about these points. The intention of the regularisation term is to enforce a consistent graph between the original representation and the embedded representation in a manner that is agnostic to the structural choices of the model to be estimated. \n\nStrengths:\n\n1) The paper provides a good organisation of existing methods for utilising topological information, and, thus, positions its contributions well in relation to existing work.\n\n2) The empirical results are presented honestly, even when they do not support the proposed method.\n\nWeaknesses:\n\nOrdered form less to more specific:\n\n1) The intention of the paper is unclear.\n\tIs this intended as a review paper of recent research on graph neural networks or to present a new regularisation term? The title and abstract seem to imply that this is intended as a review, but the paper only considers three existing methods and a single modification to VAEs. Unfortunately, the paper is not convincing in either regard, and the idea of analysing topological information to improve classifications and representations is already well discussed in the literature and a very active area of ongoing research.\n\n2)  The efficacy of the proposed regularisation is not convincingly supported either theoretically or empirically.\n\tThe authors state, ‘Notably, GR-VAE is devised to infer topological information solely from a soft constraint, without any architectural requirements such as graph convolutions’ (line 1, p. 4), but this is not discussed further. The intention of graph convolutions is to explicitly encode assumptions about the relationships present in the data, in this situation why would I prefer a soft constraint to a well motivated, explicit one? If structural constraints were unduly restricting the expressiveness of the models, I would expect to see this borne out in the empirical results, but this is not the case across the chemical reaction and citation network experiments. \n\n3) The paper struggles with clarity at points.\n\tSpecifically, equation (1), which describes the regularisation term is unclear as written: does the plus sign in the exponent denote absolute value or something else? This notation is non-standard and I would not be able to faithfully recreate the results as written.  \n\tAdditionally, the method for constructing the meta-graph G should be discussed in more detail. From what is this graph derived? Is it an existing observed graph that describes the observed relationships between the data, ex. the citation network or pixels in an image, or does it describe adjacency of the observations as defined by the observed labels or other meta-information. My concern is that using a soft-constraint which effectively focusses the model on the labels in an ‘easy’ task such as MNIST classification or the synthetic data task hides the fact that the constraint is too soft to produce a useful regularisation of the model, as evidenced by the failure of GR-VAE relative to DGI or even vanilla VAE in the citation network and chemical reactions tasks. \n\nReasons for score: \n\nI vote for rejecting the paper, as while I really do appreciate that the results of the paper are presented honestly, I think there are concerns with the current draft. \n\nQuestions for the rebuttal period:\n\nPlease refer to the questions in the weaknesses section. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EmdvrOVHYId", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==== Summary ==== \n\nThis paper proposes a variant of Variational Autoencoders (VAEs) which takes extra topological information (e.g. adjacency matrix) into account in the loss function during training. The principle objective of the proposed GR-VAEs is to use the learned latent space features as the input for improvement on downstream tasks especially for classification. \n\n==== Pros ====\n\n+ This paper is well-written and the idea is clear and easy to follow.\n+ The proposed GR-VAE, specifically the extra loss function on preserving the geodesic distance seems to be effective for learning desired latent space features from experiments.\n\n==== Cons ====\n\nMy main concern is on the experiment on showing the improvement for downstream tasks from the embedding learned by GR-VAE.\n\n-  I think the goal of the experiment is to demonstrate that, the embedding learned by GR-VAE is superior comparing to other kinds of features such as from the vanilla VAE, so I expect the embedding by GR-VAE is applied to different models (e.g. GraphSAGE, GCN, DeepWalk) instead of just DGI (at least I only notice that DGI is adapted), and on each model the result from using GR-VAE can outperform others using raw data features or other kinds of finetuned features. I think showing the improvement on different models can greatly enhance the soundness of this paper.\n\n- In addition, for the experiment on chemical reaction I think there should be another baseline showing the result of DGI by using the raw data features if possible, this can further demonstrate the importance of GR-VAE. Also, for the experiment on text representation I also expect some result like using finetuned GR-VAE as the input for DGI in the chemical reaction experiment, currently the text representation experiment only shows the strength of the DGI itself, which does not make sense to me.\n\n==== Reason for scoring ====\n\nOverall, I think the proposed GR-VAE is sound if its strength can be demonstrated by more experiment mentioned above, and I am willing to upgrade my rating and vote to accept if such concern can be addressed during the rebuttal period.\n\n==== Minor Comments ====\n\n- The plots in Figure 3 is too blurry to distinguish between the cross marker and round marker.\n- I notice for the synthetic dataset the direction of edges for each node is used as part of the input features, so what is the definition for the edge direction? Also, if we directly combine the raw data feature with embedding by some manifold learning technique, and input it into the vanilla VAE, can we get similar result (the graph topology is preserved) as GR-VAE has? \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the motivation of the paper is interesting the reviewers expressed concerns about the experimental setup, comparison to related work, and paper framing. For experiments, it was unclear why authors compared such disparate methods instead of more fine-grained adjustments (e.g., such as corrupting graphs as suggested by R3). For comparison, other methods such as Deep Walk and VGAE (as suggested by R1) seemed missing. I think the biggest issue however was with framing: as the reviewers pointed out, it was not clear enough how looking at downstream performance relates to looking at the manifold. In fact the paper title is much too general and is also well-known: manifold learning has been around for 15+ years. I would urge the authors to take the recommendations of reviewers and either design new experiments that explicitly target the manifold or reframe the paper to design new evaluation metrics for latent (possibly structured) generative models.
Decision: Reject