Title: Skip-graph: Learning graph embeddings with an encoder-decoder model
Authors: John Boaz Lee
Xiangnan Kong
Venue: ICLR 2017
Keywords: Unsupervised Learning
Deep learning

Abstract: In this work, we study the problem of feature representation learning for graph-structured data. Many of the existing work in the area are task-specific and based on supervised techniques. We study a method for obtaining a generic feature representation for a graph using an unsupervised approach. The neural encoder-decoder model is a method that has been used in the natural language processing domain to learn feature representations of sentences. In our proposed approach, we train the encoder-decoder model to predict the random walk sequence of neighboring regions in a graph given a random walk along a particular region. The goal is to map subgraphs — as represented by their random walks — that are structurally and functionally similar to nearby locations in feature space. We evaluate the learned graph vectors using several real-world datasets on the graph classification task. The proposed model is able to achieve good results against state-of- the-art techniques.

TL;DR: An unsupervised method for generating graph feature representations based on the encoder-decoder model.

Review discussion:
What is the classifier that you used for ECFP? In the text you only say that you used a library from a prior work. I think that for a correct comparison you should  use the same classifier for ECFP and skip-graph embeddings, because your dataset is very small and skewed, and the choice of regularization might be very important.
Thank you for your comments and suggestions. We have updated our paper.
In particular we added the following things:
(a) short discussion on how the method can be used with unlabeled graphs (sec. 2.2.1)
(b) short discussion and figure of visualization of the embeddings for one of the datasets (sec. 3.5, figure 4)
(c) experiments were rerun to use a common classifier (neural network) for all tested methods (sec. 3.2)
Furthermore, in response to your suggestion in the review, we have included DeepWalk as an additional baseline method which we tested against. The additional information is included in sec. 3.2 and sec. 3.3.
The paper presents a method to learn graph embeddings in a unsupervised way using random walks. It is well written and the execution appears quite accurate. The area of learning whole graph representations does not seem to be very well explored in general, and the proposed approach enjoys having very few competitors.
In a nutshell, the idea is to linearize the graph using random walks and to compute the embedding of the central segment of each walk using the skip-thought criterion. Being not an expert in biology, I can not comment whether or not this makes sense, but the gains reported in Table 2 are quite significant.
An anonymous public comment compared this work to a number of others in which the problem of learning representations of nodes is considered.
While this is arguably a different goal, one natural baseline would be to pool these representations using mean- or max- pooling. It would very interesting to do such a comparison, especially given that the considered approach heavily relies on pooling (see Figure 3(c))
To sum up, I think it is a nice paper, and with more baselines I would be ready to further increase the numerical score.
I would like to thank the authors for following my suggestion and trying the averaged DeepWalk embeddings. Unfortunately, the results seem to show that the proposed method outperforms DeepWalk only of two datasets out of four.
Hence, I can not increase the score that I gave to the paper. Arguably, DeepWalk method is much simpler, and the complexity of the proposed method has to be backed up by a strong experimental advantage.
Alternatively, an additional analysis of the representation could be done.
You say that any off the shelf classifier can be used to make the prediction based on embedding, what are the actual classifiers you used in your experiments?
Hello, we used a neural network in our experiments. The same classifier (using the same set of parameters) was used with all the evaluated methods. This information can be found in the latter part of section 3.2 of the paper.
For someone to reproduce your experiment it would be helpful to give more details -- how big was the network (how many units in each layer), which parameters was the grid search used for.
Thank you for your response, we have updated the paper to include information about the parameters we tested. This can be found in sec. 3.2.
Authors take the skip-graph architecture (Kiros 2015) and apply it to classifying labeled graphs (molecular graphs). They do it by creating many sentences by walking the graph randomly, and asking the model to predict previous part and next part from the middle part. Activations of the decoder part of this model on a walk generated from a new graph are used as features for a binary classifier use to predict whether the molecule has anti-cancer properties.
Paper is well written, except that evaluation section is missing details of how the embedding is used for actual classification (ie, what classifier is used)
Unfortunately I'm not familiar with the dataset and how hard it is to achieve the results they demonstrate, that would be the important factor to weight on the papers acceptance.
Thank you for your comments. Discussion about the classifier we used can be found in section 3.2 of our paper. We used a neural network classifier. The same classifier, optimized over the same set of parameters, was used to evaluated all the compared methods.
We do not claim that the method achieves absolute best performance on the tested datasets.
However, we compared our approach against methods that have been shown to achieve state-of-the-art results on the type of graphs (molecular) that we used for evaluation. The results show that the method is competitive against these methods.
This paper studies the graph embedding problem by using the encoder-decoder method. The experimental study on real network data sets show the features extracted by the proposed model is good for classification.
Strong points of this paper:
1. The idea of using the methods from natural language processing to graph mining is quite interesting.
2. The organization of the paper is clear
Weak points of this paper:
1. Comparisons with state-of-art methods (Graph Kernels) is missing.
2. The problem is not well motivated, are there any application of this. What is the different from the graph kernel methods? The comparison with graph kernel is missing.
3. Need more experiment to demonstrate the power of their feature extraction methods. (Clustering, Search, Prediction etc.)
4. Presentation of the paper is weak. There are lots of typos and unclear statements.
5. The author mentioned about the graph kernel things, but in the experiment they didn't compare them.
Also, only compare the classification accuracy by using the proposed method is not enough.
Thanks a lot for the invaluable comment. As suggested by reviewer2, we have added a comparison with DeepWalk (the embedding method) in Table 2.
= Response to Comments 1, 2 and 5: (difference between this paper and graph kernel) =
I would like to clarify the major differences between our work and graph kernel methods.
Although recent methods on graph kernels achieved great performances on graph data, but they are actually focused on graph kernel problems, instead of the graph embedding problem. Graph kernel methods are focused on learning a kernel/similarity function between a Pair of graph objects.  Our method is learning a feature embedding for each graph object.
For example, in figure 4, we showed a graph embedding result, which cannot be obtained through graph kernel methods. This paper is the first deep learning approach to graph embedding problem.
Although our graph embedding can potentially be fed into a kernel/similarity function in order to construct a graph kernel and to compare with graph kernel methods, that would be slightly beyond the scope of graph embedding problem of this paper.
But we would be happy to add this comparison if this is crucial.
= Response to Comment 3: (more experiments on clustering, search, prediction etc. tasks) =
We plan to extend the experiments by using the embedding for graph clustering and graph search tasks in future works/journal extension.  Due to the page limit of ICLR, it is not easy to add these additional experiments without deleting some existing results in the paper. In the existing results, we have demonstrated the power of the embedding through Prediction tasks and visualization of the embedding.
= Response to Comment 4: (typo and unclear statements) =
We regret if there is any typo or unclear statement in the paper, and we will proof-read again and correct them quickly in the next revision.
Learning an embedding is equivalent to learning a kernel. See literature on metric embeddings. I am not sure if this is the first method for learning embeddings of graphs. See the Deep Graph Kernel paper
http://dl.acm.org/citation.cfm?id=2783417
Dear reviewers, thank you very much for all the insightful comments and suggestions. Please find below a summary of our response to each of the points that were raised. Please note that we have paraphrased some of the comments and combined similar ones.
Anonymous Reviewer 2:
Comment 1. Use the same classifier for all the compared methods.
We have updated the results in our paper to reflect the changes from using a common classifier for all the compared methods. The changes can be found in section 3.2.
Comment 2. Use mean- or max-pooling to aggregate the embeddings learned by methods such as DeepWalk or node2vec and include this as a baseline.
The paper has been updated to include results from Deepwalk which was added as an additional baseline. Changes were made to sections 3.2 and 3.3.
Anonymous Reviewer 3:
Comment 3. Provide more information about the exact values of the parameters that were used in the grid search.
The information have been added to the paper. This can be found in section 3.2.
Anonymous Reviewer 1:
Comment 4. Need more experiment to demonstrate the power of their feature extraction methods. (Clustering, Search, Prediction etc.). Only comparing the classification accuracy by using the proposed method is not enough.
We plan to extend the experiments by using the embedding for graph clustering and graph search tasks in future works/journal extension. Due to the suggested page limit of ICLR, it is not easy to add these additional experiments without deleting some existing results in the paper. In the existing results, we have demonstrated the power of the embedding through prediction tasks and visualization of the embedding.
Comment 5. Presentation of the paper is weak. There are lots of typos and unclear statements.
The latest version of the paper has been proofread by several individuals and to the best of our knowledge we have removed the major typos and/or unclear statements.
Comment 6. What is the difference from graph kernel methods? The comparison with graph kernel is missing.
We agree that the problem studied in the deep graph kernel paper seems to be very similar, or even the same, to the one we are studying.
However, the underlying approach is slightly different and we argue that that in itself is a good motivation for the work as there has not been too many work published in the area. We use an encoder-decoder model which has not been used, and for one we can identify functionally similar subgraphs.
Regrettably, due to time constraints we are unable to add a comparison with Deep Graph Kernels.
However, we have tested against methods that have been shown to achieve good results on the type of graphs we used (ECFP and NeuralFPS) and we have also tested against DeepWalk. The introduction section has been updated to include some discussion on this.
Anonymous Reviewer 4:
Comment 7. Comparisons with recent work like LINE and node2vec are missing. You can compare them easily by applying the same aggregation strategy to their node embeddings.
We have already addressed the comment of a previous reviewer and have included DeepWalk as a baseline. We did not add comparisons against node2vec and LINE as these, in general, belong to the same family of methods.
Comment 8. As the current datasets are small (e.g., the average number of nodes per graph is around 30), it would be great to explore larger graph datasets to further investigate the method.
To compensate in a way, we tested the method on four different molecular graph datasets. Regrettably, we are unable to add experiments on more datasets at the moment.
Comment 9. The description about how to split the random walk sequence into 3 sub-sequences is missing.
Also, the line “l_min >= (n_k - 1), … >= l_max” in section 2.2.2 is a mistake.
Section 2.2.2 has been updated to correct this. Thank you very much for pointing this out.

Meta-review:
role: PC
data: comment: The idea of applying skip-graphs to this graph domain to learn embeddings is good. The results demonstrate that this approach is competitive, but do not show a clear advantage. This is difficult, as the variety of approaches in this area is rapidly increasing. But comparisons to other methods could be improved, notably deep graph kernels.

Decision: Reject