Title: Structured Sequence Modeling with Graph Convolutional Recurrent Networks
Authors: Youngjoo Seo
Michaël Defferrard
Pierre Vandergheynst
Xavier Bresson
Venue: ICLR 2017
Keywords: Structured prediction

Abstract: This paper introduces Graph Convolutional Recurrent Network (GCRN), a deep learning model able to predict structured sequences of data. Precisely, GCRN is a generalization of classical recurrent neural networks (RNN) to data structured by any arbitrary graph. Such structured sequences can be series of frames in videos, spatio-temporal measurements on a network of sensors, or random walks on a vocabulary graph for natural language modeling.The proposed model combines convolutional neural networks (CNN) on graphs to identify spatial structures and RNN to find dynamic patterns. We study two possible architectures of GCRN, and apply the models to two practical problems: predicting moving MNIST data, and modeling natural language with the Penn Treebank dataset. Experiments show that exploiting simultaneously graph spatial and dynamic information about data can improve both precision and learning speed.

TL;DR: This paper introduces a neural network to model graph-structured sequences

Review discussion:
I have some questions about section 5.2
1. It seems that you used the same dataset (and split) as in Zaremba et al. (2014).
But in Zaremba et al. experiment there are 929k training words, 73k validation words, and 82k test words whereas you reported 887k/70k/78k. Can you explain the difference?
2. I understand that the input at each time step x_t is a n x d matrix where n is the vocabulary size and d is the word vector dimensions. How did you construct x_t for language modelling?
3. Because the graph is constructed based on word2vec, you can't fine tune the word embeddings, right? (otherwise the graph will change after each update)
4. how did you tune the hyper-parameters for those models reported? Did you employ early-stop to halt all training after 8 epochs?
5. Table 2 shows perplexities of those models. The LSTM has a much higher perplexity compared to the LSTM reported in Zaremba et al (263.85 vs 114.5 of non-regularized LSTM). Can you explain why?
1. Thank you for the question. We checked and realized that we did a mistake during the preparation of the datasets, we basically missed the end-of-sentence. The mistake propagated to the perplexity results of all models. We fixed it, re-ran the experiments, and rewrote significantly Section 5.2. We encourage the reviewer to look at it.
2. We used two representations for the word x_t: (i) a word vector of dimension 200 and (ii) a one-hot code of dimension 10,000. (i) is only used for the model introduced by Zaremba et al. For our model, which takes as input a vocabulary graph of 10,000 nodes, we use (ii), which is then a signal on the vocabulary graph. For comparison, we also tested Zaremba et al. with (ii). We clarified this in the revision.
3. Your're right: the graph is fixed, and the embedding is only used to construct the graph. The network's input is that fixed graph and a time-series of one-hot vectors.
4. We did not tune the hyper-parameters but followed those of the small configuration of:
https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/models/rnn/ptb/ptb_word_lm.py
which are based on the paper of Zaremba et al. (2014).
5. The high perplexity was due to the mistake describe in 1. We are now able to reproduce Zaremba et al.
Thanks to your comment, we fixed a mistake and the presentation of that experiment should be clearer.
1) is it fair to say that the proposed model is the same as that of Shi et al. (2015), except the convolutions are generalized to graph convolutions exactly as those in Defferrard et al. (2016), so that non-grid data is allowed?
Except it seems that the model is not strictly more powerful as it treats all edges equally and can't distinguish between them, if my understanding is correct.
2) Is there a reason why Model 1 is not evaluated on Moving MNIST?
3) Can the authors clarify how the actual prediction is made, for example for Moving MNIST? I assume y is bernoulli based on linear projection of h_t?
Thanks for your questions!
1. Model 2 is exactly what you describe. You are right that our spectral filters are isotropic, and thus do not distinguish between edges. All edges are indeed treated equally (they could be treated differently by having one graph per type of edge for example). This naturally makes sense on graphs with only one type of edge, like the vocabulary graph used in Section 5.2. On images however, it is not evident whether this is good or not. That's exactly what we wanted to test in Section 5.1: could our isotropic filters perform better than 2D filters on video prediction ? And the answer is yes.
Moreover, our spectral filters require much less parameters than 2D filters for the same diameter, e.g. 5x5=25 corresponds to K=3, 9x9=81 corresponds to K=5.
2. No particular reason. For a fair comparison with Shi et al. (2015), we used the same architecture as them, which is Model 2.
3. On moving MNIST, we followed the same setup as Shi et al. (2015), only replacing the 2D convolution by a graph convolution.
So the actual prediction is given by a 1x1 convolution on h_t. On PTB, the chosen word is the one with highest probability.
I have few questions about the penn treebank experiment,
1) I am not sure to understand what is the input used? Are you using the same input for the RNN/LSTM/GRU baselines and the Model1/Model2 variations?
2) What is the number of parameters of each model?
3) I also share the same interrogations than reviewers 3 for the model hyperparameters and the high perplexities of the baseline.
Thanks for your questions!
1. We used two representations for a word: (i) a word vector of dimension 200 and (ii) a one-hot code of dimension 10,000. (i) is only used for the model introduced by Zaremba et al. For our model, which takes as input a vocabulary graph of 10,000 nodes, we use (ii), which is then a signal on the vocabulary graph. For comparison, we also tested Zaremba et al. with (ii). We modified Table 2 to state the input representation for clarification.
2. We added the number of parameters of each model in Table 2.
3. Please see our answers to reviewer 3.
The authors address the problem of modeling temporally-changing signal on a graph, where the signal at one node changes as a function of the inputs and the hidden states of its neighborhood, the size of which is a hyperparameter. The approach follows closely that of Shi et al. 2015, but it is generalized to arbitrary graph structures rather than a fixed grid by using graph convolutions of Defferrard et al. 2016. This is not a strict generalization because the graph formulation treats all edges equally, while the conv kernels in Shi et al. have a built in directionality. The authors show results on a moving MNIST and on the Penn Tree Bank Language Modeling task.
The paper, model and experiments are decent but I have some concerns:
1. The proposed model is not exceptionally novel from a technical perspective. I usually don't mind if this is the case provided that the authors make up for the deficiency with thorough experimental evaluation, clear write up, and interesting insights into the pros/cons of the approach with respect to previous models. In this case I lean towards this not being the case.
2. The experiment results section is rather terse and light on interpretation. I'm not fully up to date on the latest of Penn Tree Bank language modeling results but I do know that it is a hotly contested and well-known dataset. I am surprised to see a comparison only to Zaremba et al 2014 where I would expect to see multiple other results.
3. The writing is not very clear and the authors don't make sufficiently strong attempt to compare the models or provide insight or comparisons into why the proposed model works better.
In particular, unless I'm mistaken the word probabilities are a function of the neighborhood in the graph. What is the width of this graph?
For example, suppose I sample a word in one part of the graph, doesn't this information have to propagate to the other parts of the graph along the edges?
Also, it's not clear to me how the model can achieve reasonable results on moving MNIST when it cannot distinguish the direction of the moving edges. The authors state this but do not provide satisfying insight into how this can work. How does a pixel know that it should turn on in the next frame? I wish the authors thought about this more and presented it more clearly.
In summary, the paper has somewhat weak technical contribution, the experiments section is not very thorough, and the insights are sparse.
Thanks for your review.
The Model1 that we used in PTB experiment uses graph convolution that gives filtered output of input. And the filtered output has activation on possible candidate nodes around given an input word node.
Therefore, it is right that the input information propagation along with the edges.
For the moving MNIST experiment, the isotropic property of our graph convolution may not capture the fine structure of the image, but it is easier to capture the direction of its moving. We will update the paper by showing the learned filter and feature maps.
This paper investigates the  modeling of graph sequences . Authors propose Graph Convolutional Recurrent Networks (GRCN)  that extends convLSTM  (Shi et al. 2015) for data having an unregular graph structure at each timestep. They replace the 2D convolution with a graph convolutional operator from (Defferrad et al., 2016).
Authors propose two variations of the GRCN model. In Model 1,  the graph convolution is only applied on the input data. In Model 2, the graph convolution  is applied on both  input data and the previous hidden states. They evaluate their approaches on two different tasks, video generation using the movingMNIST dataset and world-level language modelling using Penntreebank.
On movingMNIST authors show that their GRCN 2 improves upon convLSTM.
However, they evaluate only with one-layer convLSTM, while Shi et al. report better results with 3 layers (also not as good as  GRCN) . It would be nice to evaluate GCRCN in that setting as well.
While the authors show an improvement of GRCN relatively to convLSTM, GRCN on this task seems relatively weak compared to recent works such as the Video Pixel Networks (Kalchbrenner et al., 2016). It contradicts the claim that "Model 2 has shown good performance in the case of video prediction" in the conclusion.
For the Penntreebank experiments, author compares  their model 1 with FC-LSTM, with or without dropout.
However, the results in (Zaremba et al., 2014) still seems different than the one reported here. In (Zaremba et al., 2014), they  reports a test perplexity of 78.4 for the large regularized LSTM in their table 1 which outperforms the score of the GRCN.
Also, following works such as variational dropout or zoneout have since improve upon Zaremba results. Is there some differences in the experimental setting?  It would be nice to have results that are directly comparable to previous work.
Pros:
- Interesting model,
Cons:
- Overall, the proposed contribution is relatively incremental compared to (Shi et al. 2015) and (Defferrad et al., 2016).
- Weak results of GRCN relatively to previous works in the experiments, that do not convince of the GRCN advantages.
Thanks for your review.
We recently noticed that the paper Video Pixel Networks(Kalchbrenner et al., 2016).  And surprised by the results.
However, in our experiment, the main purpose is to show that the GRCN works similar or better than the traditional convolutional method on 2D grid structure data.
Therefore, we thought that it is not necessary to compare with state-of-the-art results on moving MNIST.
For the experiment on PTB, we compared with small settings that correspond to non-regularized LSTM experiment setting in Zaremba et al., 2014 to check the simplest setting first. We will conduct a further experiment on PTB with various configurations.
The paper proposes to combine graph convolution with RNNs to solve problems in which inputs are graphs. The two key ideas are: (i) a graph convolutional layer is used to extract features which are then fed in an RNN, and (ii) matrix multiplications are replaced by graph convolution operations. (i) is applied to language modelling, yielding lower perplexity on Penn Treebank (PTB) compared with LSTM. (ii) outperformed LSTM + CNN on the moving-MNIST.
Both two models/ideas are actually trivial and in line with the current trend of combining different architectures.
For instance, the idea of replacing matrix multiplications by graph convolution is a small extension for Shi et al.
Regarding to the experiment on PTB (section 5.2), I'm skeptical about the way the experiment carried out. The reason is that, instead of using the given development set to tune the models, the authors blindly used an available configuration which is for a different model.
Pros:
- good experimental results
Cons:
- ideas are quite trivial
- the experiment on PTB was carried out improperly
Thanks for your review.
The main reason why we keep the same experimental settings of Zaremba et al.,on PTB experiment is for fair comparison and checking the difference.
But I agree that we need to find our own best experimental setting and show it together. We will update the PTB results with best learning schedule and settings.

Meta-review:
role: PC
data: comment: While graph structures are an interesting problem, as the reviewers observed, the paper extends previous work incrementally and the results are not very moving.
 
 pros
 - interesting problem space that has not been thoroughly explored
 cons
 - experimental evaluation was not convincing enough with the results.
 - the method itself is a small incremental improvement over prior papers.

Decision: Reject