Title: GEOSEQ2SEQ: INFORMATION GEOMETRIC SEQUENCE-TO-SEQUENCE NETWORKS

Abstract: The Fisher information metric is an important foundation of information geometry, wherein it allows us to approximate the local geometry of a probability distribution. Recurrent neural networks such as the Sequence-to-Sequence (Seq2Seq) networks that have lately been used to yield state-of-the-art performance on speech translation or image captioning have so far ignored the geometry of the latent embedding, that they iteratively learn. We propose the information geometric Seq2Seq (GeoSeq2Seq) network which abridges the gap between deep recurrent neural networks and information geometry. Specifically, the latent embedding offered by a recurrent network is encoded as a Fisher kernel of a parametric Gaussian Mixture Model, a formalism common in computer vision. We utilise such a network to predict the shortest routes between two nodes of a graph by learning the adjacency matrix using the GeoSeq2Seq formalism; our results show that for such a problem the probabilistic representation of the latent embedding supersedes the non-probabilistic embedding by 10-15%.

1. INTRODUCTION:

Information geometry situates itself in the intersection of probability theory and differential geometry, wherein it has found utility in understanding the geometry of a wide variety of probability models (Amari & Nagaoka, 2000) . By virtue of Cencov's characterisation theorem, the metric on such probability manifolds is described by a unique kernel known as the Fisher information metric. In statistics, the Fisher information is simply the variance of the score function. In Bayesian statistics, it has found utility in terms of Riemannian Markov Chain Monte Carlo (MCMC) methods (Girolami & Calderhead, 2011) while for computer vision it has resulted in the Fisher kernel encoding (Perronnin & Dance, 2006) . Practitioners have also used the geometric make-up of feature vectors obtained from a deep convolutional neural network (dCNN) to rank images (Qian et al., 2017 ) by encoding them using Fisher kernels. Apart from traditional signal processing methodologies like Kalman filters or Hidden Markov Models, recurrent neural networks that have proved to be beneficial for sequential data haven't quite utilised the geometry of the latent structure they learn.

There are two paths of an intersection of recurrent networks with Riemann geometry. The first lies in using the natural gradient to optimize loss functions of deep neural networks (Pascanu & Bengio, 2013) . This affords invariance to the optimization procedure by breaking the symmetry in parameter space. The other is utilizing the geometry of the latent space to augment classification accuracy. In this paper, we combine a specific sort of recurrent network -the Sequence-to-Sequence (Seq2Seq) model -and utilize the Riemann geometry of the embedded space for boosting the performance of the decoder. We test the algorithm on a combinatorially hard problem called the shortest route problem. The problem involves a large graph wherein the shortest route between two nodes in the graph are required. Specifically, we use a meta-heuristic algorithm (a vanilla A * algorithm) to generate the shortest route between two randomly selected routes. This then serves as the training set for our GeoSeq2Seq network.

2. RELATED WORKS:

Recently, the research direction of combining deep learning with methods from information geometry has proven to be an exciting and a fertile research area. In particular, natural gradient methods in deep learning have recently been explored to model the second-order curvature information. For example, Natural Neural Networks (Desjardins et al., 2015) have sped up convergence by adapting their internal representation during training to improve the conditioning of the Fisher information matrix. On the other hand, it is also possible to approximate the Fisher information matrix, either with a Gaussian graphical model, whose precision matrix can be computed efficiently (Grosse & Salakhudinov, 2015) or by decomposing it as the Kronecker product of small matrices, which capture important curvature information (Grosse & Martens, 2016) .

More closely related to the topic of this paper, Fisher vector encodings and deep networks have been combined for image classification tasks (Sydorov et al., 2014) . For example, Fisher vector image encoding can be stacked in multiple layers (Simonyan et al., 2013b) , showing that convolutional networks and Fisher vector encodings are complementary. Furthermore, recent work has introduced deep learning on graphs. For example, Pointer Networks (Vinyals et al., 2015) use a Seq2Seq model to solve the travelling salesman problem, yet it assumes that the entire graph is provided as input to the model.

To the best of our knowledge, there has been very little work on using the Fisher vectors of a recurrent neural encoding, generated from RNNs (recurrent neural networks), LSTMs (Long shortterm memory) and GRUs (Gated recurrent units) based sequence-to-sequence (Seq2Seq) models. Therefore, the work presented here is complementary to the other lines of work, with the hope to increase the fidelity of these networks to retain the memory of long sequences.

3. METHODS:

In this section, we describe the data-sets, the procedure for generating the routes for training/test datasets, and the deployment of information geometric Sequence-to-Sequence networks that forms the novel contribution of this paper. All of the calculations were performed on a i7-6800K CPU @ 3.40GHz workstation with 32 GB RAM and a single nVidia GeForce GTX 1080Ti graphics card.

3.1. DATASETS:

The graph is based on the road network of Minnesota1 . Each node represents the intersections of roads while the edges represent the road that connects the two points of intersection. Specifically, the graph we considered has 376 nodes and 455 edges, as we constrained the coordinates of the nodes to be in the range [-97, -94] for the longitude and [46, 49] for the latitude, instead of the full extent of the graph, i.e., a longitude of [-97, -89 ] and a latitude of [43, 49] , with a total number of 2,642 nodes.

3.2. ALGORITHMS THE A * META-HEURISTICS:

The A * algorithm is a best-first search algorithm wherein it searches amongst all of the possible paths that yield the smallest cost. This cost function is made up of two parts -particularly, each iteration of the algorithm consists of first evaluating the distance travelled or time expended from the start node to the current node. The second part of the cost function is a heuristic that estimates the cost of the cheapest path from the current node to the goal. Without the heuristic part, this algorithm operationalises the Dijkstra's algorithm (Dijkstra, 1959) . There are many variants of A * ; in our experiments, we use the vanilla A * with a heuristic based on the Euclidean distance. Other variants such as Anytime Repairing A * has been shown to produce superior performance (Likhachev et al., 2004) .

Paths between two nodes selected uniformly at random are calculated using the A * algorithm. On an average, the paths are 19 hops long. The average fan-in/fan-out of a randomly selected node is 2.42.

We increase the combinatorial difficulty of the shortest route by not constraining the search to the local fan-out neighbourhood, rather the dimension of the search space is n -1 with n representing the number of nodes in the graph.

RECURRENT DEEP NETWORKS:

We utilised Sequence-to-Sequence (Seq2Seq, Sutskever et al. (2014) ) recurrent neural networks for the shortest route path prediction. Specifically, we use the following variants:

• An LSTM2RNN, where the encoder is modelled by a long short term memory (LSTM, Hochreiter & Schmidhuber (1997) ), i.e.

EQUATION

while the decoder is a vanilla RNN (Goodfellow et al., 2016) , i.e.

EQUATION

followed by a softmax output layer, i.e. y(t) = logsoftmax(Ch(t) + c),

(3) which gives the probability distribution on the following node, choosing it among the other n -1 nodes.

• A GRU2RNN, where the encoder is modelled by a gated recurrent unit (GRU, Cho et al. (2014) ), i.e.

EQUATION

while the decoder is again a vanilla RNN with a softmax, as in Equations ( 2)-(3). • An LSTM2LSTM, where both the encoder and the decoder are modelled by an LSTM as in Equations (1). • A GRU2LSTM, where the encoder is a GRU (see Eqn. ( 4)) and the decoder is an LSTM (see Eqn. ( 1)). • GeoSeq2Seq, our novel contribution, where the context vector obtained as in one of the previous models is further encoded using either Fisher vectors or vectors of locally aggregated descriptors (VLAD; see Figure 1 ), as described in the following section.

For all networks, the input is represented by the [source, destination] tuple (Figure 1 ), which is encoded in a context vector (W ) and subsequently decoded into the final sequence to obtain the shortest path connecting the source to the destination. Moreover, during the test phase, we compute two paths, one from the source to the destination node and the other from the destination to the source node, that forms an intersection to result in the shortest path. 

INFORMATION GEOMETRY:

Classical results from information geometry (Cencov's characterisation theorem; Amari & Nagaoka (2000) ) tell us that, for manifolds based on probability measures, a unique Riemannian metric exists -the Fisher information metric. In statistics, Fisher-information is used to measure the expected value of the observed information. Whilst the Fisher-information becomes the metric (g) for curved probability spaces, the distance between two distributions is provided by the Kullback-Leibler (KL) divergence. It turns out that if the KL-divergence is viewed as a curve on a curved surface, the Fisher-information becomes its curvature. Formally, for a distribution q(ψ) parametrised by θ we have, First, a set of D dimension context vector is extracted from a sequence and denoted as W = w 1 , ...w i , ..., w N : w ∈ R D . A K component GMM with diagonal covariance is generated (Simonyan et al., 2013a; Cimpoi et al., 2015) on the training set with the pa- 

KL(θ, θ ) = E θ log q(ψ |θ) q(ψ |θ ) + E θ log q(ψ |θ ) q(ψ |θ) = dθ T g(θ)dθ + O(dθ 3 ) g ij (θ) = +∞ -∞ q(ψ, θ) ∂ ln q(ψ, θ) ∂θ i ∂ ln q(ψ, θ) ∂θ j dψ KL[θ 0 + δθ : θ 0 ] . = 1 2 g ij (θ 0 )(δθ) 2 .

rameters {Θ = (ω k , µ k , Σ k )} K k=1 ,

EQUATION

For each component k, mean and covariance deviation on each vector dimension j = 1, 2...D are

EQUATION

where q ik is the soft assignment weight of feature w i to the k th Gaussian and defined as

EQUATION

)

Just as the sequence representation, the dimension of Fisher vector is 2KD, K is the number of components in the GMM, and D is the dimension of local context descriptor. After l 2 normalization on Fisher vector, the embedding can be learnt using an arbitrary recurrent neural network based decoder (Perronnin et al., 2010) . In our experiments (see Section 4), as a proof of concept, we fixed the number of GMMs to K = 1, since more Gaussians would have increased the dimension acted upon by the decoder, making the training computational time prohibitive (for D = 256, choosing K ≥ 2 implies a Fisher vector's dimension greater than a thousand).

VLAD encoding In this case, the context vectors are represented as Vector of Locally Aggregated Descriptors (VLAD, Jégou et al. (2010) ; Arandjelovic & Zisserman (2013) ). VLAD is a feature encoding and pooling method, similar to Fisher Vectors. It encodes a set of local feature descriptors extracted from a sequence and denoted as W = (w 1 , . . . , w n : w ∈ R D ), using a clustering method such as K-means. Let q ik be the hard assignment of data vectors w i to the k th cluster, such that q ik ≥ 0 and K k=1 q ik = 1. Furthermore, the assignments of features to dictionary elements must be pre-computed, for example by using KD-trees (Beis & Lowe, 1997; Silpa-Anan & Hartley, 2008) . VLAD encodes features W by considering the residuals

EQUATION

where µ k ∈ R N is the k th cluster centre (or mean). Then, these residuals are stacked together to obtain the final context vector.

GeoSeq2Seq Our novel contribution is to couple a Seq2Seq model with information geometric representation of the context vector. To do so, we start with a general Seq2Seq trained model. For each source-destination tuple, we use this encoder to compute the context vectors, which have identical same size (i.e., either 256 or 512 as specified in the Results section). Then, we train the GMM on the context vectors obtained from the training sequences and finally use the means and variances to construct the Fisher Vectors. Similarly, for the VLAD-based approach, we train the centres and assignments from K-means and KD-trees on the context vectors obtained from the training sequences.

We then use them to generate the VLAD encoding. Subsequently, we use the new context vectors encoded with FV/VLAD as an initial hidden state to train the decoder, finally providing us with the shortest path as output.

As the work of Sydorov et al. (2014) suggests it is possible to learn the Fisher kernel parameters in an end-to-end manner for a convolutional neural network. A future development for this current work will be to inquire whether end-to-end training of Fisher encoding for Seq2Seq models can be attained.

4. RESULTS:



4.1. SHORTEST PATH PROBLEM:

For the graph of Minnesota with n = 376 nodes and 455 edges, we generated 3,000 shortest routes between two nodes picking them uniformly at random) and using the A * algorithm with a heuristic based on the Euclidean distance. We used these routes as the training set for the Seq2Seq algorithms using a 67-33% training-test splits; we train the network for 400 epochs, updating the parameters with an Adam optimisation scheme (Kingma & Ba, 2014) , with parameters β 1 = 0.9 and β 2 = 0.999, starting from a learning rate equal to 10 -3 . Since Fisher encoding requires a double length for the context vector (it considers means and covariances, see Eqn. ( 5)), we compared it to a basic Seq2Seq with 256 and 512 hidden units. On the other hand, VLAD encodes only the residual (see Eqn. ( 8)), therefore we instantiate it with 256 and 512 hidden units. Moreover, in the decoder, the following node in the predicted sequence is computed choosing among all the other n -1 nodes from a softmax distribution. If we stop the training early, for example, after only 40 epochs (see Figure 2 ), the network cannot reproduce possible paths (see Figure 3 on the left-hand side). On the other hand, if we train the network for more epochs, the training converges after 400 epochs (green dot in Figure 2 ). Therefore, as shown in Figure 3 (right), if we compute two paths, one starting from the source (magenta dash-dotted line) and the other from the destination (cyan), intersecting them allows us to predict the shortest path (red). This is equal to the one generated by A * (green). This means that the network is capable of learning the adjacency matrix of the graph.

Then, comparing different approaches, the prediction accuracy on the test data-set is reported in Table 1 . As we can see, for what concerns the RNN decoder, doubling the hidden state dimension marginally increases the percentage of shortest paths (1%) and the successful paths, that are not necessarily the shortest (0.2% and 1.6% for GRU and LSTM encoders, respectively). Our proposed information geometry based Fisher encoding achieves an increased accuracy in finding the shortest paths (56% and 60% for LSTM and GRU, respectively). Furthermore, if a VLAD encoding is employed, GRU networks have a higher approximation capability with more than 65% of accuracy on the shortest paths and 83% on the successful cases. Similarly, if the decoder is an LSTM, the accuracies on the shortest paths of simple Seq2Seq models are around 50%, while our GeoSeq2Seq models with Fisher encoding we get close to 60% and even above with the VLAD encoding, achieving 65% (and 82% of successful paths) in the case of the LSTM encoder. This means that our probabilistic representation of the latent embedding supersedes the non-probabilistic one by 10-15%.

- Here, we show an example of a prediction of shortest path on the Minnesota dataset (blue graph). The A * shortest path between the source (black dot) and destination (yellow dot) nodes is represented in green. On the left-hand side, we show the prediction (magenta) after only 40 epochs of training (Figure 2 ). On the right-hand side, instead, we show the prediction at the end of the training: we compute two paths, one starting from the source (magenta) and the other from the destination (cyan), and finally, we intersect them to compute the predicted shortest path (red). The 'flying-object syndrome' only occurs during the earlier phase of training. In order to provide more generality to our method, in this section, we apply the GeoSeq2Seq model to solve algorithmic tasks similar to those on which Neural Turing Machines (NTM) were evaluated Graves et al. (2014) , albeit without a need for an external memory module. Specifically, we present results for a simple algorithmic task such as copying, and a more complex semantic task such as associative recall. For these tasks, we used an LSTM2LSTM with VLAD encoding and followed a configuration of hyper-parameters similar to the one in Graves et al. (2014) i.e. we used RMSProp algorithm with momentum set to 0.9, a learning rate equal to 1e-4 and clipped the gradient during the backpropagation to the range (-10,10). Copying task The copying task consists in remembering an eight bit random binary input sequence between a start flag and a stop flag and copying it in the output sequence after the stop flag (see Figure 4 ). As we can see, our method can copy both shorter (Figure 4a ) and longer sequences (Figure 4b ). We run 1e5 experiments as test sequences, with random lengths between 2 and 20. Just like NTMs all of them were reproduced at the output.

Method:

Figure 5 : Associative recall task. Two results for the associative recall task. On the left-hand side (a), we considered a sequence of 3 items, with the query of the second one. The third item is correctly repeated as output by our GeoSeq2Seq method. On the right-hand side (b), instead, we show a longer sequence (i.e. 9 items) with the query of the second one. The third item is repeated almost correctly: the first two columns are correct, while the last one misses some components.

Associative recall task The associative recall task consists in remembering an input sequence of item bounded on the left and right by a delimiter flag and repeating the item following the one between the two query flags in the output sequence (see Figure 5 ). We define an item as a sequence of three random binary columns. For 1e5 test experiments with a random length of the sequence between 2 and 12 items, we achieve an accuracy equal to 13.39%. This apparently low accuracy is because, as we can see, our GeoSeq2Seq can reproduce the correct next item (Figure 5a ) for shorter sequences, while it may fail in few components when the sequence becomes longer (Figure 5b ), which is considered an error, as well.

5. DISCUSSION:

This paper proposes the GeoSeq2Seq, an information geometric Seq2Seq architecture that utilises an information geometric embedding of the context vector. The RNN is tasked with learning the adjacency matrix of the graph such that at each decision step the search space becomes n -1, where n is the total number of nodes in the graph. Unlike algorithms like q-routing (Boyan & Littman, 1994) that constrains the search space to contain only the connected neighbours, our instantiation of the Seq2Seq operates under a larger search space. Indeed, the accuracy of our algorithm is expected to increase where we use neighbourhood information from the adjacency matrix. In summary, such a recurrent network shows increased fidelity for approximating the shortest route produced by a meta-heuristic algorithm.

Apart from encoding, context vector stacking using dual encoders have been proven to be beneficial (Bay & Sengupta, 2018) . Utilising homotopy based continuation has been a different line of work where the emphasis lies in smoothing the loss function of the recurrent network by convolving it with a Gaussian kernel (Bay & Sengupta, 2017) . All of these strategies have shown to improve the temporal memory of the recurrent network. This line of work is distinct from architecture engineering, that has placed emphasis on constructing exquisite memory mechanism for the recurrent networks. For example, Neural Turing Machines (Graves et al., 2014) are augmented recurrent networks that have a (differentiable) external memory that can be selectively read or written to, enabling the network to store the latent structure of the sequence. Attention networks, on the other hand, enable the recurrent networks to attend to snippets of their inputs (cf. in conversational modelling Vinyals & Le (2015) ). Similarly, it is also possible to embed the nodes of the graph as preprocessing step using methods such as -(a) DeepWalk (Perozzi et al., 2014) , that uses local information obtained from truncated random walks to encode social relations by treating walks as the equivalent of sentences, (b) LINE (Tang et al., 2015) , which optimises an objective function preserving both local and global structures, and (c) node2vec (Grover & Leskovec, 2016) , that learns continuous feature representations for nodes in networks. Nevertheless, for the long-term dependencies that shortest paths in large graphs inherit, these methods are small steps towards alleviating the central problem of controlling spectral radius in recurrent neural networks.

The 4-5% gain in accuracy for VLAD in contrast to Fisher encoding is a bit surprising. As a matter of fact, Fisher encoding should be more precise since it takes covariances into account. We notice that the covariances matrix (instantiated as a diagonal matrix due to computational constraint) has a high condition number. The condition number κ of a matrix M is defined as κ(M ) = M M -1 , or equivalently it is the ratio of the largest singular value of that matrix to its smallest singular value (Belsley et al., 2005) . Practically, it measures how much small variation in the input argument can propagate to the output value. In particular, we obtain a condition number equal to 2.69e3 and 1.70e3 for the LSTM and GRU encoders, respectively. We believe it is for this reason that the GRU encoder leads to a better accuracy than the LSTM encoder, as shown in Table 1 .

The use of Riemann geometry to encode context (feature) vector has a long history in computer vision (Srivastava & Turaga, 2015) , our work demonstrates yet another way to embed the curved geometry of the context vector for decoding. The Riemannian metric for a recurrent network can be evaluated in two ways -one where we describe the probability distribution over the entire sequence and another where we describe a conditional distribution at time i conditioned on time i -1. We anticipate that the latter is more suited to a dynamic scenario (where the structure of the graph may be slowly changing) while the former is more suitable for static graphs. Analytically, averaging over time and assuming ergodicity, both metric should be fairly close to one another, nonetheless, it is only with further experiments we can demonstrate the value of one over the other.

In this paper, we have constrained the use of the Riemannian metric on the encoding end, one can follow a similar treatise for the decoder. Together with architecture engineering (Graves et al., 

:

https://www.cs.purdue.edu/homes/dgleich/packages/matlab_bgl

Figures and Tables:

Figure 1: Information geometric context vectors. The context vectors (W) that are learnt using the recurrent neural networks are further encoded using either a Fisher vector based encoding or a Vector of Locally Aggregated Descriptors (VLAD) based encoding (see text). Such an encoded context vector is finally fed to a recurrent neural network based decoder.

encoding We use a Gaussian Mixture Model (GMM) for encoding a probabilistic sequence vocabulary (W ) on the training dataset. The context vectors are then represented as Fisher Vectors (FV, Perronnin & Dance (2006)) -derivatives of log-likelihood of the model with respect to its parameters (the score function). Fisher encoding describes how the distribution of features of an individual context differs from the distribution fitted to the feature of all training sequences.

Figure 2: Training error. We illustrate the negative log-likelihood loss function during the training phase. The adjacency matrix is iteratively learnt during such a training phase. We highlight the situation after 40 epochs (red dot) and at the end of the training (after 400 epochs, green dot) when the back-propagation through time algorithm has converged. The paths are shown in Figure 3.

Figure3: Predicted path. Here, we show an example of a prediction of shortest path on the Minnesota dataset (blue graph). The A * shortest path between the source (black dot) and destination (yellow dot) nodes is represented in green. On the left-hand side, we show the prediction (magenta) after only 40 epochs of training (Figure2). On the right-hand side, instead, we show the prediction at the end of the training: we compute two paths, one starting from the source (magenta) and the other from the destination (cyan), and finally, we intersect them to compute the predicted shortest path (red). The 'flying-object syndrome' only occurs during the earlier phase of training.

Figure 4: Copying task. Two results for the copying task. On the left-hand side (a), we considered a binary sequence of length 4 bounded by the start flag and the stop one. The sequence is correctly repeated as output by our GeoSeq2Seq method. On the right-hand side (b), instead, we show a longer sequence (i.e. of length 19), which is correctly copied too.

Results on the Minnesota graph. Percentage of shortest path and successful paths (that are not necessarily shortest) are shown for a wide-variety of Seq2Seq models, with context vector dimension equal to either 256 or 512. All scores are relative to an A * algorithm, that achieves a shortest path score of 100%.

Table content: <table><tr><td/><td colspan="2">Shortest Successful</td></tr><tr><td>LSTM2RNN (256)</td><td>47%</td><td>69.5%</td></tr><tr><td>LSTM2RNN (512)</td><td>48%</td><td>71.1%</td></tr><tr><td>GRU2RNN (256)</td><td>48.3%</td><td>73.1%</td></tr><tr><td>GRU2RNN (512)</td><td>49%</td><td>73.3%</td></tr><tr><td>LSTM2RNN with FV</td><td>56%</td><td>76.4%</td></tr><tr><td>GRU2RNN with FV</td><td>60.1%</td><td>79%</td></tr><tr><td>LSTM2RNN with VLAD (256)</td><td>59.3%</td><td>76.8%</td></tr><tr><td>LSTM2RNN with VLAD (512)</td><td>60.1%</td><td>76.2%</td></tr><tr><td>GRU2RNN with VLAD (256)</td><td>64%</td><td>79.9%</td></tr><tr><td>GRU2RNN with VLAD (512)</td><td>65.7%</td><td>83%</td></tr><tr><td>LSTM2LSTM (256)</td><td>50.3%</td><td>72.7%</td></tr><tr><td>LSTM2LSTM (512)</td><td>54%</td><td>73.3%</td></tr><tr><td>GRU2LSTM (256)</td><td>48%</td><td>69.5%</td></tr><tr><td>GRU2LSTM (512)</td><td>48.2%</td><td>73.5%</td></tr><tr><td>LSTM2LSTM with FV</td><td>57.9%</td><td>78.8%</td></tr><tr><td>GRU2LSTM with FV</td><td>59%</td><td>79.3%</td></tr><tr><td>LSTM2LSTM with VLAD (256)</td><td>62.4%</td><td>79.8%</td></tr><tr><td>LSTM2LSTM with VLAD (512)</td><td>65.2%</td><td>82.1%</td></tr><tr><td>GRU2LSTM with VLAD (256)</td><td>63%</td><td>80.6%</td></tr><tr><td>GRU2LSTM with VLAD (512)</td><td>63.7%</td><td>81.6%</td></tr></table>

Peer review (iclr2018):
 The paper proposes a method for augmenting sequence-to-sequence (seq2seq) methods with Fisher vector encodings, allowing the decoder to better model the geometric structure of the embedding space. Experiments are performed on a shortest-route problem, where augmenting standard seq2seq architectures with Fisher vectors improves performance. Pros: - Combining deep learning with methods from information geometry is an interesting direction for research - Method is a generic drop-in replacement for improving any seq2seq architecture - Experimental results show modest performance improvements over vanilla seq2seq methods Cons: - Missing references for prior work combining information geometry and deep learning - Insufficient explanation of the method - Only experimental results are a nonstandard route-finding task - Missing references and baselines for prior work on deep learning on graphs The general research direction of combining deep learning with methods from information geometry is an exciting and fertile area for interesting work. Unfortunately this paper fails to cite or discuss much recent work in this area; for example natural gradient methods in deep learning have recently been explored in [1, 2, 3]; more closely related to the topic of this paper, [4] and [5] have combined Fisher vector encodings and deep networks for image classification tasks. Although these prior methods do not consider the use of recurrent networks, the authors should discuss how their method compares to the approaches of [4] and [5]. The method is not described in sufficient detail. How exactly is the Fisher encoding combined with the recurrent neural network? In particular, how is GMM fitting interleaved with learning the RNN? Do you backpropagate through the GMM fitting procedure in order to jointly learn the RNN parameters and the GMM for computing Fisher encodings? Or is GMM fitting an offline step done once, after which the RNN decoder is learned on top of the Fisher encodings? The paper should clarify along these points. As a side note, it also feels a little disingenuous to describe the method in terms of GMMs, but to perform all experiments with K=1 mixture components; in this setting the GMM degrades to a simple Gaussian distribution. The proposed method could in theory be used as a drop-in replacement for seq2seq on any task. Given its generality, I am surprised at the nonstandard choice of route-finding in a graph of Minnesota roads as the only task on which the method is tested; as a minimum the method should have been tested on more than one graph. More generally, I would have liked to see the method evaluated on multiple tasks, and on more well-established seq2seq tasks so that the method could be more easily compared with previously published work. Strong results on machine translation would be particularly convincing; the authors might also consider algorithmic tasks such as copying, repeat copying, sorting, etc. similar to those on which Neural Turing Machines were evaluated. I am not sure that seq2seq is the best approach for the route-finding task. In particular, since the input is encoded as a [source, destination] tuple it has a fixed length; this means that you could use a feedforward rather than recurrent encoder. The paper also fails to cite or discuss recent work involving deep learning on graphs. For example Pointer Networks [6] use a seq2seq model with attention to solve convex hull, Delaunnay Triangulation, and traveling salesman problems; however Pointer Networks assume that the entire graph is provided as input to the model, while in this paper the network learns to specialize to a single graph. In that case, the authors might consider embedding the nodes of the graph using methods such as DeepWalk [7], LINE [8], or node2vec [9] as a preprocessing step rather than learning these embeddings from scratch. From Table 1, seq2seq + VLAD significantly outperforms seq2seq + FV. Given these results, are there any reasons why one should use seq2seq + FV instead of seq2seq + VLAD? Overall I think that this paper has some interesting ideas. However, due to a number of missing references, unclear description of the method, and limited experimental results I feel that the paper is not ready for publication in its current form. References [1] Grosse and Salakhutdinov, “Scaling Up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix”, ICML 2015 [2] Grosse and Martens, “A Kronecker-factored approximate Fisher matrix for convolution layers”, ICML 2016 [3] Desjardins et al, “Natural Neural Networks”, NIPS 2015 [4] Simonyan et al, “Deep Fisher Networks for Large-Scale Image Classification”, NIPS 2013 [5] Sydorov et al, “Deep Fisher Kernels - End to End Learning of the Fisher Kernel GMM Parameters”, CVPR 2014 [6] Vinyals et al, “Pointer Networks”, NIPS 2015 [7] Perozzi et al, “DeepWalk: Online Learning of Social Representations”, KDD 2014 [8] Tang et al, “LINE: Large-scale Information Network Embedding”, WWW 2015 [9] Grover and Leskovec, “node2vec: Scalable Feature Learning for Networks”, KDD 2016