Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: The authors propose an input switched affine network to do character-level language modeling, a kind of RNN without pointwise nonlinearity, but with switching the transition matrix & bias based on the input character. This is motivated by intelligibility, since it allows decomposition of output contribution into these kappa_s^t terms, and use of basic linear algebra to probe the network.

Regarding myself as a reviewer, I am quite sure I understood the main ideas and arguments of this paper, but am not an expert on RNN language models or intelligibility/interpretability in ML.
I did not read any papers with a similar premise - closest related work I'm familiar with would be deconvnet for insight into vision-CNNs.

PRO:
I think this is original and novel work. This work is high quality, well written, and clearly is the result of a lot of work.
I found section 4.5 about projecting into readout subspace vs "computational" subspace most interesting and meaningful.

CON:
+ The main hesitation I have is that the results on both parts (ISAN model, and analysis of it) are not entirely convincing:
   (1) ISAN is only trained on small task (text8), not clear whether it can be a strong char-LM on larger scale tasks,
   (2) nor do the analysis sections provide all that much real insight in the learned network.

(1b) Other caveat towards ISAN architecture: this model in its proposed form is really only fit for small-vocabulary (i.e. character-based) language modeling, not a general RNN with large-vocab discrete input nor continuous input.

(2a) For analysis: many cute plots and fun ideas of quantities to look at, but not much concrete insights.
(2b) Not very clear which analysis is specific to the ISAN model, and which ideas will generalize to general nonlinear RNNs.
(2c) Re sec 4.2 - 4.3: It seems that the quantity \kappa_s^t on which analysis rests, isn't all that meaningful. Elaborating a bit on what I wrote in the question:
For example: Fig 2, for input letter "u" in revenue, there's a red spot where '_' character massively positively impacts the logit of 'e'. This seems quite meaningless, what would be the meaning of influence of '_' character? So it looks ot me that the switching matrix W_u (and prior W_n W_e etc) are using previous state in an interesting way to produce that following e. So that metric \kappa_s^t just doesn't seem very meaningful.
This remark relates to the last paragraph of Sec4.2.

Even though the list of cons here is longer than pro's, I recommend accept; specifically because the originality of this work will in any case make it more vulnerable to critiques. This work is well-motivated, very well-executed, and can inspire many more interesting investigations along these lines.
review_title: Original and creative work - hesitation about results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:  The authors present a simple RNN with linear dynamics for language modeling. The linear dynamics greatly enhance the interpretability of the model, as well as provide the potential to improve performance by caching the dynamics for common sub-sequences. Overall, the quantitative comparison on a benchmark task is underwhelming. It’s unclear why the authors didn’t consider a more common dataset, and they only considered a single dataset. On the other hand, they present a number of well-executed techniques for analyzing the behavior of the model, many of which would be impossible to do for a non-linear RNN. 

Overall, I recommend that the paper is accepted, despite the results. It provides an interesting read and an important contribution to the research dialogue. 

Feedback

The paper could be improved by shortening the number of analysis experiments and increasing the discussion of related sequence models. Some of the experiments were very compelling, whereas some of them (eg. 4.6) sort of feels like you’re just showing the reader that the model fits the data well, not that the model has any particularly important property. We trust that the model fits the data well, since you get reasonable perplexity results. 

LSTMS/GRUs are great for for language modeling for data with rigid combinatorial structure, such as nested parenthesis. It would have been nice if you compared your model to non-linear methods on this sort of data. Don’t be scared of negative results! It would be interesting if the non-linear methods were substantially better on these tasks. 

You should definitely add a discussion of Belanger and Kakade 2015 to the related work. They have different motivations (fast, scalable learning algorithms) rather than you (interpretable latent state dynamics and simple credit assignment for future predictions given past). On the other hand, they also have linear dynamics, and look at the singular vectors of the transition matrix to analyze the model. 

More broadly, it would be useful for readers if you discussed LDS more directly. A lot of this comparison came up in the openreview discussion, and I recommend folding this into the paper. For example, it would be useful to emphasize that the bias vectors correspond to columns of the Kalman gain matrix. 

One last thing regarding LDS: your model corresponds to Kalman filtering but in an LDS you can also do Kalman smoothing, where state vectors are inferred using the future in addition to the past observations. Could you do something similar in your model?

What if you said that each matrix is a sparse/convex combination of a set of dictionary matrices? This parameter sharing could provide even more interpretability, since the characters are then represented by the low-dimensional weights used to combine the dictionary elements. This could also provide more scalability to word-level problems. 

review_title: null

final_score: rating: null
review_content: You may as well say linear rather than affine.  The b term in (1) doesn't seem to add any modeling power, since we could equally well write the system in linear form as h'_t = W'_{x_t} h'_{t-1} where h'_0 = (h_0; 1) and W'_x = (W_x,b_x; 0,1).  Then h'_t = (h_t; 1) for all t.  

With this change, the model looks a great deal like the observable operator models of Jaeger (1999).  The main difference seems to be the emission distribution.  You take a linear transform of the hidden state, exponentiate, and renormalize to get a probability distribution over the alphabet.  He skips the exponentiation step (instead restricting the W' matrices to ensure positivity).  

As it happens, another paper this year has proposed skipping the exponentiation step for language modeling, though within a rather different architecture: https://transacl.org/ojs/index.php/tacl/article/view/561 .  So maybe skipping it would work for you too.  

What do you think about these connections?  Can you please compare experimentally to an observable operator model, which seems like a natural baseline since you are apparently proposing a variant on that, which adds an exponentiation step (softmax)?

-----------

A further question is whether this connection allows you to get a consistent estimator of the parameters of this model (given data generated from the model), as in the breakthrough work of Hsu et al. (2009) for consistent estimation of HMMs under mild conditions.  That paper treated HMMs as a special case of OOMs.  I haven't thought carefully about whether it would generalize to estimating OOMs, let alone your model.

-----------

Separately, I wonder whether we should think of your model as a product of experts of n-gram models of different orders.  To say that the effect of x_s on x_t is "purely linear" (in log space) is misleading, since the linear function depends on the intervening characters.  A more natural way of interpreting (5)-(6) that the logit vector l_t is a sum of vectors computed from the unigram, bigram, trigram, etc. preceding time t.  Each of those n-gram vectors (for n=t-s) is given by a product of a bias vector for the first character, n-1 transition matrices for the remaining characters, and the emission matrix W_{ro}.  But in that product, the bias vector for the first character doesn't play any special role.  It's just a compositional matrix for the n-gram. 

Thus, your approach may be rather related to convnet-style language models, with the difference that instead of adding up embedding vectors for several preceding context characters, one adds up compositional embedding vectors for several preceding context n-grams.  Is that fair?

------------

What happens to your performance if you limit the n-gram order, so that the sum in (4) ranges only over s=(t-n+1) to t, rather than s=0 to t?  I'm not suggesting that this is a better or simpler model.  In fact, it's probably slower.  But it would be interesting to see where performance degrades (i.e., on which characters it degrades) if you only consider a finite history.  You could do this both with and without retraining the model.  

An advantage to limiting the n-gram order is that then you would have an actual n-gram model, with a particular kind of smoothing that might work well compared to other n-gram models such as Kneser-Ney smoothing, convnets, or other previous neural net language models that retain a Markov property.  n-gram models are amenable to dynamic programming (e.g., for exact computations over lattices of possible strings in speech recognition or machine translation).





review_title: relation to observable operator models?

final_score: rating: null
review_content: (Last-minute assigned, so quick scan of this paper while nips is going on).
+ Did you consider linear recurrence but nonlinear readout? See some work in
  echo state networks / reservoir networks, where linear ESN with nonlinear readout
  achieves very good performance close to nonlinear ESN with simple readout.
+ Is it possible to provide some intuition for the expressive power of these networks?
  Linear dynamics of hidden states ok, but the switching is a nonlinear dependence on input?
  Examples of dynamical systems they express exactly? Simple LDS are exact models for certain physical phenomena, something similar exists here?
+ 4.3 Figure 5: is the y-axis word index, increasing downwards?
  (i.e. y-axis from top to bottom "the", "annual", ... ?)
  Influence of "the" on revenue is claimed but I can't see in plot.
+ 4.4. Fig6: correlated sure, but what is the interpretation here?
  Also "unnormalized log probability" -> which quantitiy is this? positive log
  so this must be log (values > 1)? log(l_t)? 
+ Fig 8: correlation between consonents and vowels - wouldn't you also expect common diphthongs
   like "ou", "ea"? But I don't see them stand out.
+ The premise of most analysis is that this k_s^t quantity really is essentially the
  influence of character at s.
  But doesn't that sweep under the rug the fact that one of the main
  influences of time s comes of W_{x_s} on previous timesteps, s'<s.
  i.e. imagine W_{x_s} being near-zero, this will be a huge influence
  on characters s'<s to cut off all previous characters. This is in current analysis
  not attributed to character s.

review_title: pre-review questions

final_score: rating: null
review_content: What kind of language modeling performance does this model get on larger data, e.g. the One Billion Words language modeling benchmark? At that scale LSTMs have been shown to significantly outperform count-based and other simpler language models. 

While input-switching makes this more suited for character-level modeling due to the number of parameters, have the authors experimented with byte-pair encoding to use larger atoms?

How does this model compare to a simple linear dynamical system (no input-switching) with a softmax link function to the observations?
review_title: Questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a character language model that gains some interpretability without large losses in predictivity. 

CONTRIBUTION:

I'd characterize the paper as some experimental investigation of a cute insight.  Recall that multi-class logistic regression allows you to apportion credit for a prediction to the input features: some features raised the probability of the correct class, while others lowered it.  This paper points out that a sufficiently simple RNN model architecture is log-linear in the same way, so you can apportion credit for a prediction among elements of the past history.  

PROS:

The paper is quite well-written and was fun to read.  It's nice to see that a simple architecture still does respectably.
It's easy to imagine using this model for a classroom assignment.  
It should be easy to implement, and the students could replicate the authors' investigation of what influences the network's predictions.
The authors present some nice visualizations.

Section 5.2 also describes some computational benefits.

CAVEATS ON PREDICTIVE ACCURACY:

* Figure 1 says that the ISAN has "near identical performance to other architectures."  But this appears true only when comparing the largest models.  

Explanation: It appears that for smaller parameter sizes, a GRU still beats the authors' model by 22% to 39% in the usual metric of perplexity per word (ppw).  (That's how LM people usually report performance, with a 10% reduction in ppw traditionally being considered a good Ph.D. dissertation.  I assumed an average of 7 chars/word when converting cross-entropy/char to perplexity/word.)  

* In addition, it's not known whether this model family will remain competitive beyond the toy situations tested here.

Explanation: The authors tried it only on character-based language modeling, and only on a 10M-char dataset, so their ppw is extremely high: 2135 for the best models in this paper.  By contrast, a word-based RNN LM trained on 44M words gets ppw of 133, and trained on 800M words gets ppw of 51.  [Numbers copied from the paper I cited before: https://transacl.org/ojs/index.php/tacl/article/view/561 .]  Those are language models that are good enough to use for something; maybe the authors' model would continue to fare well in this regime, but we just don't know.  (Has the Text8 benchmark in this paper been seriously used for language modeling before?  It was designed for text compression, a rather different setting where smaller datasets are meaningful because compression is done online, without a training/test split as done for language modeling.  The baseline results in this paper are drawn from a contemporaneous submission with many of the same authors.)

CAVEATS ON INTERPRETABILITY:

I liked the visualizations as an educational tool.  Maybe they'll inspire other visualization ideas for other models.

On the other hand, I'm not sure whether one gets much actionable information from these visualizations:

* Sometimes, visualization is used as a way to understand what a model is doing wrong so that you can fix the model.  But that might not work here: this model doesn't seem to have a lot of room for adjustment before it would stop being interpretable.  (Although you could leave the model alone and preprocess the input data, I guess ...)

* Sometimes, visualization is used to explain a single machine prediction to a human who will make the final decision about whether to trust that prediction (e.g., Singh et al.'s LIME paper).  It's hard to imagine how that would work in this kind of SEQUENTIAL prediction setting, though.

OTHER COMMENTS:

Most of my technical reactions are already given in my pre-review questions.  Thanks to the authors for their answers, and I appreciate that they are running followup experiments for the next version of the paper!

review_title: A character language model that gains some interpretability without large losses in predictivity

Meta-review: All reviewers have carefully looked at the paper and weakly support acceptance of the paper. Program Chairs also looked at this paper and believe that its contribution is too marginal and incremental in its current form. We encourage the authors to resubmit.
Decision: Reject