EINS: Long Short-Term Memory with Extrapolated Input Network Simplification

Abstract: This paper contrasts the two canonical recurrent neural networks (RNNs) of long short-term memory (LSTM) and gated recurrent unit (GRU) to propose our novel light-weight RNN of Extrapolated Input for Network Simplification (EINS). We treat LSTMs and GRUs as differential equations, and our analysis highlights several auxiliary components in the standard LSTM design that are secondary in importance. Guided by these insights, we present a design that abandons the LSTM redundancies, thereby introducing EINS. We test EINS against the LSTM over a carefully chosen range of tasks from language modelling and medical data imputation-prediction through a sentence-level variational autoencoder and image generation to learning to learn to optimise another neural network. Despite having both a simpler design and fewer parameters, this simplification either performs comparably, or better, than the LSTM in each task.

Reviews and author responses:

The paper tries to analyze LSTM and GRU through differential equations and proposes an alternative formulation.

I vote to reject this paper. I think the analysis has many flaws and only minor connection to performance of RNNs, and empirically the success is marginal at best.

The paper main point is based on re-writing the discrete dynamical system as a discrete integration of a ODE.  The way it is formulated in the paper has several flaws.
 First, the mixed timestamp is unnatural for ODEs. if s'=a then the Euler integrator would be s_t=s_{t-1}+h*a_{t-1} not s_t=s_{t-1}+h*a_t (eq.2). 
Second, the timescales isn't part of the continuous ODE but part of the discrete integration. Integrators with dynamic step size defiantly do exist, but the step size does not depend on the data like in GRU/LSTM but based on approximation accuracy. In any case even if there was a real strong connection, I didn't see "an illuminating perspective" from the ODE connection.

Detailed remarks:
- I didn't get the connection of the Newton-Rapson method and ODEs, as the Newtow-Rapson is an iterative algorithm that has nothing to do with ODEs (or no connection was made in the paper).
- In 3.4 you claim to do theoretical work and make strong remarks such as "In other words, LSTMs have the potential to have more equilibrium states" but I didn't find any support in the work itself. There was not even a clear well defined theorem/claim let alone a proof.
- Comparing the EINS to LSTM regarding the number of parameters is a bit misleading, as the right comparison is with GRU which is the common "slim" alternative.
- Experimentally the work only shows (table 1+3) very little improvement over LSTM and no comparison to GRUs.


Minor remark:
- In the first sentence, "difficult to interpret" is not really related to the paper.



General Comments
The authors proposed a new type of gated recurrent unit for RNNs, and show comparable performance on a number of RNN applications, some achieved with less number of parameters.  

Detailed Comments

Section 3 (Interpretation)
This section basically draws similarity between the RNN update equations with discrete temporal difference equations.  It’s unclear why showing that the equations look similar is motivating.  It would be better if the authors can borrow analyses done on differential equations and apply them here.

Section 5 (Experiments) 
To me, the main contribution of this work lies (potentially) in the newly proposed EINS unit.  
To demonstrate that it’s practically appealing, below are my suggestions.  
# of parameters is only practically meaningful for saving storage. More interesting analyses should include memory cost during training/test (i.e., the total number of activations, and the required gradients for learning), and computational time.   
Ease of tuning (i.e., is this method easier to tune than LSTM/GRU).  One benefit of a new method can be that it’s more robust to tuning-/hyper-parameters.  If the authors can show that EINS is easier to tune, this can be a good result.
Experiments are diverse, not I rather see a large scale experiment because the strength of the proposed method was motivated as a “simpler” method.  The provided experiments do not strengthen the claim that EINS is “simpler” for the practitioners. Understandably, this will require a lot of effort, but many of the interesting applications of RNNs as of today are large-scale speech recognition, language modelling, and video recognition.  It’s unclear whether EINS has any benefits in that regime.

Overall, the authors demonstrated a variant of RNN updates, but neither the theoretical nor  empirical evidence was strong enough to 1. Convince practitioners to switch from LSTM/GRU to EINS, 2. Provide insight/understanding to the workings of RNNs.  
It’s interesting to look for new building blocks for neural nets.  However, perhaps more interesting than manually designing it, maybe you can use the reasonings in this paper to design constraints for neural architecture search model.  

Best,


This work proposes a simplified variant of LSTM, motivated by the connection of gated variants of RNN, LSTM and GRU, and Hopfield network. Technically, the author shows that the GRU is a special case of the dynamic Hopfield model. And then the author proposes three simplifications on LSTM, which are suggested by the formulation of the dynamic Hopfield model. A wide range of tasks are included in the experiment and the results prove the effectiveness of the simplification. 

However, I found it really hard to align this work with other existing works with similar or the same efforts. There has been already many works on variants of LSTM, such as SRU, fastGRNN, Clockwork RNN and etc. But none of them are employed as the baseline model, or even mentioned in the related work. Furthermore, there is not even a related work section in the paper. Therefore, this work should not be accepted.



Meta-review: The paper proposed new version of LSTM which is claimed to abandon the redundancies in LSTM.  It is weak both in theory and experiments.  All reviewers gave clear rejects and the AC agree.

Decision: reject