Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The manuscript presents a novel recurrent neural network architecture with an inductive bias for learning sparsely changing hidden state representations. This seems to be achieved by the proposed gating scheme using the novel rectified tanh activation function for the update gate values. This activation function yields true zero activations producing an exact copy of the corresponding value from the previous hidden state. By applying L0 regularization to the state-updates, the network is incentivized to minimize the number of modified dimensions in the state.
main_review: The presentation is very clear and detailed. The experiments highlight several contributions: 1) GateL0rd develops stable, sparsely updated latent state representations. 2) making the output also depend directly on the input, frees the hidden state to focus on information inferred from past time steps. To summarize the points made in the sections below, I think the work proposes an intriguing alternative to commonly used RNN architectures, but the experimental evaluation has some weaknesses unless I’m misinterpreting some observations.

Strengths
1. The proposed regularized ReTanh gating and direct dependence of the network output on the input seem like very appropriate solutions to weaknesses of other (gated) RNN architectures. Especially the potential disentanglement of observable and unobservable information might help in reinforcement learning with an unknown reward function.
1. The sparse state changes should be very useful for event-based sequence modeling.
1. GateL0rd seems to outperform other commonly used RNN architectures.
1. The manuscript is very well written. A lot of questions that came up while reading were answered either a few paragraphs later or in the supplementary material. For instance, I was a bit sceptical at first about the choice of the ReLU gradient estimator, but Section C.2 cleared that up for me.

Limitations/Questions
1. How do you think GateL0rd would perform if applied to environments with very high-dimensional visual observations, such as MuJoCo, Atari or ProcGen (of course using an appropriate encoder network for the frames)? I think that GateL0rd certainly has a better prior for disentangling factors of variation in the environment than GRUs or LSTMs. However, I also think that it may be difficult to learn this disentanglement, if at the same time a compatible input representation has to be discovered given high-dimensional observations.
1. Footnote 3: I think that tanh(s/2) = 2 * sigmoid(s) - 1. 
1. Can you quantify the impact of $f_{init}(x)$ compared to setting $h_0=0$?
1. The testing error curves in Figures 4 and 5 look like training did not  yet converge. If that is the case, the networks should be trained longer since the focus of the manuscript does not seem to be on comparing the sample efficiency of different architectures.
If training did converge, I’d suggest presenting the loss curves as well.
1. Can the test prediction error going up for GRU and LSTM during training really be attributed to the networks’ overfitting to observable information and spurious correlations? The variance of these curves seems to be very large which is often a symptom of optimization issues (e.g. exploding gradients for some runs)! 
1. In the shepherd task, suppose the sheep was allowed to move horizontally while occluded with horizontal motions sampled from some symmetric zero-mean distribution. If the motions were part of the observation, do you think GateL0rd would be able to learn to modify the memorized initial position accordingly in its hidden state? If the motion information were not observed, how would GateL0rd learn to account for the uncertainty?
1. Time indices are not consistent. In line 106 the mapping is (y_t, h_{t+1}) = f(x_t, h_t), while in line 71 it is (y_t, h_t) = f(x_t, h_{t-1}). Also in Equation 2, I believe the regularization should be applied to delta h_t.

limitations_and_societal_impact: See main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new recurrent neural network architecture (RNN) called GateL0RD, designed to learn sparsely changing latent variables of sequential data. The authors hypothesize that latent variables should be updated sparingly only if significant changes happen in the considered environment. To enforce this hypothesis in RNN, GateL0RD uses a new gating function of Rectified Tanh to output exactly zero values. For actual implementation, the considered prediction task is regularized to minimize the number of positive elements in the sampled gate inputs. Ablation study shows that GateL0RD improved prediction accuracy and planning performance compared with LSTM or GRU in several POMDP environments.
main_review: (1) Originality \
This work builds a new activation function (ReTanh) on a gating function in usual LSTM/GRU architectures where sigmoid and $\tanh$ are widely used. 

(2) Quality 
1) Simple but practical implementation of $L_0$ regularization makes this work scalable.
2) Various experimental results and the corresponding ablation study are provided, including the interpretability.
3) Main concern of this work is the support of the hypothesis that latent variables should change sparsely. It looks like there is no theoretical support or guarantee on the optimality of this approach, though several examples in the real-world and human recognition area are mentioned.

Q1. Is there any more substantial explanation or related work supporting the effectiveness of the hypothesis? \
Q2. Does the proposed method perform better than $L_1$, $L_2$ regularization methods?

(3) Clarity \
This paper is well written and easy to read. The strong points of this work are well organized regarding items: accuracy, generalization, long-term memorization, and interpretability.

(4) Significance \
As mention in the Discussion section, the proposed regularization may not be effective when the considered partial domain has severely changing characteristics. 

Q3. Is there any backup experiment (any specific environment) to check this aspect? Can this regularization be used in general in partially observable domains? Setting $\lambda=0$ and asserting that "the unregularized network performs well." is not persuasive. The proposed architecture without the regularization is not fundamentally different from LSTM / GRU. Also, note that we usually do not know a priori how much the considered environment changes.

limitations_and_societal_impact: Yes, the authors have addressed the limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes an RNN with a gating mechanism that encourages its latent state to have sparse updates by incorporating stochastic gating units trained using an approximation of L0 regularization of the state updates. The authors propose to use this RNN in partially observable problems in which the latent dynamics are known to change slowly and long-term memory is important. The authors show in a number of partially observed tasks that their proposed RNN and training objective outpeforms well known RNN architectures when predicting the environment dynamics.
main_review: * Originality and significance

Slow changing dynamics is a pervasive property of many time series and (PO)MDPs.  While designing architectures and training algorithms that reflects this property is not novel, it still remains an open research problem. The method proposed, an RNN that closely resembles LSTMs, is very closely related to some existing techniques as stated in the related work, with the main difference being in how the new gating units are modelled and trained. The authors note that these prior works have not been applied to POMDP problems but instead to conventional classification and language-based tasks. However, because the authors do not compare with these works, it's unclear how those techniques (see references [27, 28, 29]) would fair in POMDPs problems. The authors claim that a GRU does not include an output that is not its state (making it difficult to predict changing observations when the state remains constant), however that can be simply remedied by using the observation in addition to the state in the f_post component.

Evaluating these ideas in POMDPs is original, and the experiment section show a clear advantage of the RNNs sparsely changing bias over the baselines which is definitely interesting. Since the main novelty is the application to POMDP problems (and not as much the method itself), the selection of environments should be more extensive. The environments' partial observability and dynamics are fairly simple (e.g. they do not incorporate stochastic transitions). This paper would thus be strengthed if the authors compare relevant work in modelling the dynamics in challenging POMDPs and discussed the method's advantages and limitations (see Igl et al. 2018, Hafner et al. 2019 for variational methods for example).  If the main contribution of this work is to use the proposed RNN for long-term memory of unchanged environment states, then perhaps the paper would benefit from presenting it as a general module that can replace the core RNN in more complex architectures. For instance, many variational methods use an LSTM as the memory back-bone. Would their performance improve with this?

* Quality

The proposed RNN is tested in terms of next step prediction as well as its usefulness for planning. The second aspect is important to assess the quality of the predictions when unrolled over time. 
The experiments succesfully show the benefits of sparse updates in different environments and settings (e.g. with generalization to unseen trajectories), as well as show the explainability of the learned states. There's also a good discussion on the trade-offs of predictive performance and explainability (sparseness) which is important for the reader to know. Results also seem robust as the error bars are relatively small.

The L0 regularization method is well derived and closely related to [16]. However this work, contrary to the relaxation of the intractable L0 regularization, does not minimize the probability of non-zero activations as in [16] (which is differentiable) as a proxy of L0, but instead directly incorporates a non-differentiable term based on the sum of gate values, which is approximated with the straight-through estimator. It's unclear why the authors choose this particular solution which is more biased (due to using the straight-through estimators in *both* terms of the loss), and also what practical consequences this has. 
I would also like to see an ablation study where a simpler version of this idea is tried: a sigmoid gate is used with L1 regularization. In other words, to what extent is it necessary to use stochastic gates?

* Clarity

The paper is well-written and the methods are generally clear, well explained and motivated.  There are some parts where certain details are explained in the appendix but not referenced from the paper so it seemed at first a bit cryptic. For instance, in line 93 they refer to a "variation of the straight-through estimator" without specifying which variation. Similarly, many interesting ablation studies are in the appendix without any reference to them.

Some other minor issues:
- typo in line 75 (combine > combined).
- line 248, I believe the authors mean that a higher value results in *more* latent state changes?

References:

Maximilian Igl, et al., Deep Variational Reinforcement Learning for POMDPs. ICML 2018

Danijar Hafner et al., Learning Latent Dynamics for Planning from Pixels. ICML 2019
limitations_and_societal_impact: The authors have been fairly clear about the limitations and trade-offs of their method; and as a paper focused on methodology, it does not seem to have any direct social and ethical implication.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a sequence prediction method that incorporates a particular inductive bias -- infrequency of latent state updates -- as measured by an L_0 term on latent state differences. This term is approximated and made more easily optimizable using a method similar to [16], which examined L_0 regularization in non-sequential domains. The paper conducts experiments on several low-dimensional POMDP tasks and then draws the following conclusions: the method generalizes better to out-of-distribution inputs in several partially observable domains than other commonly used RNNs, exhibits long term memory in a domain, and exhibits explainability (in its latent states), sometimes at the cost of reduced prediction performance.
main_review: **Opinion summary**: The paper offers a well-motivated method for sequence prediction in partially-observed domains that contain dynamics with significant changes. The proposed method is straightforward and clear. However, the method is of questionable originality (see below). The experiments are conducted relative to basic common NN-based sequence-prediction methods, which do show significant improvements, but lack comparison to previously-proposed methods for incorporating the same inductive bias (infrequency in latent state updating). Thus, we cannot conclude that this method is better than a relevant set of previous methods for accomplishing the same goals. There are a few other issues with the experiments that could be improved.

**The paper has questionable originality, as prior work accomplishes a similar goal in a different way**: there are missing comparisons to existing methods -- [25,26] use predefined time scales for gate opening, which may be sufficient for high performance on the considered tasks, and [27,28,29] all use binarized update gates, and are thus quite related to the proposed method. The proposed method is not clearly different enough from these methods, and is therefore of questionable novelty. I think empirical results that compare the methods are needed -- without them, we cannot understand situations in which the proposed method is better or worse than prior work that attempts to accomplish the same goal.

**The interpretability is not quantitatively supported:** The method pitches sparsity in gate opening as both a way to learn a more performant system, as well as a way to understand when the dynamics of the environment are significantly changing. On the latter point -- Figs 4d and 6b illustrate examples where the gate opening makes qualitative sense (the paper describes these as "interpretable"). These results provide intuition, but are not evidence of how reliable the method is at representing "significant and systematic" changes with gate openings. It would be interesting to have a better understanding of how reliable it is, for instance by (automatically?) labeling each point in a sequence as either a "significant and systematic" change or not (like the black-outlined positions in Fig 6a and the moving-robot positions in Fig 4d), and then reporting binary classification results (e.g. precision, recall, accuracy) using the classifier p(systematic change=1 | any gate open=1)=1; p(systematic change=0 | any gate open=0)=1.

**Other issues with experiments (ordered by decreasing priority):**
- L270 "only encodes unobservable information within its latent state" this claim has, at best, anecdotal evidence (Fig 4d). It is a significant claim to state generally, so either temper it or provide sufficient evidence to support it.
- The $\lambda=0$ results in Fig 3d requires further explanation. Why does the version with no regularization exhibit the fewest latent state changes (L248 says "a higher value results in fewer latent state changes.")?
- Fig 3c is too zoomed-out. The differences between the nonzero lambda experiments are difficult to see.
- Fig 15 is referenced in the text but appears in the supplement.

---
Post-response update: My above concerns have been addressed.

limitations_and_societal_impact: The limitations were appropriately described; the societal impacts were described inside the checklist (which are essentially a statement that there are no ethical concerns specific to the work). I agree with that sentiment, but it should be incorporated either into the main text or into the appendix, rather than the checklist, which will not be present in a potential final version(?)
review_title: null

Meta-review: The paper presents a recurrent architecture with an inductive bias (via gating and L0 regularization) that encourages sparse updates to its state. This is hypothesized to lead to improved generalization, and is empirically evaluated in a number of partially observable environments.

The reviewers were initially sceptical, but the discussion with the authors has allayed most concerns, so now the consensus is towards acceptance. In general, the reviewers agree that the paper is well written and the contribution is interesting (albeit somewhat incremental). Overall, I'm happy to recommend acceptance.
Decision: Accept (Poster)