Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In summary, this paper provides a modification to the Gated Recurrent Unit (GRU) that *sparsely* updates the recurrent state by samplying from a Bernoulli distribution for the *update* term. Thus, rather than being a continuous value in GRU, this is now from $\{0, 1\}$. This is the Variational Sparse Gate (VSG). 

The Simple Variational Sparse Gate (SVSG) further modifies VSG to be made purely stochastic (supposedly - see Questions), and used to evaluate claims from a previous paper that a deterministic recurrent path is required fro competitive results.

Experiments were carried out on a novel environment, and evironments from the DeepMind Control Suite. The novel environment specifically emphasizes the key aspects of this work, namely partial observability, and remembering past states. 
strengths_and_weaknesses: *Strengths*
* Well written for the most part. Explanations and arguments are generally clear. 
* Contributions are clearly stated, and the paper is structured will to argue these.
* A satisfying range of experiments was carried out, at least in the simpler case. The ablation studies were nice to see. 
* Experiments were also carried out on more complex environments too. 


*Weaknesses*
* Interpretations of the notation (Equation 1, 2, 4) could have been written more clearly. 
* Lacks a baseline description of how GRU works (at least intuitively). This could help readers understand why the proposed changes are meaningful, and significant. 
questions: * Figure 2: in the caption, the description of $M$ does not match how it is used in Figure 2(c). 
* line 54 and line162-164: mentions difference between a *purely stochastic* component, and one that contains some deterministic parts. I'm confused by what this means. From Figure 2, it looks like all units contain both deterministic and stochastic calculations. I'm probably missing something, could you say what that could be?
* line 263: Can you really say that, generally speaking, SVSG is comparable to DreamerV2? SVSG seems to perform worse in the Cartpole and Cheetah environments. 


*General*
* What exactly is the role of the transition predictor?
* What is th meaning of *variational* in the name? 
* How exactly do the components in Equations 1, 2, 4 fit together (in their separate cases)? 
* How many seeds were used in Figure 3? Was it 3 (inferred from the checklist). If so, why is it different to the 5 seeds used in other figures?

limitations: I think limitations and future work are discussed well.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies modelling the latent dynamics model by exploiting the idea that different elements of the latent space might exhibit varying degree of changes. The authors propose variational sparse gating (VSG), which is a modification of update gate of the GRU network (Cho, et al. 2014) in the recurrent state space model (RSSM; Hafner, et al. 2019) with a stochastic gating mechanism. The authors also propose Simple VSG (SVSG), which follows the similar stochastic gating updates as VSG, but contains only a purely stochastic latent state. VSG and SVSG are empirically evaluated on DeepMind Control Suite and a novel environment (BringBackShapes) to test their ability of solving stochastic tasks with partial-observability. 
strengths_and_weaknesses: Pros: 
- The paper is well-written;
- The proposed new environment, BringBackShapes, is designed thoughtfully and sufficiently complicated to assess the ability of agents for solving tasks with stochastic transition and reward structures;

Cons:
- The motivation for SVSG is not clear, having a purely stochastic latent state does not show performance improvement in general, and no evidence/intuition of why this could be preferred over VSG (e.g., model complexity, stronger generalisability, etc.);
- There is inconsistency in the performance of VSG/SVSG with respect to varying level of stochasticity and partial-observability, e.g., one would expect monotonic increasing size of performance gap between VSG/SVSG and Dreamer V2 in the BBS environment, but none is observed (at least from the reported results in Figure 3c). Moreover, in the Atari evaluations (Figure 10), tasks such as Ms Pacman and Seaquest clearly exhibits strong stochasticity, but VSG fail to outperform Dreamer V2. 
questions: The major concerns are listed in the previous section, some minor questions are listed below.

- What is the relative computational complexity between VSG and SVSG?
- Is the sparse reward setting necessary in BBS? A complicated (stochastic, non-Markovian) reward structure might be less trivial for standard agents without sparse gating (e.g., Dreamer v2) to acquire, according to the statements in the main text;
- The Private Eye evaluation (Figure 10) and the accompanying discussion is not so convincing given the large variance in the evaluation curves;
limitations: The authors have provided thoughtful discussion on the limitation and potential future areas for investigation of the VSG model, such as combination with other recurrent architectures, increasing the complexity of BBS environment, VSG dealing with categorical latents, etc.

I see no potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes two recurrent models for model-based RL: Variational Sparse Gating (VSG), improved from DreamerV2 with binary gate, and Simple Variational Sparse Gating (SVSG) which has purely stochastic states. It also introduces a new partially-observable environment, BringBackShapes(BBS), to evaluate agents under POMDP setting. The experimental results show that the proposed VSG outperforms DreamerV2 in continuous control tasks including BBS and DeepMind Control Suite.
strengths_and_weaknesses: ## Strengths
1. VSG achieves impressive experimental results comparing to DreamerV2. 
1. A new environment with various difficulty levels is designed for POMDP. 

## Weakness
1. The significance of the proposed sparse gating needs to be clarified. In VSG, the sparse gating is the only improvement from DreamerV2, while the rest part including architecture and losses are the same as DreamerV2. The authors should discuss more about why sparse gating can improve performance.
For example, list some hypotheses to the reasons and verify them in experiments.
Besides, additional ablation studies about sparse gating may also help to demonstrate its effect. 
Otherwise, without such discussion and verification, the technical contribution of VSG is considered as minor.
2. The motivation of SVSG is unclear. Previously Stochastic State-Space Model (SSM) that has stochastic state has been empirically proved to have worse results than RSSM that has deterministic state. In the experiments of this paper, SVSG also has worse performance than VSG. The authors should clarify the motivation of proposing a new model with stochastic state. 

questions: 1. Setting $u_t$ as binary values seems make the model not recurrent anymore. The model parameters $W_v, b_c, W_c$ and $b_c$ only get update when $u_t = 1$. 
2. I am afraid that $u_t=0$ would be a big issue when planning in world model (as Figure 1(b)). If $ u_t=0 $, then $ h_t = h_{t-1} $ due to Equation (2). 
The transition predictor $ p_{\phi}(z_t|h_t) $ that predicts prior state $ \hat{z}_t $ 
depends only on $ h_t $. 

  In the case of $h_t = h_{t-1} $, the predicted state $ \hat{ z }_t $ 

  would be similar to $ \hat{z}_{t-1} $ at time step $t-1$. 
Consequently, the actor conditioned on $ \hat{z}_t $ output the similar policy as time step $t-1$. 

  If the distributions of $ p_{\phi}(z_t|h_{t-1}) $ and the policy have low variance, making $u_{k \geq t}=0$ almost surely, then the whole planning in world model takes input of the same hidden state $ h_{k \geq t} = h_{t-1} $ .

3. Does SVSG have the same losses as VSG ?
4. Why VSG and SVSG are called "variational"? It seems that there is nothing related to variational inference.
5. What setting of BBS is used in Figure 3b), i.e., the number of distractors and the size of arena?
6. In Table 1, what does first-visit time mean? Is it the time when visiting the first object, or the first time when visiting the last object (all objects)? 
7. The experiments and related work lack of an important baseline method, SLAC[1], for continuous control tasks. SLAC solves POMDP by stochastic sequential models and also defines stochastic latent states.
8. DrQV2 is a strong baseline algorithm in continuous pixel control. Showing training curves of DrQV2 in figures would be good comparisons. 
9. DeepMind Control Suite is POMDP but many algorithms assuming fully-observability solve DMC by stacking frames. It would be nice if you evaluate the methods on other public environments that are "truly" POMDPs, e.g., first-person view 3D games.

Other comments:
1. In line 85, $o_t$ is defined as state but in line 115 $s_t=[h_t,z_t]$ is also defined as state. I suppose the latter one should be correct for the definition of POMDP, since $o_t$ is the image encoding as in line 120. 
1. SLAC is listed in the legend of Figure 5, but not shown in any curves or main text. 

Reference

[1] Lee et al. Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model. NeurIPS 2020

limitations: The effect of sparse gating needs to be further discussed and verified.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes two new latent dynamics models (VSG and SVSG), which aims to incorporate a certain inductive bias: in some real world problems, the changes are very sparse, so the latent can be updated sparsely to help long term memory. They test the proposed algorithm on a new environment BBS, which shows that VSG and SVSG outperforms existing methods, and on DMC benchmark, which shows that VSG and SVSG ad comparable with existing methods.
strengths_and_weaknesses: Strengths:
- This is a very interesting paper to incorporate the inductive bias to existing latent dynamics model RSSM and SSM, and the results show that the proposed methods seem to be effective on specific problems.
- It is also every interesting to see that after the modification, the SVSG can outperform SSM and comparable with RSSM/VSG.

Weakness:

I agree that this is a very interesting work, but I think it will be better if the analysis can be more sufficient. For instance, by *randomly* selecting some latents $u_t$ to be zero in GRU, the VSG can achieve better performance on long-memory cases, which is good. However, this may mismatch the motivation “the real world changes are sparse” since the selection is purely stochastic. And this is even more significant for SVSG. I think it is important to understand why the proposed methods may work.
Can I think of this as a regularization trick, which ask for the encoder to encode the information duplicately for each dimension, since each of them might be zeroed? Is it possible for you to show an experiment that shows the effects of the randomness? For instance, given a sequence, and several different samples from Bernoulli distributions, to see if the next state can decode the same image?
questions: See above
limitations: N/A
review_title: null

Meta-review: metareview: The reviewers agreed this work is well written and the set of experiments are good. However, a general concern was that the interpretation / explanation of this method should be improved. The rebuttal seems to have addressed these points to a good degree and we urge the authors to revise the work to include the further explanations / experiments and analysis in the final version.
confidence: Less certain
Decision: Accept