Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of MaxEnt reinforcement learning, with the goal of increasing the expressiveness of the policy while still allowing for practical entropy maximization. The proposed method relies on latent variables to avoid complexities with using EBMs. 

The challenge with directly using latent variables is that the entropy is difficult to estimate, therefore entropy maximization is difficult. Typically, we'd want to optimize latent variable models via variational approaches with the ELBO, but this gives a lower bound of likelihood, hence an upper bound of the entropy term and the maxent rl objective. Maximizing the upper bound of course leads to practical issues.

Instead the paper proposes to use a technique in hierarchical inference (Sobolev & Vetrov 2019), which gives a lower bound estimate to the entropy term. The rough intuition is use multiple samples, but unlike IWAE, do so in a way that reuses samples, and this can be shown to flip the direction of the bound.

With this new estimator, the paper optimizes the maxent rl objective with variational methods on a latent variable model and shows improvements over SAC.

strength_and_weaknesses: Strengths
The general idea makes sense, and the methodology is well explained. The technical contribution is not novel, but applying it to the RL setting appears to be novel.

Weaknesses
The experiments don't seem to compare against energy based model and autoregressive model variants. The introduction says these "complicate training procedures and are inefficient in practice", but it seems important to provide experimental evidence.

clarity,_quality,_novelty_and_reproducibility: The writing is clear, and the contribution is novel to the best of my knowledge. I would believe that the results are reproducible.

summary_of_the_review: In summary the paper proposes a method for optimizing latent variable policies with the maxent RL objective, which is difficult to do with naive techniques.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use the latent variable policy for MaxEnt framework, which is testified to improve the sample efficiency and stability of RL. It uses some techniques to estimate the marginal entropy, as well as reducing the variances in the gradient. Experiments on DM control suite show the performance improvement of the proposed SMAC algorithm.

strength_and_weaknesses: Strength: 

The paper has good writing and clear delivery of the idea. Two techniques for solving the entropy estimation and variance reduction are effective. Experiments show performance improvement over previous algorithms almost consistently. Necessary ablation studies are conducted.

Weakness:

Unclear about Eq. (11). In Eq. (11), only the first sampling $s^{(0)}$ is used for sampling the action, while the other K $s$ are used for estimating the marginal entropy. Why is it? Why not use all $s$ for sampling action as well as estimating the entropy? Please explain in the paragraph.

Explanations about notations. I’m not clear about the upper scripts $(a), (b)$ in Eq.(12). Please explain this in the paragraph.

Missing references and baselines. There is a related paper (Probabilistic mixture-of-experts for efficient deep reinforcement learning. Jie et al. 2021) which also adopts the Gaussian mixture model for policy function approximation. Also I think Dreamer-v2 should also serve as an important baseline method. The paper should compare the proposed method with these.

The variance reduction in Sec. 3.2.2 seems to be a computationally heavy one. Please compare the computational efficiency in the alation study about whether this variance reduction is applied or not.

How about the on-policy algorithms? The proposed SMAC algorithm seems to only be compared with off-policy algorithms like SAC and TD3. How about on-policy algorithms like PPO? Is SMAC possible to be applied on PPO with an on-policy update as well?

In sec 5.1, is this model-free setting? If so, please write it explicitly in the paragraph.

“Reasonable number of particles”. In Sec.5.1, the paper suggests to use a reasonable number of particles. What specific number is it, 8? What’s the number of the particles in each experiment?

Performance improvement in most of the experiments over existing algorithms seems to be marginal. 

clarity,_quality,_novelty_and_reproducibility: The paper has good clarity, but some details about the algorithms need to be explained more. 

The novelty of the proposed algorithm is not enough. The two main techniques in SMAC seem to be borrowed from previous literature, though may not in the RL papers. The factorized Gaussian distribution has been investigated a lot in previous research.

summary_of_the_review: Overall, the paper is written well. But the novelty of the proposed algorithm is not high enough, and the experiments show marginally improved performance in most environments. Some important baselines are missed, and more experiments should be added to clarify the problems I asked in the Weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to marginalize (or rather, draw multiple samples from) latent variable policies as a means of improving estimates of policy entropy and Q-value estimates. The authors point out that a naïve ELBO estimate can result in a loose bound (i.e., a biased estimate), so they suggest using a nested estimator and multi-level Monte Carlo. They perform experiments using continuous control environments from DeepMind control suite, both in the setting of joint-based and pixel-based observations. They find that their method outperforms SAC and TD3 in the joint-based setting and outperforms various world-model-based setups in the pixel-based setting.
strength_and_weaknesses: **Strengths**

- **Proposes a principled approach to improving entropy and Q-value estimates.** To the best of my understanding, the method proposed here appears principled, building off of previous methods for entropy estimation. While many previous works have generally ignored the fact that world model latent variables induce hierarchical policies, few previous works have taken advantage of this fact to improve entropy and value estimators.

- **Builds off of popular existing methods.** The paper builds off of several established algorithms in the literature, namely SAC and world models. These algorithms have fairly widespread adoption in the academic literature, so there’s a higher likelihood that methods built upon these algorithms will be more widely adopted, particularly if they are not difficult to implement.

- **Uses standard benchmark setups/environments.** The authors evaluate SMAC using the DeepMind control suite, a standard set of continuous control environments built on MuJoCo. These environments are fairly commonly used in the literature, so demonstrating results here will likely be compelling to the deep RL sub-field.

**Weaknesses**

- **Fairly complex method for a fairly minimal performance boost.** Early in the paper, the authors lament that “the use of more expressive models have not gained nearly as much traction in the community…these constructions often result in complicate[d] training procedures and are inefficient in practice.” Two thoughts here: First, I would disagree with this characterization and analysis. Normalizing flow-based policies do not yield a significantly more complicated or inefficient procedure. Indeed, many modern neural network libraries (PyTorch, Tensorflow, and JAX/Distrax) make it trivial to implement these policies and sample / evaluate log-densities. It’s not that these more expressive policies aren’t useful. Rather, the limited boost in performance in our fairly simple benchmark environments hasn’t sufficiently justified their widespread adoption. Second, the method presented here strikes me as more complicated than, for example, normalizing flow-based policies. I find it difficult to believe that SMAC would receive widespread adoption when normalizing flow-based policies have not.

- **Unclear if the baselines are really appropriate/fair comparisons.** The authors, at times, appear to confuse the distinction between several proposals: 1) using hierarchical policies, 2) using more sophisticated/multi-sample estimators, and 3) their particular approach for estimation, which itself consists of multiple components (entropy and Q-value estimators). For instance, in some of their experiments, the authors only compare with SAC and TD3, claiming to show the benefits of hierarchical policies. This is hardly a novel finding, as it’s well-known that more expressive policy distributions generally outperform less expressive ones. At the very least, I would have expected comparisons with normalizing-flow-based policies, which, having personally implemented these in one afternoon, are not difficult to use (and these do, indeed, outperform Gaussian policies in SAC). Likewise, in the only other set of experiments, the authors compare with “Latent-SAC”, SLAC, and Dreamer, but it’s not at all clear what these baseline comparisons are meant to accomplish. Is the boost in performance coming from using multiple samples, or is it the estimator itself? Is it from having better value estimates, entropy estimates, or do they both contribute? Likewise, as far as I understand, this estimator could also be used with a model-based value estimator, so it’s unclear whether the comparison with Dreamer tells us much about the setup proposed here.

- **Somewhat minimal contribution.** While I’m not discounting the insight and effort required to develop the main techniques (nested estimator with MLMC) in the paper, the main contribution of the paper reduces to just drawing more samples for estimation. Assuming we have a reasonable estimator, it’s hardly surprising that drawing additional samples should yield improved entropy and value estimates, and likely, improved performance. The proposal of the paper is effectively to use a well-known knob within the existing RL framework to improve performance, which I view as a somewhat minimal contribution.
clarity,_quality,_novelty_and_reproducibility: **Clarity** — The clarity of the paper, while generally good, could be somewhat improved. One unclear aspect was the notation. For instance, the authors conflate the latent state of the environment with the latent state of the model, when these two variables need not be defined in the same manner. In Eq. 9, the authors seeming write the entropy as being over the history (rather than the latent state and action), while in Eq. 10, they write the entropy as being conditional on the history. (Note: in Eq. 9, there’s also a typo (?) in which r is denoted as a distribution over the latent state). And in Eq. 11, the authors refer to $\pi$ as both the action policy as well as the posterior over latent states.

I found Section 3.2.1. somewhat difficult to follow, as it’s unclear exactly where the authors are employing amortized variational inference vs. exact probabilistic inference, etc. That is, it’s not clear how many additional models are required to implement the authors’ approach. In a similar vein, the authors claim to use “standard variational inference,” when, as far as I can tell, they are making an assumption of amortization, i.e., I doubt that they are optimizing $q$ or $\pi$ on a per-state basis.

In some cases, the underlying model assumptions go unstated, making it difficult to understand where the authors are making deliberate choices vs. following convention. To give some examples, the authors employ a filtering variational posterior over the world model latent state, yielding a looser bound than a smoothing variational posterior, without justifying this choice. They also insist on conditioning the policy purely on the current latent state, rather than (correctly) allowing the policy to depend on the entire history. 

**Quality** — Overall, the quality of the paper is reasonable, but there are several aspects that could be improved. In some places, the authors use sweeping generalizations or mischaracterizations.

- For instance, on the first page, the authors claim that they want to learn complex, multi-modal behaviors, whereas many existing algorithms rely on local perturbations around a singe action. First, even with a uni-modal policy mapping, compounding local perturbations across time will still yield multi-modal trajectory densities. Second, this restriction to local perturbations is, in fact, a result of using parametric policies (i.e., neural networks), whereas an iteratively-optimized Gaussian policy (e.g., optimized through planning) can recover multiple modes. Finally, the authors do not definitively demonstrate that their approach results in complex multi-modal behavior.

- The authors state that “a majority of approaches for handling partial observability make use of world models,” which is not well-supported by the literature. A variety of existing works do not use latent variable models to handle partial observability, instead relying on recurrent networks (see, e.g., MuZero/Muesli) to integrate information.

- Elsewhere in the paper, the authors claim that “SAC is often restricted to the use of policies where the entropy can be computed efficiently, e.g. a factorized Gaussian policy…” However, in the case of SAC, the authors employ a tanh on the policy output, using only a sample-based estimate of the entropy. Likewise, in many KL-regularized RL methods, e.g., MPO (Abdolmaleki, et al., 2018), the authors resort to sample-based estimates.

- The authors claim, without any rigorous justification, that “with standard neural network architectures, a latent variable policy can universally approximate any distribution if given sufficient capacity.” This is merely presented as a “proposition,” without any follow-up analysis to determine the degree to which this holds in practice, e.g., does the KL between the policy and Boltzmann optimal policy not decrease with added latent variables? If so, this would seem to contradict the finding from the hierarchical VAE and diffusion community, which is that hierarchy depth improves distribution flexibility. Is this purely a function of the “capacity” of the policy networks, or are there other considerations?

- The authors claim that “latent variables can be used within the $Q$-function to better aggregate uncertainty” but they do not verify this empirically.

As mentioned previously, the baselines for empirical comparison do not isolate the factors of the authors’ proposal (i.e., hierarchy, multiple samples, variance reduction, etc.). This makes it difficult to assess the origin of the performance benefits, diminishing the quality of the empirical analysis.

Minor point: Figure 5 is missing axis numbers.

**Novelty** — The novelty of the proposed approach is somewhat limited. The main contribution of the paper is in combining several previously proposed ideas to improve multi-sample entropy and value estimates in the case of latent variable policies. This is entirely within existing framings, i.e., SAC and world models. And the specific combination and application of these techniques is novel to RL, it’s fairly intuitive that using additional samples to improve entropy and values estimates should yield improved performance. In this sense, I found the proposed approach to be only marginally novel.

**Reproducibility** — While it’s difficult to assess, my guess is that the results are reproducible. The performance of the baselines looks to be in-line with previous works, suggesting that the authors’ implementation is reasonable. And there is a clear trend of SMAC being marginally better than previous methods across environments (with multiple seeds), demonstrating that the improved estimator likely does improve performance. I appreciate that the authors included some ablations to assess the effect of different hyperparameters.
summary_of_the_review: The paper provides a technique for improving entropy and values estimates in latent variable policies using multiple samples. While I believe that this is a useful idea, the somewhat limited novelty, combined with the marginal performance boost (over inappropriate baselines) leads me to feel that this paper is somewhat below the acceptance threshold. The paper could be improved by improving the baselines to illuminate the various aspects (i.e., hierarchy vs. multi-sample vs. variance) of the proposed method. The authors may also consider whether this method allows one to scale up to larger hierarchical policies to tackle more challenging tasks where performance gains may be more significant.


**UPDATE:**
I appreciate that the authors have corrected various issues in the text that I raised. The normalizing flows experiments also add a useful multi-modal baseline to the results. I have increased my score from 5 --> 6 to acknowledge the effort on the part of the authors. However, I still feel that the complexity of the method and its limited novelty do not warrant the somewhat minimal boost in performance as currently demonstrated in the paper. For instance, in light of the normalizing flows experiments (Figure 11), it appears SMAC has statistically significant performance benefits on only 1 (or maybe 2) of the 24 DM control suite environments. I do not find this to be a compelling argument in favor of using SMAC.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes a new max entropy RL technique that enables multimodal stochastic policies via a latent variable model.

Strengths:
* The problem of modeling multimodality in max entropy RL is an important one
* The use of a latent variable model to achieve multimodality in stochastic policies is novel
* The empirical results demonstrate the effectiveness of the approach

Weaknesses:
* The idea is not transformative.  There are alternative techniques based on mixture models and normalizing flows, but it is not clear whether the latent variable model has an inherent theoretical advantage.
* The proposed method requires the estimation of an upper bound on entropy that is conceptually complex and requires multi-level samping.
* The empirical results show a slight edge for the proposed technique, but it is not clear that the improvement is significant

Since the weaknesses are not significant and the proposed latent variable policies will be of interest to the RL community, this work deserves to be published.
justification_for_why_not_higher_score: * The idea is not transformative.  There are alternative techniques based on mixture models and normalizing flows, but it is not clear whether the latent variable model has an inherent theoretical advantage.
* The proposed method requires the estimation of an upper bound on entropy that is conceptually complex and requires multi-level samping.
* The empirical results show a slight edge for the proposed technique, but it is not clear that the improvement is significant

justification_for_why_not_lower_score: * The problem of modeling multimodality in max entropy RL is an important one
* The use of a latent variable model to achieve multimodality in stochastic policies is novel
* The empirical results demonstrate the effectiveness of the approach

The reviewers all agreed that this paper should be published and will be disappointed if it is not published.
Decision: Accept: poster