Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper fits in the category of survey/review paper. While this kind of paper is not common at Neurips, it doesn't mean that it should be disregarded. It does a good job at trying to unify a popular body of work on policy gradient methods under the umbrella of "control as inference". The paper is well-structured: it defines a taxonomy which differentiates between "EM-based" methods and those based on "direct KL minimization". The paper focuses on four algorithms: Maximum a posteriori Policy Optimisation (Abdolmaleki et al. 2018), Advantage-Weighted Regression (Peng et al. 2019), Soft Actor-Critic (Haarnoja et al. 2017) and Relative Entropy Policy Search (Peters et al. 2010). The  paper starts by explaining how the RL problem can be cast as a probabilistic inference problem and tries to explain how an evidence lower bound can obtained. It then goes into the specifics of each algorithm and how each either parametrizes the prior or the posterior and whether  it uses hard constraints (with closed-form expressions) or soft/regularized re-formulations. 

The empirical section then compares the raw performance three of those algorithms (SAC, MPO and AWR) on six Mujoco environments. 

Regarding the "co-adaptation" expression in the title, I wouldn't say it defines the paper. The "co-adaptation" findings really pertain to the impact of specific implementation details on the final performance in those six Mujoco environments. More specifically, the authors study the impact of double-q learning + clipping, ELU activations vs ReLu, network size and layer normalization. The empirical findings are nicely summarized through tables. 
main_review: Overall, the structure is great and I like the ablatation study. Furthermore, I have learned new things via the EM vs direct minimization taxonomy. That being said, I think that the writing could be improved. I find the technical presentation is not solid enough. Not that the authors should go with a measure-theoretic presentation, but the paper would gain clarity if the notation is improved. Don't get me wrong, I'm not complaining about the choice of symbol, but rather the symbol-to-meaning mapping which I find deficient. For the kind of paper that the authors are aiming for, the background section needs to be near-perfect and crystal clear. 

Regarding the main contribution, the co-adaptation claim, I think that it should be de-emphasized. Just in terms of space in the text, pages 1-6 are dedicated to the presentation of SAC, MPO and AWR under a unified narrative. The co-adapation findings are only developped in pages 7-9 and the overall takeaway seems to be that pretty much everything should stay the same for MPO and AWR but SAC is a bit more tolerant and can do better with a few tweaks taken from the other methods. 

# Specific Comments

The expression "co-adaptation" is interesting, and I can see how it expresses the idea that you are trying to convey. That being said, I wonder if there would be another way to describe the issue without introducing new terminology (in this case borrowing from biology). For example, from the very begining in the abstract, the sentence:

"tanh Gaussian policy and network sizes are highly adapted to algorithmic types " (line 17) is a bit obscure to the non-initated 

On line 53, "are highly co-adapted to KL-minimization-based", you could perhaps say "are highly specific" instead.

If you insists on using "co-adaptation", you should perhaps state explicitely what it means in the first sentence or so. Something like "we define co-adaptation as the implementation decisions that do not stem directly from the conceptual algorithmic development, but from empirical considerations."

On line 27, "there have been a series" I think that you should write "has" 
On line 29, "on maximum entropy", "on a"

On line 88, "Our experiments prove", I suggest to keep "prove" on the math side. Instead: "demonstrate"/"highlight" 

Caption of figure 1, "when the state-value function is parameterized instead of Q-function" ("instead of the"). This sentence fragment is unclear.

On line 93, "transition probability distribution" => "transition probability function", also "initial state distribution" not formally (Puterman) part of the MDP definition

Line 95: The use of "u" as a time index is unusual. How about "k" instead? 

Line 95: "standard RL setting", I wouldn't say that this is necessary "standard". How about "we consider a policy gradient setting"?

Line 95: " for its parameter that maximizes" => "for parameters that maximize"

Line 99, " advantage function" cite Baird 1993.

Line 100, "is known to be a" => "is the"

Line 105, "For simplicity, we consider a finite-horizon MDP" I know that it is common in RL to make this assumption, but it's very important to keep in mind that the properties of finite-horizon MDPs and infinite-horizon discounted ones are not the same. The beauty of infinite horizon discounted is that it allows us to restrict our attention to stationary policies when searching for optimal policies. But it doesn't have to be the case in the finite horizon setting! In fact, the contraction/fixed-point based perspective is no longer appropriate. Instead, you would want to talk about backward induction and so on... 

In the ELBO derivation around line 112, you have $\tau$ on the rhs but not the lhs. This is because $\mathcal{O}$ is function of $\tau$. Perhaps writing $\mathcal{O}(\tau)$ would make things clearer. Also, writing $\text{Pr}(\mathcal{O}|\pi_p)$ suggests that $\pi_p$ is a random variable, and that there is conditional distribution defined over those quantities. If so, I suggest being a bit clearer about that.

Line 116, "a popular choice is the one that factorizes", making that assumption of an MDP earlier, I don't think that you have much of a choice here in your factorization.

Line 123-124: are these other choices valid? As in: integrating/summing up to 1?

Line 125: You jump from an expectation over trajectories, to one where there exists a joint over state-action pairs. This is not a trivial step.

Line 132: "unnormalized state distribution", if it's unnormalized, it can't be a distribution and writing $\mathbb{E}_{d_\pi}$ doesn't quite make sense. think what you mean is something like $d_\pi = p_1^\top(I - \gamma P_\pi)^{-1}$. You then write an equality-constrained program where your equality constraints is the actual normalization. This is confusing.

Line 152: "converts the hard-constraint optimization". Just to be clear, the Lagrangian step is not about "converting" an constrained optimization problem into an unconstrained one (the constrained solution need not coincide with an unconstrained min or maximum of the Lagrangian). 

Line 170: "an exact case". You should be clearer about what you mean by "exact" (lack of function approximation). 

Line 173: "control problem, except the policy parameterization." The sentence sounds incomplete. Please expand

Line 182: "reverse KL penalty (second term) in the Lagrangian (Eq. 2) with a reverse KL constraint". This sentence is confusing, but I think what you mean is that it's actually solving a different constraint optimization problem. One where you not only require to have distributions (integrates to 1) but also that the KL is equal (or perhaps) less than some designated value. You should then write the equivalent constrained program as you did in equation 1. It would be clearer than talking about changes to the Lagrangian. 

Line 194: "For a paramterized policy" => "parameterized" (you should use Grammarly to catch those prior to submitting) 

Line 204: "replaced by samples of actions from a replay buffer" I don't quite understand how this works out to be similar.

Line 219: "the dual function of entropy constrained Lagrangian " I don't think that this sentence makes sense. I think all you want to say is that they treat $\eta$ as a Lagrange multiplier and they they then do gradient-ascent descent on the Lagrangian. 

Section 5.1: "Distribution of the Policy". This is about the policy parametrization. Not about some "distribution over policies". I think that you should avoid this expression
limitations_and_societal_impact: I don't want to be that reviewer who says: "more experiments". Of course we all want more experiments... But I need to be

I need to say that I'm becoming increasingly concerned about the over-reliance on the Mujoco domains for benchmarking policy gradient (Berkeley-style) methods in RL. As a community, we're using these environments so often that it's becoming increasingly difficult to assess where we stand in terms of "real-world" impact.  Are we overfitting? 

Because this is essentially a survey paper, I would have like to see a more extensive evaluation. 

How about taking more time and writing a much longer a journal paper? Like those great survey papers from the people at Darmstadt. How about the benchmark track at Neurips?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Provides a unifying mathematical framework for off-policy inference based algorithms such as MPO, AWR (that use pseudo-likelihood objective) and SAC, classifying them into EM based or KL based algorithms (uses the control as inference framework to do so). This allows decoupling the effect of high-level algorithmic choices from low-level code optimizations, to identify where the benefit from these algorithms really comes from, and further analyzing which design choices are transferrable across different algorithm families. 
main_review: **Originality**: The control as inference framework has been used by the authors of MPO and SAC in their respective papers, and isn't a new way of looking at these algorithms. More specifically, the EM class formulation closely follows a similar characterization in the MPO paper. Nevertheless presenting these together as part of the same framework seems new (where KL control is shown to be almost a sub-problem for EM control). The paper cites a lot of the previous work dealing with meta-analysis of RL algorithms which point out that performance is often dependent on the specific code implementation used. 

**Quality**: The mathematical formulation seems technically sound and well supported (and are based on prior work in the field). The experimental evaluation includes 5 openAI gym mujoco environments and is quite exhaustive as it includes ablations over a number of different design choices, such as network size, activations and normalization etc, in order to study which low-level code optimizations are important.

**Clarity**:  The paper is well written and easy to follow and understand. The authors first present their unified control as inference framework, then clearly show how MPO, AWR, SAC fall into it. The experimental evaluation is well organized for analysis into each separate ablation axis. 

**Significance**: Irreproducibility is a well known issue for RL, especially the dependence of algorithm performance on the specific implementation used. This work takes a step towards addressing this issue. The exhaustive experimental evaluation might be useful to other researchers, where the authors analyze what aspects of the different algorithms are critical. They find that certain choices that are important for SAC (clipped double Q and squashed tanh), do not improve performance of other algorithms. Further they also find useful design choices in MPO (ELU and LayerNorm) which improve performance of SAC. That said, some of these observations might be specific to the gym environments tested, and may not transfer.
limitations_and_societal_impact: The main limitation are novelty of the formulation (which has been used in prior work, albeit separately for the different algorithms), and the uncertainty over whether the low level design choice recommendations made here (with regard to network size, normalization etc) will apply to other agents/robots.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Comparing the performance of different RL algorithms can sometimes be difficult because algorithmic contributions can be confounded with differing implementational choices between the agents. This paper attempts to examine the relative contributions of algorithmic vs implementational choices in a series of inference-based, off-policy actor-critic algorithms -- MPO, AWR, and SAC. The authors' starting point is a unified view of these algorithms from the perspective of control-as-inference. In this framework, policies as cast as posterior probabilities that a given action in a given state will be optimal. The authors derive an ELBO for this posterior that, when optimised, yields an approximate optimal policy. The authors then describe how optimization can be approached in two ways: an EM approach (where both a prior and posterior policy are optimized) and a KL approach (where only the posterior policy is optimized). MPO and AWR belong to the EM family of algorithms, while SAC belongs to the KL family. The authors further detail how these algorithms fit into their unified framework.

In the second half of the paper, the authors focus on the implementational differences between these algorithms. They focus specifically on four choices: clipped double q-learning, tanh-squashing in the policies, the choice of activation function and layer norm, and the network size. Specifically, through a series of ablation studies, the authors examine how critical some of these choices are to specific algorithms and which of these can be transferred to other algorithms.
main_review: Overall, this was an interesting paper. For me, one of the paper's strengths was its theoretical framework casting MPO, AWR, SAC and other algorithms as special cases of a more unified control-as-inference objective, and the discussion of EM control vs KL control was rather enlightening. It was unclear to me, however, whether the authors were regarding the "Implementational Details" of Section 4.2 as possible transferable implementational choices, on the same level as activation functions and network sizes, or whether these were to be interpreted as different algorithms.

My main hesitations, however, lie with the empirical side of this paper, and for the reasons below, I do not believe the paper yet ready for acceptance:
* the authors ablated the different implementational choices individual, but my concern is that there may be co-dependencies between different implementational choices, and individual ablations may fail to capture that. For instance, whether the tanh-squashing helps or hurts the policy network depends on whether and how the policies are regularised, something which may be more pertinent to KL control rather than EM.
* there also seems to be a thematic disconnect between the four implementational choices the authors have focused on and the preceding control-as-inference discussion. The four implementational choices do not seem to depend in any way on the fact that MPO, AWR, and SAC can be treated in a unified, inference-based manner. Whether to use layer norm, ELU or RELU, large or small networks, tanh squashing, or double Q-learning is a question that applies equally well to any deep RL algorithm, not just the ones discussed here. Hence it is puzzling why the authors chose these four differences to focus on given the topic of their paper. Furthermore, if the differences in Section 4.2 can be regarded as implementational differences, it would have been more interesting and thematically-consistent with the first half of this paper to examine the transferability of those choices instead.
* for algorithms that originally included a certain implementational choice, it is clear that ablation leads to a decline in performance. However, I find in several cases that the authors' claim of improved performance less convincing. For instance, the authors claim that using ELU and layer norm across all algorithms will improve performance. While doing this certainly didn't hurt performance and even led to a higher mean score in some cases, the score increases do not seem to reach the level of statistical significance given the size of the standard deviations.
* finally, the scope of the empirical studies seemed a bit limited to me. The authors only looked at ablating four implementational choices, and it would have strengthened the paper significantly had the authors examined a wider selection (see e.g. Hessel et al, 2017 for an example of a more extensive ablation study in RL).

Minor errors:
* the authors reference equation 3 several times (e.g. lines 187 and 203), but this equation is never numbered in the main text. Presumably, it is the equation that appears in the M-step, after line 157.
* typo in line 268:  “achieve” rather than “archive”.
* typo in 294: “agreed upon default choice” rather than “agreeable default choice”.





--- Review update ---

After further consideration and based on the clarification and additional results the authors have provided, I have decided to upgrade my score to 6. To me, this work offers two very valuable contributions to the community. The first is the theoretical demonstration that many disparate policy gradient algorithms can actually be seen within a unified control-as-inference framework, a framing that I find novel and exciting. This unified framework not only provides a deeper understanding of pre-existing algorithm, but it also offers a new means to study and inform the design of future algorithms. The second is, as both reviewer CYmr and the AC have pointed out, the paper's attempt to survey the various algorithms already in the literature using this framework, examining how transferable certain implementational or code components are within the KL or EM families. What the authors are attempting with this work is definitely of great interest to the community.

Given the huge scope, I appreciate that exhaustive exploration on the empirical side is not possible. The new results were rather interesting, but for me, certain questions remain. For instance, the latest results have not entirely addressed my question about co-dependencies. From their limited transfer studies, the authors have tried to draw general conclusions about transferability between the KL or EM families; yet, I still wonder whether there are synergies between components that the authors break with their ablations and whether some of the performance drop has more to do with breaking these synergies than whether the algorithm is KL or EM. Again, as an example, I mention the possibility that soft-Q functions may synergise more with tanh-gaussian policies and clipped double q-learning. In the ablation studies, the authors found that using tanh-gaussianity and/or clipped double q-learning didn't transfer well to EM algorithms. But this could be because tanh-gaussianity and double clipped q-learning work best with a soft-Q function, and these components performed worse because soft-Q  was absent. This is just one example of how possible co-dependencies may confound the conclusions that the authors sought to make in their ablation studies. Regarding the new results, the failure case of alternative $\pi_p$ was also rather intriguing. In short, there is a lot in the space of algorithms that remains to be explored, but doing an exhaustive exploration would be an enormous undertaking.

In the end, I am inclined to support accepting support of the paper not only for the reasons I mentioned at the beginning, but also because I believe doing so will provoke much important discussion and inspire new research to address the questions that may come out of it.
limitations_and_societal_impact: Given that the foundational nature of this work, there would not be any malignant implications beyond that which is already inherent in basic RL, so a discussion of the potential societal impact is not relevant for this work.
review_title: null

Meta-review: While the reviewers thought there were various ways that this paper could be improved, there was also general consensus that this framework which unites multiple existing algorithms was interesting and useful, prompting one reviewer to comment on the new insights it brought them. The authors are encouraged to address the feedback from the reviewers in their camera ready. 


Decision: Accept (Poster)