Title: Entropic Desired Dynamics for Intrinsic Control
Authors: Steven Stenberg Hansen
Guillaume Desjardins
Kate Baumli
David Warde-Farley
Nicolas Heess
Simon Osindero
Volodymyr Mnih
Venue: NeurIPS 2021
Keywords: intrinsic control
skill discovery
exploration
unsupervised reinforcement learning

Abstract: An agent might be said, informally, to have mastery of its environment when it has maximised the effective number of states it can reliably reach. In practice, this often means maximizing the number of latent codes that can be discriminated from future states under some short time horizon (e.g. \cite{eysenbach2018diversity}). By situating these latent codes in a globally consistent coordinate system, we show that agents can reliably reach more states in the long term while still optimizing a local objective. A simple instantiation of this idea, \textbf{E}ntropic \textbf{D}esired \textbf{D}ynamics for \textbf{I}ntrinsic \textbf{C}on\textbf{T}rol (EDDICT), assumes fixed additive latent dynamics, which results in tractable learning and an interpretable latent space. 
Compared to prior methods, EDDICT's globally consistent codes allow it to be far more exploratory, as demonstrated by improved state coverage and increased unsupervised performance on hard exploration games such as Montezuma's Revenge.

TL;DR: Discriminating between globally consistent codes yields exploratory behavior and a map of the controllable environment without the requirement of supervisory signals or rewards.

Review discussion:
The paper proposes an interesting and promising method, and appears to fix an important issue with VIC.
Further, the method is adequately evaluated and achieves good performance, outperforming some prior work.
However, the method appears to have a hidden assumption that the initial state distribution is narrow, as I detail below. This is an important issue since in many practical environments, or in those considered by Eysenbach’18, the initial state distribution is wide, and the method might need to be modified to account for that. Because this issue can be addressed in the rebuttal, I am not providing my final decision at this time.
The major issue with the proposed approach is the treatment of the initial skill $z_0$. It appears from the text below l114 that its prior distribution is unconditional p(z_0). This distribution does not seem to be specified in the paper.
However, it seems to me that any unconditional distribution will be inadequate. Consider an initial state distribution $p(s_0)$ wide enough so that navigating between two states sampled from it is hard. In the extreme case, the initial state distribution can be the entire state space (as commonly done e.g. for some navigation tasks). Since the skills have global meaning and correspond to states, the agent will often observe an initial skill that is too hard to execute, which is exactly the issue that the current submission attempted to address! This is because the prior $p(z_0)$ distribution is too wide, and the reverse predictor distribution will also learn to match it. Perhaps the agent should be evaluated on environments with more diverse initial state distributions to confirm the agent is robust to this.
There are also a number of more minor issues with clarity
- What is O in Eq 2? On l133 Eq 2 is stated to be the agent’s objective.
However, Eq 2 drops action entropy terms. Why are they dropped? l90 states “In practice, we optimize the above objective using a value-based reinforcement learning algorithm and -greedy policies (in lieu of a Boltzmann policy), and thus drop these terms altogether.” It is rather unclear to me what this means. Eq 1 is the objective for the policy, it does not change with the chosen exploration strategy. If the proposed method is not derived from control as inference, why is the background section in 3.1 necessary?
- There is no difference between Eq 2 and Eq 3. The $O^{(i)}$ term is expanded in one and compressed in the other. The text implies that Eq 3 does not consider long-term consequences, but it does, up to M-1.
- l142, “we thus condition on the code most likely to have yielded state si, under the reverse predictor.” This is not what Eq 4 says. In Eq 4, the code is sampled from the reverse predictor instead of taking the mode. Which one is actually used?
- l174 is confusing. Is the function $f$ trained through f(s_i+1), or both f(s_i+1) and f(s_i)? This sentence implies that both, but algorithm 1 says that only f(s_i+1).
- The choice of notation is extremely confusing. Why is prior over actions $\pi^0$ but prior over latents $q$? Why is the forward distribution denoted as $p^z(\tau)$ instead of the conditional distributions $p(\tau|z)$ as used elsewhere in the paper? All 4 distributions in Eq (1) are denoted differently.
Finally, the experimental evaluation omits some important baselines such as DISCERN, which was also evaluated on Atari. Since the proposed method is motivated by learning hierarchical skills, it would also be instructive to provide a comparison on downstream hierarchical RL, as done in DIAYN and DADS.
No
We thank the reviewer for their detailed comments and constructive feedback.
## Initial state distribution and z_0
There is no assumption that the initial state distribution is narrow, since there is no assumption that z_0 is unconditional.
While we primarily treat hindsight substitution as a ‘trick’ to avoid open loop updates, it is also how z_0 is computed i.e. it is the most likely code under our current reverse predictor q(z_0 |s_0). This should be visible in the algorithm block, but we acknowledge that the derivation perhaps obscures this fact. We will rework the text to make this more apparent. As an aside, to the best of our knowledge, DIAYN doesn’t use environments with wide initial state distributions. The OpenAI gym control tasks have near deterministic initial conditions, not dissimilar to that of the Atari games we run on.
Regardless, since we don’t reset the environment between skills, the initial state distribution is not that significant to our results (episodes are typically in the 10000s of steps).
## Baselines
We agree that more baselines would always be better, but with limited time and resources we can’t hope to compare against everything. DISCERN is quite related to our included baselines, and so hopefully those serve as a rough surrogate for the time being. We note that the number and variety of baselines included here is at least comparable to that of related works.
That said, we are adding a DIAYN baseline and 2 more ablations; we hope this addresses this concern.
## [minor]
* The distinction between Eq. 2 and 3 is indeed subtle and could be clearer. On line 134, we specify that the expectation in Eq. 3 is wrt. a *fixed* distribution $p^\pi(\Lambda_{<i})$, from which $z_{i-1}$ and $\tau_{i-1}$ are drawn. This was meant to convey that rewards from option periods >= i had no bearing on stochastic decisions from option periods < i. In retrospect, this was a suboptimal choice of notation. We will overhaul the notation in favor of a variant of Eq. 2 featuring discount factors, which are 1 during option execution, and 0 at option boundaries. This should make clear that no bootstrapping is performed across options.
* l142.
When using the hindsight trick, we replace $z_{i-1}$ with the mean of the reverse predictor’s output distribution. This corresponds to a lower variance estimator of Eq 4. We will add a note to this effect.
* L174 Upon consideration, we realize our notation for the reverse predictor is a source of confusion. $q_w(z \mid \tau)$ should be interpreted as the general case, with $q_w(z \mid s_0, s_T)$ being the specific form employed by VIC, and $q_w(z_{i-1} | si)$ that employed by EDDICT. We propose using $q_w(z \mid \tau)$ (no resampling) or $q_w(z_i \mid \tau_i)$ (with resampling) for generality, and $q_w^{VIC}$ and $q_w^{EDDICT}$ when discussing the specific reverse predictor used by each algorithm.
* [choice of notation] The notation $\pi_0$ for the prior is borrowed from DISTRAL [43]. We use $q$ for the reverse predictor (even if it belongs to the prior), to make explicit the connection to variational inference [33, 18]. We will unify our notation for conditional distributions throughout the paper.
The rebuttal addresses some of my concerns but not others.
> While we primarily treat hindsight substitution as a ‘trick’ to avoid open loop updates, it is also how z_0 is computed
This is fine, but the inference network has to be trained to match the unconditional prior (the same way it is trained to match the prior for further timesteps). Are you saying that p(z_0) is not actually used in the algorithm? If the loss term for this prior is omitted, this has to be explicitly mentioned in the paper. Ideally, there would also be a justification for why this is done.
After rereading the algorithm, it is also unclear to me why the prior term is omitted from reward computation (perhaps this is explained somewhere in the paper?).
> L174 Upon consideration, we realize our notation for the reverse predictor is a source of confusion
This response does not answer my question.
In fact, I am not quite sure what question it answers. My question though was whether the function f is trained with gradients from f(s_t+1) or gradients from both f(s_t), or f(s_t+1).
## $p(z_0)$
With Markov chains in general, the initial state becomes less important than the transition dynamics as the number of transitions increase (under a few benign assumptions such as ergodicity). And as previously mentioned, EDDICT works fine using the hindsight substitution for z_0, which avoids the issues you initially raised (i.e. wide initial state distributions) as this allows the latent state to correspond to whichever environmental state happened to be sampled.
In case there is any confusion, we want to stress the p(z_0) is the probability of the latent state at time 0, not an unconditional prior as in DIAYN.
That said, you are correct that there is a conditional prior that should contribute to the rewards: $p(z | z_{prev})$ (expressed as the final term in equation 2, indexing notation changed here for clarity). It was excluded from the algorithm block for simplicity, as for all but one experiment this prior was uniform, and thus only contributed a constant to the reward.
But for the experimental results in Figure 4b, we had a Gaussian distributed conditional prior, and so included it in the rewards as it was no longer a constant. All of this is to say that we agree with the reviewer; in general the prior should show up in the rewards and its optional exclusion for the special case of a uniform prior explicitly noted. We will make these changes to the paper. For reference, the reward in algorithm 1 thus becomes: $r \leftarrow \log q_w(z \mid s_K) - \log p(z \mid z_{prev})$
## Line 174
Apologies for the confusion, we mistook your question for something else entirely. Gradients flow through both f(s_t) and f(s_t+1).
Regarding the algorithm differing from the text, this is not the case, but we can see how it might appear that way. In the algorithm, note that the parameters $w$ contribute to $\log q_w(z \mid s_K)$ from both time steps -- the former is less obvious, but occurs in the algorithm line where z is updated via $\mu_w$ (hindsight substitution).
That said, several reviewers the found the $mu_w$ terminology confusing, and it will be replaced in the algorithm to better reflect the fact that $q_w$ is being used in both cases -- just as described in the line 174.
Post-rebuttal
Having read other reviews and after further exchanges with the authors, I am now getting a clearer perspective on the paper and its motivations. The approach might be interesting, and the changes proposed by the authors might strengthen the submission.
However, I think that a complete assessment of its value would require a re-evaluation of the updated manuscript. My evaluation on the work as it is remains borderline.
To my understanding, the crucial idea of this paper is to consider a temporal correlation between the latent codes that are sampled along a trajectory, so that to induce a deeper exploration of the environment w.r.t. previous methods that just sample the latent code randomly.
However, the general goal of those prior works seem slightly different from the one pursued here: They seek to learn a set of diverse (i.e., discriminable) skills instead of temporally-correlated skills that are combined in a sequence.
Beside this point, two other factors are making me skeptical on the actual value of the paper:
1) I think the paper fails to convey strong motivations or intuitions behind the algorithmic choices, which are mostly presented as empirical tricks. Of the three ingredients incorporated in the VIC methodology, i.e., greedy optimization, hindsight correction, and latent dynamics, only the latter is clearly motivated in the text;
2) Even if some results are promising (such as the average return in Montezuma), I found the experimental analysis not sufficiently convincing for an essentially empirical work.
On the one hand, the method is only compared against VIC, RVIC and a reduced version of EDDICT without the latent dynamics (which is quite competitive), whereas other methods in the same thread, e.g., DIAYN [14], DISCERN [46], VISR [20], VALOR [1], are not considered.
On the other hand, it is not even clear if those methods are the most relevant comparison, as opposed to deep exploration algorithms, such as [10,11] that are mentioned in the text, or hierarchical reinforcement learning.
For the reasons described above, I am currently providing a negative evaluation to this work. I report below some additional comments.
PRESENTATION
Unfortunately, I found this paper not really easy to read. Several sentences are too convoluted, the terminology and notation are sometimes confusing. The text does not provide much intuition on what is going on with the method.
RELATED WORKS
The paper by (Campos et al., Explore discover learn: Unsupervised discovery of state-covering skills, 2020) also identified state-covering as the main limitation of mutual information methods.
Even if they seem to address the problem from a fairly different perspective, I think it should be mentioned in the paper.
Another concurrent work (Liu & Abbeel, APS: Active pretraining with successor features, 2021) combines a mutual information method (e.g., VISR) with a specific state-covering incentive, which might also be a relevant comparison to EDDICT.
EXPERIMENTS RESULTS
The actual significance of the performance reported in Table 1 is unclear without providing confidence intervals around the average of the three runs.
No
We thank the reviewer for their comments and feedback.
“However, the general goal of those prior works seem slightly different from the one pursued here: They seek to learn a set of diverse (i.e., discriminable) skills instead of temporally-correlated skills that are combined in a sequence.”
We view the goal of our method to be the natural extension of prior work. As you note, prior work aims to learn diverse skills. As seen in the derivation (and is transparent in the reward function), EDDICT also tries to learn diverse skills. Where EDDICT differs from prior work is by highlighting the fact that it is typically unrealistic to expect all skills to be executable from all environmental states, and loosening this tacit assumption is the core contribution.
“I think the paper fails to convey strong motivations or intuitions behind the algorithmic choices, which are mostly presented as empirical tricks. Of the three ingredients incorporated in the VIC methodology, i.e., greedy optimization, hindsight correction, and latent dynamics, only the latter is clearly motivated in the text”
Optimization of the conditional policy pi_z is very much analogous to the optimization of goal-conditional policies elsewhere in the hierarchical RL literature, except that in our case the (subgoal) reward function is given by the reverse predictor and learned.
Greedy optimization here merely means that the policy is only trained with the local goal-attainment reward which indicates how well  the conditional policy achieves the current goal z within K timesteps. The reward for optimizing the policy for the current goal does not incorporate the reward for achieving future goals.
This approach is customary in the HRL literature where each option period is often treated as a separate episode when optimizing the low-level (subgoal) policy.
Thus, achieving each subgoal is treated as a separate problem. This greatly facilitates credit assignment and reduces variance when optimizing the subgoal policy.
All of that being said, we will also include ablations of greedy credit assignment and the hindsight correction, which demonstrate their utility empirically.
“Even if some results are promising (such as the average return in Montezuma), I found the experimental analysis not sufficiently convincing for an essentially empirical work.
On the one hand, the method is only compared against VIC, RVIC and a reduced version of EDDICT without the latent dynamics (which is quite competitive), whereas other methods in the same thread, e.g., DIAYN [14], DISCERN [46], VISR [20], VALOR [1], are not considered.
On the other hand, it is not even clear if those methods are the most relevant comparison, as opposed to deep exploration algorithms, such as [10,11] that are mentioned in the text, or hierarchical reinforcement learning.”
We agree that more baselines would always be better, but with limited time and resources we can’t hope to compare against everything. [14,46,20,1] are all quite related to our included baselines, and so hopefully those serve as rough surrogates for the time being. We note that the variety of baselines included in our initial work is at least comparable to that of the papers referenced here.
That said, as previously mentioned, we are adding a DIAYN baseline and 2 more ablations; we hope this addresses this concern.
We fail to see how the EDDICT ablation is `quite competitive’.
While it does have reasonable scores on some games, it never achieves non-zero reward on Montezuma’s Revenge (Figure 5) nor learns an interpretable latent space (Figure 4a). We noticed that Figure 4a contains a typo -- it states the comparison is against VIC when it is really against the EDDICT ablation. We hope this was the source of confusion on this point.
As mentioned in the paper, comparing to methods that do not yield conditional policies is not a primary concern, but we will be adding a new table showcasing previously published results on Montezuma’s Revenge, including [10,11].
“The actual significance of the performance reported in Table 1 is unclear without providing confidence intervals around the average of the three runs.”
Error bars were only excluded due to typesetting constraints, but a version of this already exists in the appendix. We will add a footnote in the main text referencing this.
I would like to thank the authors for their detailed replies and their remarkable effort in addressing reviewers' comments.
However, I am still doubtful about the motivation of the proposed method. I provide below some follow-up comments.
> We view the goal of our method to be the natural extension of prior work. As you note, prior work aims to learn diverse skills. As seen in the derivation (and is transparent in the reward function), EDDICT also tries to learn diverse skills. Where EDDICT differs from prior work is by highlighting the fact that it is typically unrealistic to expect all skills to be executable from all environmental states, and loosening this tacit assumption is the core contribution.
This is fine!
But the reason why previous works intend all skills to be executable in any state is that they usually assume that a skill starts from the initial state and runs until the episode ends. I think it is actually consistent with their motivation.
Instead, my main problem with EDDICT is that I have not fully understood its intended goal:
- Is it an hard-exploration method? I would rather use RND to this purpose;
- Is it a method to maximize state coverage in unsupervised settings? Why not using an approach that is directly optimizing this objective, such as (Liu and Abbeel, Behavior from the void: Unsupervised active pre-training, 2021) or (Seo et al., State entropy maximization with random encoders for efficient exploration, 2021);
- Is it a method to discover diverse skills? From the reported experiments, I am not sure EDDICT is better than DIAYN or VIC in this regard, as the average return in Montezuma's Revenge does not tell much on the discriminability (or any other measure of diversity) between skills.
My feeling (which might be completely wrong) is that EDDICT is essentially a skills-discovery method that is compared with prior works in hard-exploration tasks.
> As mentioned in the paper, comparing to methods that do not yield conditional policies is not a primary concern, but we will be adding a new table showcasing previously published results on Montezuma’s Revenge, including [10,11].
Can the authors describe what has emerged from this comparison?
> Error bars were only excluded due to typesetting constraints, but a version of this already exists in the appendix. We will add a footnote in the main text referencing this.
I see the table in the Appendix, but I believe that all the results with overlapping confidence intervals should be bolded. From this comparison, I would say that EDDICT is significantly better than the baselines only in Montezuma. Averaging over more seeds will perhaps help in separating the confidence intervals.
We thank the reviewer for their continued engagement.
## EDDICT motivation
>This is fine!
But the reason why previous works intend all skills to be executable in any state is that they usually assume that a skill starts from the initial state and runs until the episode ends. I think it is actually consistent with their motivation.
We agree that this is consistent with the motivation found in the papers first introducing VIC and DIAYN.
But more recent work has begun applying the same methods to domains where episode length are impractically long (e.g. Atari is often >>10,000 steps per episode), and do so by iteratively resampling skills within an episode [7,20]. We correct our paper to not attribute this sampling scheme to the original papers, and will describe their evolution as above.
>Instead, my main problem with EDDICT is that I have not fully understood its intended goal:
The goal for EDDICT (alongside all other intrinsic control methods) is to maximize the mutual information between a latent variable its downstream consequences on the environment. This is effectively the amount of control an agent has over its environment, which is distinct (but related) to exploration and state coverage/entropy. This becomes apparent with the forward decomposition of the mutual information: $I(z;s) = H(s) - H(s|z)$. As you can see, maximizing state entropy is involved, but only to the extent that can be explained by the latent variable z. This is one explanation as to why these methods aren't typically SOTA when evaluated purely on coverage/exploration/state entropy -- their objective contains another term.
>Is it a method to discover diverse skills? From the reported experiments, I am not sure EDDICT is better than DIAYN or VIC in this regard, as the average return in Montezuma's Revenge does not tell much on the discriminability (or any other measure of diversity) between skills.
Figure 4b is a direct comparison between EDDICT a more traditional VIC method (EDDICT-$\delta$ is both an ablation of EDDICT as well as being VIC with Gaussian codes) on an estimate of the mutual information objective they both care about. The dashed lines show that while EDDICT's codes are somewhat less discriminable, this is more than made up for by the increase effective number of codes being utilized. We will clarify this in the final draft.
>My feeling (which might be completely wrong) is that EDDICT is essentially a skills-discovery method that is compared with prior works in hard-exploration tasks.
This feeling is not unfounded. Figure 4b does directly compare intrinsic control/skill discovery methods on their own terms, but the rest of the experiments involve some surrogate for coverage/exploration/state entropy. This is due to the intrinsic control objective being quite abstract and hard to relate to the wider community.
Thus, we felt the need to provide experiments that highlight the respectable performance our method achieves on metrics related (but not identical to) its objective.
>Is it a method to maximize state coverage in unsupervised settings? Why not using an approach that is directly optimizing this objective, such as (Liu and Abbeel, Behavior from the void: Unsupervised active pre-training, 2021) or (Seo et al., State entropy maximization with random encoders for efficient exploration, 2021);
I hope the above comments address the general sentiment, but as an aside I want to point out that both of these particular works are currently under review at this conference, so this isn't a reasonable request.
## Comparisons to pure exploration methods
>Can the authors describe what has emerged from this comparison?
As was already mentioned in the draft, both [10] and [11] do far better in terms of unsupervised score (and likely any other state entropy surrogate) than any intrinsic control method (including EDDICT). We believe this comparison highlights the need for further work on intrinsic control methods, as this gap is representative of the additional difficulty introduced by constrained the maximization of state entropy by the need to be explainable in terms of a latent variable.
## Statistical significance
>I see the table in the Appendix, but I believe that all the results with overlapping confidence intervals should be bolded. From this comparison, I would say that EDDICT is significantly better than the baselines only in Montezuma. Averaging over more seeds will perhaps help in separating the confidence intervals.
While our initial way of treating spread is on par with the rest of the field, we agree with the reviewer that this standard is not ideal, so we have run additional experiments to bring the total seeds per condition to 10. Here is a [link](https://imgur.com/a/DPr5Xrv) to the new results, with only clear winners bolded (i.e. no overlap in CI).
While there are still fewer clear winners than under the more liberal bolding scheme (even with the extra seeds), EDDICT is now the *only* method significantly beats all other baselines on any game by any metric.
Indeed, EDDICT wins by at least one metric on 4 of the 6 games.
We also commit to running 10 seeds for VIC and DIAYN and updating the table for the final draft, but were unable to do so during the response period due to resource limitations. We prioritised RVIC and the EDDICT ablation because our initial results suggested that they were the strongest baselines.
I thank the authors for their additional comments and clarifications. I think that I am now getting a clearer perspective on the paper and better appreciating its value. In my view, the main message of this work is that a simple change (i.e., temporal correlation between skills) in existing mutual information approaches can provide significant benefits to their performance in some scenarios.
However, I am still doubtful that mutual information is actually the best way to tackle those hard-exploration scenarios where the introduced change helps the most.
Anyway, the paper with the additional changes promised by the authors, along with a clearer presentation of the algorithm motivations, might provide sufficient value.
But I believe that a robust assessment would require a re-evaluation of the updated manuscript.
> We agree that this is consistent with the motivation found in the papers first introducing VIC and DIAYN.
But more recent work has begun applying the same methods to domains where episode length are impractically long (e.g. Atari is often >>10,000 steps per episode), and do so by iteratively resampling skills within an episode [7,20]. We correct our paper to not attribute this sampling scheme to the original papers, and will describe their evolution as above.
Thanks for clarifying, I was not aware of these further developments in the use of mutual information approaches. I now see where EDDICT can improve their performance, but is a mutual information approach the best way to address Atari in the first place?
> The goal for EDDICT (alongside all other intrinsic control methods) is to maximize the mutual information between a latent variable its downstream consequences on the environment.
Right, I would say this is the objective rather than the goal.
But mutual information is a surrogate objective in nature, my question was more related to the actual goal for which this objective is relevant: I think I am satisfied by other comments on this point.
> Figure 4b is a direct comparison between EDDICT a more traditional VIC method (EDDICT-$\Delta$ is both an ablation of EDDICT as well as being VIC with Gaussian codes) on an estimate of the mutual information objective they both care about. The dashed lines show that while EDDICT's codes are somewhat less discriminable, this is more than made up for by the increase effective number of codes being utilized. We will clarify this in the final draft.
I see that the mutual information induced by EDDICT is higher, but am I wrong to say that the marginal code entropy of VIC (EDDICT-$\Delta$) is fixed a priori by the designer? EDDICT-$\Delta$ is doing well on the conditional entropy, could it handle a more flat code distribution to discriminate even more codes?
On a side note, I found the experiment of Figure 4 to be much more informative about the method than all of the subsequent hard-exploration tasks. I would suggest the authors to consider other direct comparisons of the mutual information.
> I hope the above comments address the general sentiment, but as an aside I want to point out that both of these particular works are currently under review at this conference, so this isn't a reasonable request.
Let me clarify that I was not asking for a comparison with these methods, but just saying that a work that proposes a method for state coverage should compare with them (it does not seem to be the case here). To be completely precise, these papers have been posted on Arxiv in the early 2021, whereas at least (Seo et al., 2021) is published (ICML).
Instead, as a more relevant related work to EDDICT (still very recent) I would suggest (Liu & Abbeel, APS: Active pretraining with successor features, ICML 2021) that I mentioned in my original review.
Thank you for your continued discussion.
>I am still doubtful that mutual information is actually the best way to tackle those hard-exploration scenarios where the introduced change helps the most.
This is not an unreasonable opinion, however we strongly believe this question, while interesting in its own right, is orthogonal to our paper and thus should not preclude publication of these results. Our aim was to show that temporal correlations amongst codes yields a novel intrinsic control algorithm that models the controllable environment and improves on existing intrinsic control methods across various metrics. We hope that you'll agree we've done this.
>am I wrong to say that the marginal code entropy of VIC (EDDICT-$\Delta$) is fixed a priori by the designer? EDDICT-$\Delta$ is doing well on the conditional entropy, could it handle a more flat code distribution to discriminate even more codes?
You are correct, the conditional code entropy is fixed apriori for both EDDICT and EDDICT-Delta, but EDDICT is able to increase the marginal entropy via the temporal correlations between codes. DIAYN highlighted the difficulty of learning the conditional code distribution (it's solving another RL problem simultaneously) and subsequent work has abandoned this approach. Increasing the fixed conditional entropy would benefit both EDDICT-Delta and EDDICT, but preliminary experiments saw little effect for either condition.
>Anyway, the paper with the additional changes promised by the authors, along with a clearer presentation of the algorithm motivations, might provide sufficient value.
But I believe that a robust assessment would require a re-evaluation of the updated manuscript.
We are glad to have addressed your concerns and those of the other reviewers through additional baselines, a commitment to open-sourcing, more experimental rigor, and textual clarifications. Only this last point would significantly benefit from an evaluation of the updated manuscript (which can't be done during this submission process). We ask the reviewer to weigh this benefit against the additional time and effort entailed by requiring a new set of reviewers and another discussion period. You might not find intrinsic methods particularly useful for certain tasks of interest (e.g. exploration), but we strongly believe that there is a large enough body of prior work in the area to pass the bar of significance without being SOTA across a much broader class of algorithms.
We kindly ask the reviewer to consider raising their score. Your initial review highlighted the need for additional baselines and more methodological rigour. These have been substantially improved with the addition of multiple DIAYN baselines (sweeping over action entropy weights) and the ongoing extension of all results to include 10 seeds and only highlight statistically significant results.
> We are glad to have addressed your concerns and those of the other reviewers through additional baselines, a commitment to open-sourcing, more experimental rigor, and textual clarifications. Only this last point would significantly benefit from an evaluation of the updated manuscript (which can't be done during this submission process). We ask the reviewer to weigh this benefit against the additional time and effort entailed by requiring a new set of reviewers and another discussion period. You might not find intrinsic methods particularly useful for certain tasks of interest (e.
g. exploration), but we strongly believe that there is a large enough body of prior work in the area to pass the bar of significance without being SOTA across a much broader class of algorithms. We kindly ask the reviewer to consider raising their score. Your initial review highlighted the need for additional baselines and more methodological rigour. These have been substantially improved with the addition of multiple DIAYN baselines (sweeping over action entropy weights) and the ongoing extension of all results to include 10 seeds and only highlight statistically significant results.
This request from the authors is not unreasonable: I will raise my score from 5 to 6 to reflect my shifting opinion about the paper, and also to acknowledge their outstanding effort in addressing reviewers' concerns.
I agree with them that reviewers should be careful in forcing valuable papers to be resubmitted. Unfortunately, even if I now recognize that this work is interesting, I am still not confident enough about the significance of their contribution to raise my evaluation beyond borderline, and I would leave to the AC the responsibility to weigh the cost/benefit of rejecting the paper.
This paper clearly motivates and constructs EDDICT, making a compelling argument for the usefulness of imposing a latent dynamics model on the latents used for VIC, by showing such a formulation improves over standard VIC in state coverage. The results of the paper provide a useful contribution in demonstrating the benefits of learning relative latent goals in the framework of VIC.
I believe the paper could be strengthened by providing more understanding of how EDDICT performs as an exploration method, as well as how it compares to VIC.
Specifically:
- The authors frame the problem of VIC as a form of empowerment-based exploration, however, in the process ignore one of the original motivations of VIC, namely as a means for unsupervised option discovery. I believe it is fair to focus on the exploration aspect of VIC, but if the aim is to present EDDICT as an exploration method, comparison to at least one or two standard exploration baselines (e.g. RND, RIDE) should be included in the experiments section.
- It would be useful to see how the state coverage performance compares to DIAYN, which claims to improve over VIC in terms of diversity of sampled goals.
- Further, as the focus is on EDDICT as an exploration method, it would be valuable to include experiments showing learning curves for EDDICT in combination with extrinsic rewards.
- On the other hand, while not the focus of the paper's narrative around exploration, it could be interesting to understand how the latent, relative options learned by EDDICT can be used by a manager policy, similar to the HRL experiments performed in DIAYN. This would help us understand the relative merits of the relative goal representation in terms of discovering useful options, which was an original motivation of VIC, which the paper focuses on as the primary baseline.
Additional feedback
- The paper claims to have solved the "noisy TV problem," however none of the environments in the experiments clearly feature a noisy TV.  Providing results on an environment with such a noisy TV (e.g. adding a noisy TV to the Point Mass environment) would provide more support for this claim.
- Presentation of results for "Code norm as difficulty" section could be improved.
For example, showing a plot of code norm vs distance to the center frame of the starting room in Montezuma's Revenge. The current qualitative results are a bit hard to parse.
- In regards to the above point, what exactly is being shown in the right side of Figure 2? Is it the final state of several latent codes for each quartile, all overlaid on top of each other? If so, this should be clarified.
- Results in Table 1 are only over 3 seeds and the number of seeds is not detailed for the other experimental results. Error bars would allow for a more principled comparison of the methods.
- Related to the previous point on including exploration baselines, the conclusion claims "But in terms of pure exploration, EDDICT falls short of state of the art methods that learn a single policy", however, none of the results presented in the experiments section actually seem to directly support this claim, as the performance for these alternative SoTA exploration methods are not included in the reported results.
No
We thank the reviewer for their very detailed comments and constructive feedback.
“The authors frame the problem of VIC as a form of empowerment-based exploration, however, in the process ignore one of the original motivations of VIC, namely as a means for unsupervised option discovery. I believe it is fair to focus on the exploration aspect of VIC, but if the aim is to present EDDICT as an exploration method, comparison to at least one or two standard exploration baselines (e.g. RND, RIDE) should be included in the experiments section.”
We frame EDDICT in representational learning terms, but we view this as an extension to the notion of unsupervised option discovery. EDDICT still discovers options, but additionally situates them in a learned representational space.
For example, a particular option might correspond to ‘add [1.0, -0.5] to your current state’, whereas VIC options are independent of each other and any notion of state (e.g. ‘execute option 3’).
However, we acknowledge that this connection may be insufficiently explicit in the current draft, and will revise it with this in mind.
This ties into why we didn’t initially compare to ‘standard exploration approaches’, as they don’t yield conditional policies, and are thus solving a different problem. It is our position that these two classes of algorithms are interesting in their own right and at this relatively early stage in the field’s development exhaustively beating methods from the other class shouldn’t be a requirement for publication.
That said, we agree that showing results from these approaches would better contextualize our performance, and as such we commit to adding a new table comparing published results for unsupervised reward achievement on Montezuma’s Revenge. This would include RND, ICM, NGU, and others. Unfortunately, to the best of our knowledge RIDE has not been evaluated in this context, but we will still reference it as it is known to improve upon ICM and thus the ICM results can serve as something of a lower bound for potential RIDE results.
“It would be useful to see how the state coverage performance compares to DIAYN, which claims to improve over VIC in terms of diversity of sampled goals.”
We agree, and are now currently running 3 new baselines. The first is DIAYN with the same code resampling setup we use for EDDICT and VIC: a new code is sampled at short fixed intervals without resetting the environment. The second new baseline is DIAYN with the code duration swept to over two orders of magnitude (20, 200, 2000), which should account for the fact that the DIAYN paper used longer code duration (though in a different environment).
Finally, we are running versions of DIAYN and EDDICT with policy entropy terms (by switching the base RL algorithm from Q(\lambda) to soft-Q).
The initial results of the first baseline, shown after 500M steps in the table linked here, support our claim that EDDICT is not superseded by DIAYN, but might have complementary strengths. As can be seen in the table below, DIAYN has strong performance on a few of the games, but fails to match the progress on Montezuma’s Revenge, which remains our key result due to being well-known to provide a hard exploration challenge. We still need to sweep over entropy weight and skill duration, and will update this post as more results become available; we are particularly curious about the effect of policy entropy on both EDDICT and DIAYN.
“Further, as the focus is on EDDICT as an exploration method, it would be valuable to include experiments showing learning curves for EDDICT in combination with extrinsic rewards.”
While exploration is not the sole focus of this work, we agree that the combination with extrinsic rewards is a valuable use case. We have included a new experiment showing that co-training an exploration policy with EDDICT, a standard RL agent maximizing rewards can achieve up to [2.5k results on Montezuma to around 100M steps](https://imgur.com/a/0Dspd2a).
“On the other hand, while not the focus of the paper's narrative around exploration, it could be interesting to understand how the latent, relative options learned by EDDICT can be used by a manager policy, similar to the HRL experiments performed in DIAYN. This would help us understand the relative merits of the relative goal representation in terms of discovering useful options, which was an original motivation of VIC, which the paper focuses on as the primary baseline.”
We agree that HRL is an interesting potential use case, but as simply using EDDICT for exploration in a RL context has fewer design decisions, we have decided to focus our resources there as our primary additional experiment. It is also worth noting that the VIC paper didn’t actually perform any HRL experiments, despite its motivation.
“The paper claims to have solved the "noisy TV problem," however none of the environments in the experiments clearly feature a noisy TV. Providing results on an environment with such a noisy TV (e.g. adding a noisy TV to the Point Mass environment) would provide more support for this claim.”
As mentioned in the text, the "target position" in the Pointmass environment serves as a "noisy TV" in that it's a randomly positioned red dot in the agent's observation, which is uncontrollable but would nonetheless be represented by traditional representation learning methods (e.g. via generative modeling).
We initially excluded a figure explicitly showing this, as it’s rather uninteresting to look at (Figure 4a bottom, but with the colors bearing no relationship to the x nor y axis), but we will include it in the supplement so as to make this evidence explicit.
“Presentation of results for "Code norm as difficulty" section could be improved.
For example, showing a plot of code norm vs distance to the center frame of the starting room in Montezuma's Revenge. The current qualitative results are a bit hard to parse.”
Distance to the center frame doesn't quite capture the same information, as the agent occasionally leaves the first room, in which case the distance to the center of the frame isn't meaningful.
That said, we acknowledge that the current visualization is quite clustered and might be hard to parse (though other work has used a similar approach e.g. DISCERN).
As such, we will include a new version of Figure 2 in the appendix that shows 2D histograms of agent's x,y position in each code norm quartile.
“Results in Table 1 are only over 3 seeds and the number of seeds is not detailed for the other experimental results. Error bars would allow for a more principled comparison of the methods.”
Error bars were only excluded due to typesetting constraints, but a version of this already exists in the appendix. We will add a footnote in the main text referencing this.
“Related to the previous point on including exploration baselines, the conclusion claims "But in terms of pure exploration, EDDICT falls short of state of the art methods that learn a single policy", however, none of the results presented in the experiments section actually seem to directly support this claim, as the performance for these alternative SoTA exploration methods are not included in the reported results.”
As mentioned above, we will now report the alternative methods in a new results table.
Thanks for the responses! I still have some lingering concerns around some of these points:
> We frame EDDICT in representational learning terms, but we view this as an extension to the notion of unsupervised option discovery. EDDICT still discovers options, but additionally situates them in a learned representational space.
For example, a particular option might correspond to ‘add [1.0, -0.5] to your current state’, whereas VIC options are independent of each other and any notion of state (e.g. ‘execute option 3’).
However, we acknowledge that this connection may be insufficiently explicit in the current draft, and will revise it with this in mind.
I see your point, but I'm still not clear on how EDDICT's relative options can be transferred to additional environments. It seems that one advantage of a method like VIC or DIAYN is that you can learn a collection of options and use them as options in a HRL controller on a transfer task.
> This ties into why we didn’t initially compare to ‘standard exploration approaches’, as they don’t yield conditional policies, and are thus solving a different problem.
Related to the above point—how can such conditional policies be used at evaluation time?
That is, what should you condition on at test time? I feel that this point is not clearly addressed in the work.
> As mentioned in the text, the "target position" in the Pointmass environment serves as a "noisy TV" in that it's a randomly positioned red dot in the agent's observation, which is uncontrollable but would nonetheless be represented by traditional representation learning methods (e.g. via generative modeling).
This is not the definition of noisy TV, which specifically describes states emitting high entropy observations that aim to distract the agent from solving the MDP. The goal state is quite the opposite, as the agent needs to visit this state to solve the MDP.  Under your definition, all goal-based navigation tasks in RL are inherently noisy TV problems, which leads to reductio ad absurdum.
> Distance to the center frame doesn't quite capture the same information, as the agent occasionally leaves the first room, in which case the distance to the center of the frame isn't meaningful.
To clarify, by center frame of the starting room, I was referring to the center position in the first room, so an agent leaving the first room would continue to increase this distance from the center.
Nevertheless, your histogram solution sounds like a good approach.
> Error bars were only excluded due to typesetting constraints, but a version of this already exists in the appendix. We will add a footnote in the main text referencing this.
The more important issue here is that error bars are not that informative if you only have 3 seeds.
Given these remaining concerns, I will keep my rating at a 5 for now.
> I see your point, but I'm still not clear on how EDDICT's relative options can be transferred to additional environments. It seems that one advantage of a method like VIC or DIAYN is that you can learn a collection of options and use them as options in a HRL controller on a transfer task.
>Related to the above point—how can such conditional policies be used at evaluation time?
That is, what should you condition on at test time? I feel that this point is not clearly addressed in the work.
For HRL, transfer, and other use cases, you can simply forget the semantics underlying the EDDICT objective and just recall that EDDICT conditions it's policy on external noise ($\Delta_z$) drawn from a fixed distribution (Uniform on the unit N-disk in most of our experiments). This distribution thus specifies the space of conditional policies and can be used in the same way options derived from DIAYN or VIC can (the only real difference in terms of implementation is that we're left with an *infinite* collection of options).
While a full HRL treatment would involve too many additional design decisions to include in this paper, we have added an experiment that shows a very simple way to utilize EDDICT's options in this black-box manner. We train EDDICT alongside a standard task reward maximizing policy (i.e. the standard reward function of game score). We flip a coin every 20 steps to decide whether or not to follow the RL policy for the next 20 steps or execute one of EDDICT's options (chosen at random).
Despite its simplicity, this method achieves a score of 2.5k points on Montezuma's Revenge in around 100M steps ([link to learning curves](https://imgur.com/a/0Dspd2a)), which is quite competitive with state of the art methods. We hope this explanation and proof of concept experiment address this concern.
>This is not the definition of noisy TV, which specifically describes states emitting high entropy observations that aim to distract the agent from solving the MDP. The goal state is quite the opposite, as the agent needs to visit this state to solve the MDP. Under your definition, all goal-based navigation tasks in RL are inherently noisy TV problems, which leads to reductio ad absurdum.
We didn't mean to imply that goal-based navigation tasks are noisy TV problems, and apologise for the confusion.
While the "target position" in the point mass environment does act as a goal *when the standard task reward function is present*, in the unsupervised setting considered in this experiment (the reward is not observed nor used intended for evaluation), it is simply a red point on the screen, randomly repositioned at regular intervals. In this context, it is just a source of observation entropy akin to a noisy TV -- one which EDDICT's learned representational space is invariant to.
>To clarify, by center frame of the starting room, I was referring to the center position in the first room, so an agent leaving the first room would continue to increase this distance from the center.
Nevertheless, your histogram solution sounds like a good approach.
Indeed, we agree that would be a useful metric, but one that is unfortunately surprisingly hard to compute, due to some quirks of the way the Atari emulator exposes information. We are glad the histogram solution will suffice, and we thank the reviewer for pushing us towards more accessible visualisations.
>The more important issue here is that error bars are not that informative if you only have 3 seeds.
We initially chose 3 seeds due to the amount of overall time and computation required for these experiments, and for its standard usage in prior work (likely for these same reasons).
That said, we acknowledge that more seeds would strengthen our claims of significance, and as such we hereby commit to running a total of 10 seeds per condition if that would substantially change your evaluation of this work.
Thanks for the additional explanations! Based on your clarifications, I'm happy to raise my score if you can provide some indications that the main results still hold with more seeds.
We have run additional experiments to bring the total seeds per condition to 10. Here is a [link](https://imgur.com/a/DPr5Xrv) to the new results, with only clear winners bolded (i.e. no overlap in the confidence intervals).
While there are still fewer clear winners than under the more liberal bolding scheme (even with the extra seeds), EDDICT is now the only method significantly beats all other baselines on any game by any metric.
Indeed, EDDICT wins by at least one metric on 4 of the 6 games.
We also commit to running 10 seeds for VIC and DIAYN and updating the table for the final draft, but were unable to do so during the response period due to resource limitations. We prioritised RVIC and the EDDICT ablation because our initial results suggested that they were the strongest baselines.
Strengthens:
Compared to prior methods, the globally consistent codes of the proposed method allow it to be far more exploratory. And the corresponding experimental results also demonstrated that the proposed method could promote the state coverage and increase unsupervised performance
Weakness:
The experimental result part is not sufficient.
It seems that there are some errors in Fig.2 and Fig.3. Inserting problem?
I am not an expert in this research field. Even my current score is 5, I will change my score in the rebuttal stage and refer to other reviews. I will check the response carefully and give my final rating in the rebuttal stage.
No
We thank the reviewer for their comments. Other reviewers had concerns about some specific aspects of the results, which we have substantially improved. We hope these changes address your concerns as well, but if not then please let us know if there is a specific shortcoming that we have failed to address in the other rebuttals.
We’ve collected all of the things we’ve promised reviewers below. We commit to adding the results of 1-5 as they come in during the discussion period (via anonymous imgur links), and we commit to carrying out the full set of changes in the final version of the paper, with open-sourcing to be completed prior to the conference date.
1) [DIAYN baseline](https://imgur.com/a/zMo9xEs), sweeping entropy weight and goal duration
2) EDDICT with dense rewards and policy entropy (akin to DIAYN)
3) Random policy baseline to contextualize the coverage metric
4) EDDICT ablations for hindsight correction and greedy optimization
5) Co-training EDDICT and task reward on Montezuma’s Revenge ([learning curve](https://imgur.com/a/0Dspd2a))
6) Open-source a stand-alone version of EDDICT for running on toy environments
7) Qualitative results other DM Control Suite tasks (HalfCheetah, Reacher)
8) New figure showing EDDICT's representations don't capture uncontrolled distractor in PointMass
9) Additional version of Figure 3(right), where images are replaced by 2D histogram of avatar screen position
10) New table comparing previously published unsupervised scores on Montezuma’s Revenge
11) Comparison to ICM in the main results figures
12) Add societal implications section
# Comparison to single policy exploration methods.
Several reviewers brought up the relationship between EDDICT and other purely exploration focused methods, such as RND, CPT, and intrinsic curiosity methods, e.g. ICM, RIDE. As stated in the paper, we fundamentally believe that head-to-head comparison should not be the focus of this work, as all of these methods learn a single exploratory policy, which have different potential applications compared to the conditional policies learned by EDDICT and other intrinsic control methods (e.g. DIAYN, VIC).
For example, with a single policy it’s less clear how to do HRL, goal-achievement, and imitation; all of which were addressed in the DIAYN paper (amongst others e.g. RVIC) and can be broadly applied to this class of methods with minor modification.
That said, in the updated paper we will include a table of previously published results so as to better contextualize EDDICT’s unsupervised performance on Atari.
# Comparison to DIAYN
Diversity Is All You Need is a prominent intrinsic control method, which was not initially used as a baseline in this paper. Our position was that it was sufficiently similar to Variational Intrinsic Control (VIC) so as to obviate its inclusion.
But several reviewers point out that its dense reward signal might significantly change performance relative to our other baselines. We agree, and are now currently running 3 new baselines. The first is DIAYN with the same code resampling setup we use for EDDICT and VIC: a new code is sampled at short fixed intervals without resetting the environment. The second new baseline is DIAYN with the code duration swept to over two orders of magnitude (20, 200, 2000), which should account for the fact that the DIAYN paper used longer code duration (though in a different environment).
Finally, we are running versions of DIAYN and EDDICT with policy entropy terms (by switching the base RL algorithm from Q(\lambda) to soft-Q).
The initial results of the first baseline, shown after 500M steps in the table ([linked here](https://imgur.com/a/zMo9xEs)), support our claim that EDDICT is not superseded by DIAYN, but might have complementary strengths. As can be seen in the table below, DIAYN has strong performance on a few of the games, but fails to match the progress on Montezuma’s Revenge, which remains our key result due to being well-known to provide a hard exploration challenge. We still need to sweep over entropy weight and skill duration, and will update this post as more results become available; we are particularly curious about the effect of policy entropy on both EDDICT and DIAYN.
# Utility in conjunction with extrinsic rewards
We strongly believe that learning a rich set of skills which are tailored to the particular environment dynamics, but agnostic to the reward function is key to improving data efficiency in reinforcement learning. This paper is primarily concerned with the first step in this direction, improving on SoTA intrinsic control methods by introducing temporally correlated skills which improve coverage and improved representations.
While we do believe the resulting options could be used as part of a full HRL agent (as done in DIAYN), we believe this is best left as a follow-up paper. We do have reason to be optimistic however: by co-training an exploration policy with EDDICT, a standard RL agent maximizing extrinsic rewards can achieve up to 2.5k results on Montezuma in around 100M steps ([learning curve shown here](https://imgur.com/a/0Dspd2a)).
# Code release
Reviewers raised the point that the replicability of EDDICT would benefit from a concrete implementation. We agree wholeheartedly, and commit to releasing a stand-alone IPython notebook, prior to the conference in December, that will allow anyone to run simple experiments that replicate our claims about EDDICT’s representations (e.g. distractor robustness), exploratory behavior and utility in conjunction with task rewards.
# Effect of Distractors
Reviewers expressed concerns about the qualitative Point Mass experiments which we believe to be of general interest. There was a semantic typo in Figure 4a, wherein we wrote we’re comparing to VIC, when we were actually comparing to the ablated version of EDDICT (EDDICT-\delta). With this correction in place, we hope that the intent and findings are clear, i.e.: this experiment demonstrates the necessity of the latent dynamics (as this was all that was removed by the ablation). The other claim in this experiment is the demonstration that EDDICT inherits the property (shared by all intrinsic control methods) of being agnostic to uncontrollable aspects of the environment. The Point Mass environment naturally has one such distractor in the form of the “target position” -- a static red circle whose location is randomly determined every 1000 steps. We claimed to be agnostic to the position of this distractor, but this is not obviously the case from the present Figure.
As such, we’re adding a new figure to the appendix depicting the effect of varying the distractor position on the latent representation while holding the point-mass’ position constant (there is no effect).
# Limitations And Societal Impact
Reviewers mention that our discussion of societal impact is limited to the checklist, and that some further discussion is warranted. We concur, and here is the additional statement we will be including in the final version of the paper:
Unsupervised reinforcement learning in general, and intrinsic control methods in particular, are far from being commercialized due to their insufficient data efficiency and lack of validation in real world environments.
However, when this is no longer the case, these methods could significantly reduce the human cost of setting up systems that interact with humans (e.g. robotics), as these methods limit the need for handcrafted reward functions and the collection of human preferences.
But this benefit comes with a cost to interpretability and safety. The information theoretic objectives of the methods lead to behavior that can be very hard to predict a priori (e.g. what does ‘controlling your environment’ look like?).
Furthermore, safety constraints might be harder to specify in the absence of a closed-form reward function. As these methods mature, the emphasis should shift from raw performance to a more nuanced approach that addresses these societal concerns head on.
Two of the reviewers expressed concern over the significance of our results, and we agree that while 3 seeds is not uncommon in the field, we should aim to do better.
We have run additional experiments to bring the total seeds per condition to 10. Here is a [link](https://imgur.com/a/DPr5Xrv) to the new results, with only clear winners bolded (i.e. no overlap in the confidence intervals).
While there are still fewer clear winners than under the more liberal bolding scheme (even with the extra seeds), EDDICT is now the only method significantly beats all other baselines on any game by any metric.
Indeed, EDDICT wins by at least one metric on 4 of the 6 games.
We also commit to running 10 seeds for VIC and DIAYN and updating the table for the final draft, but were unable to do so during the response period due to resource limitations. We prioritised RVIC and the EDDICT ablation because our initial results suggested that they were the strongest baselines.

Meta-review:
role: PC
data: comment: To reliably reach a large number of different states in the environment requires AI agents to be able to manipulate the environment effectively. 
Practically, this type of exploration reward commonly aims to disambiguate between a set of latent codes using short horizons of future states, which limits the “exploration depth” of the agent.

Instead, this work proposes to generate conditional latent variables which effectively let the agent explore states that cannot be reached from a single unrolling.

The paper presents compelling motivation and theory for the method. However, as pointed out by multiple reviewers, the experimental results are not very compelling. In particular, the number of seeds evaluated is small and the EDDICT results are obtained by maximizing the performance over the size of the codebook, which clearly will bias these numbers. 


One important topic of discussion was the experimental validation of the method. The authors ran additional seeds for some of the experiments in the paper and found that EDDICT still significantly beats a baseline (RVIC) and an ablation (EDDICT-Delta). 

This line in the paper is concerning: “For EDDICT-based agents, we pick the best metric across code sizes.” 
This sounds like the  _best_ code size for EDDICT was reported for each metric. If so, that will both introduce bias and make the results incomparable to the baselines. 

As pointed out by the reviewers, the current results fail to provide convincing experimental evidence that EDDICT works better than the baselines. This issue is compounded by the potentially biased evaluation maximizing over different code-sizes. Given these concerns, I do not believe the paper is ready for acceptance at this point. 


Decision: Accept (Poster)