Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of covariate shift in recent adaptive curriculum learning methods. They propose to address this by changing how the adversarial approach interacts with aleatoric (stochastic) parameters and demonstrate it on two 2D grid environments.

main_review: The problem here appears to be that an adversarial curriculum approach is allowed to modify a stochastic parameter that is not observable at run-time by the agent, but drastically changes the optimal policy. I had difficulty understanding the authors' approach, which at first glance appears very complicated but somehow reduces to something extremely simple in their experiments. This might be due to their choice of rather simple 2D grid-world type environments. I believe they may be correct in identifying a problem in this type of approach, but I am not sure that this is novel.

I have some concerns with this paper:
- Adversarial curriculum learning over a parameter that completely changes the optimal policy seems like an obviously bad thing. The (Dennis et al, 2020) paper you cite even mentions in the abstract that minimax often leads to unsolvable environments. Can you not just exclude these manually?

- Their approach is introduced in sec 2.2 by analogy to "cryptic conventions" in cooperative MARL, which is unfortunately not very intuitive unless you have a background in this subject. Reading this, I am not sure this connection is needed or helpful. 

- "In practice, it is often the case that environment dynamics are uniquely determined by θ′ so that
P(θ′|τ) = 1" - EDIT for clarity: The math indicates you mean there other way around - that the curriculum parameter can be uniquely determined by the trajectory. This might be correct in your simple fruit environment, but seems like a questionable assumption in general? One could move a box in the environment that does not impact the policy, or if you have random elements in the agent actions, noise in agent transitions, or observations, you might get similar trajectories using very different curriculum parameters. This will also complicate the modelling you make when learning \bar{P}(theta|tau), in the general case it seems like you might need a more complex density estimator than e.g. a regular Guassian model.

- The experiments use very simple environments. As I understand it, there is only one curriculum parameter in the fruit world?

Minor:
typo: "persisitent"
summary_of_the_review: While automated curriculum learning is an important topic, the purported problem they are attacking seems a bit too obvious. I am not sure how novel this is, and their solution appears needlessly confusing and demonstrated only on toy examples.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a solution to the problem of covariate shift appeared in adaptive curriculum learning where the distribution of parameters of the environment at test time is different from the one at training time. This is caused in algorithms like PAIRED or PLR because the adaptive curriculum learning algorithm biases towards regret maximising environments and as such trains the learner on a biased distribution instead of the ground truth one. The authors propose a solution to the problem under the assumption that the designer knows the ground truth distribution of the parameters of the environment. The solution relies on using fictitious transitions to compute the value function, using Off Belief learning — an algorithm for cooperative MARL.
main_review: **Strengths:**

This work delivers what is promised experimentally. It tests the algorithm on NetHack and MiniHack and demonstrates that the proposed algorithm reduces the gap between training and test time evaluation.

**Weaknesses:**

There are a several issues with the clarity. I've expanded those comments in the clarity section of the review.

**Detailed Comments:**

C1: In figure 1 text you say ground-truth distribution $P(\theta)$ where in the main text you use $\bar{P}(\theta)$ for the ground-truth. I assume that the ground-truth is notated as $\bar{P}$, can you verify that this is correct and fix the figure 1?

C2: The belief is $P(\tau_t \bar \tau^i_t, \theta)$. What is $\tau_t$ since $\tau$ is $(s1, a1, s2, a2, \dots)$? Is the subscript index the state only? I know that this is inherited by off belief work but would be good to clarify the notation.

C3: What is the difference between $\bar{Q}$ and $Q$?

C4: "If $\bar{P}(\theta \dots)<1$ .. then some $\theta'$ result in irreducible uncertainty". Let's examine an example. Say we have $\theta_1$, $\theta_2$ and $\theta_3$. Say we have a trajectory $\tau$ for which the transitions and emissions where the same for all three environments but in the future all three environments end up in different states. Therefore, $\bar{P}(\theta \bar \tau_t)<1$ (since we can't tell which one of the parameters is for sure). Your claim is that this is aleatoric uncertainty however, you'll be able to reduce such uncertainty if you progress in the environment and acquire more evidence. 

C5: You say "we see that $\bar{Q}*$ remains optimal ... as long as $\bar{P}(\theta' \bar \tau_t) = 1$" - Can you explain why is this the case? I can't see why you rely on $\bar{P}(\theta' \bar \tau_t)=1$ for the optimality argument.

C6: Shouldn't "if d = 0 then" be "if d = 0 or $\Lambda$ is empty" in algorithm 1?

C7: You say $\lambda=\bar{P}(\theta)$. Is $\Lambda$ a distribution or a parameter? Also what is $M^\Lambda$? Is it the simulator where you sample parameters from the distribution $\Lambda$? Is it fair to say that in the expectation in eq 5) you sample a $\theta \sim \Lambda$ and then play $\pi$ in the environment parametrized by $\theta$ and gather the trajectory $\tau$?

C8: Related to C7 but at some point you say $\bar{V}^\theta$ which confuses me. Is this an expectation like $\bar{V}^\Lambda$ ?

C9: In the experiments, can you clarify how many seeds were run and what's the shaded area means?


C10: Related work: [https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14378](https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14378) - this work is using importance sampling to fix the bias of sampling rare environments
summary_of_the_review: I'm suggesting marginal below acceptance threshold because I find certain parts of the paper unclear. The experiments support the argument of the paper which is great however I'd like to understand and see the paper improved in a few aspects:
- Understand if the notion of aleatoric parameters is the correct one here. See C4
- Understand C6 and C7 better in order to follow the complete argument and the proofs.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aim at generating curricula for training an RL agent to solve tasks sampled from an unknown distribution. Based on Prioritized Level Replay and Off-Belief Learning, an algorithm is proposed to train the policy using trajectories sampled from a learned belief model. The proposed method is evaluated in two goal-reaching tasks of discrete state and action spaces.
main_review: Strengths:

- This paper studies unsupervised environment design which is an intriguing and important problem to facilitate reinforcement learning with minimum human intervention. 

- The proposed method consistently outperforms TLR and DR baselines in two discrete space task domains.

Weaknesses: 

- The problem formulation does not seem to be entirely convincing to me. In this paper, an inference model $\bar{P}(\theta | \tau)$ is trained with the policy to predict the environment parameter $\theta$ given $\tau$. Given this belief model, why do the authors still stick to a Q value function as a marginalization over θ in Eqn. 2 instead of using $\bar{P}(\theta | \tau)$ to conduct online system identification and learns a policy $\pi(a | s, \theta)$ conditioned on the inferred $\theta$? Wouldn't that lead to better performance?

- The definition of the belief model in Eqn. 4 does not seem to be correct. Based on the Bayes' theorem, the RHS is supposed to be $\sum_{\theta'} \bar{P}(s_t | \tau, \theta') \bar{P}(\theta' | \tau)$. The marginalization is missing here. 

- Several definitions are missing, which makes the paper hard to read. For example, how is $\mathcal{M}^{\Lambda}(\pi)$ defined? In Sec.2, $\mathcal{M}$ is defined as the tuple of UPOMDP but in Sec. 6 it seems to become a probability distribution. The definition of $\bar{Q}^*_{\bar{P}}(a_t | \tau_t, \theta)$ is also missing.

- Could the authors provide more discussions on how the fictitious trajectories help mitigate the curriculum-induced covariate shift? I do not think the explanation is sufficiently convincing in the current draft.

- The experiment setups seem to be a bit too simple. As shown in Figure 3 and Figure 6, the different goals are always placed next to each other in the same room. And there is no penalization for reaching the wrong goal. Therefore, the policies for solving different $\theta$ will only have minor differences. Specifically, the policy only needs to distinguish the goals in the last 1-2 steps, but the actions before that will be exactly the same. As a result, the learned Q value functions will be insensitive to the changes of $\theta$ except for the final state. Without evaluating the proposed method in a more challenging setup (e.g. different goals placed in different rooms, and negative rewards for reaching the wrong goal), it would be hard to justify the effectiveness of the proposed method. 

- Although the authors claim that the proposed method generates curricula to facilitate the training of the policy, there is no quantitative or qualitative results that demonstrate what kind of curricula are generated. It would be better to include an analysis of how $\theta$ evolves during training and how it facilitates the training. 
summary_of_the_review: Although the paper tackles an important problem, the proposed technical solutions do not seem to be entirely convincing and the experiments are insufficient. I believe this paper needs another iteration before being ready for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Paper proposes a method to fix the covariate shift issue induced by curriculum learning RL methods when dealing with parameterized POMDPs. Specifically, curriculum learning methods choose a training distribution over the parameters of the POMDPs that might differ from the distribution that the agent might face during test-time, therefore the optimal solution under the distribution induced by the curriculum method might not be optimal under the test-time distribution. To fix this problem, the paper assumes access to the simulator such that given any previous trajectory the next state can be sampled under the true/test distribution, which is then used to update the policy. 
main_review: 1. I am not sure what the "unsupervised environment design" in the title has got to do with the paper. The proposed method only selects/corrects for a distribution over a known set of environments. 

2. I am really confused about what is the formal problem statement that the method is trying to solve, and what are the assumptions and properties available to the agent. Having a clear formal statement with assumptions after the introduction could be really helpful.

3. Since the method builds so heavily upon the previous work on PLR, it is worth including a summary of it in the background.

4. What do super-scripts of $\mathcal M$ in the POMDP tuple stand for? They are not used anywhere else again.

5. Why is the observation function assumed to deterministically map states to observations? Why is reward only a function of state, and not (s,a,s')? How do these choices impact the method?

6. In the definition of $\mathcal B$, it seems like $\tau_t$ is used as both the trajectory till step $t$ and the state in $\tau$ at step $t$. I am not sure how to interpret this.

7. In eqn 1, what does it mean to have $\pi_0 \rightarrow \pi_1$ in superscript? I don't see this notation being used like this anywhere else in the paper. 

8. The paper introduces again introduces a new terminology and additional discussions on _mistake_ vs _misunderstanding_, but does not really use it again after. Further, I don't really see the difference, _mistake_ seems to be a sub-problem resulting due to _misunderstanding_, resulting from distribution mismatch where supports are not overlapping.

9. Paper uses "grounding X" everywhere, where X is some parameter. I am not sure what the authors mean by that. 

10. One of the biggest drawbacks of the work is the assumption about access to the simulator. To perform the off-belief learning, the simulator is used in parallel to sample the next state transition under the true distribution of POMDP parameters instead of the distribution induced by the curriculum learning procedure.
 - a) The entire point of model-free methods is to not rely upon access to the simulator and just work with the data.
 - b) If the simulator/model is assumed to be known and accurate, then the comparisons should be made with other methods that leverage the simulator access as much as possible.
- c) Even when the simulator is known, how tractable is it to reset to a state corresponding to a particular trajectory's history under a different parameters of the POMDP, specifically when transition dynamics are stochastic? That is, how do you bring the parallel stochastic simulation to a state that corresponds exactly to observing that specific history?

11. It is later mentioned that "in practice, it is often the case the environment dynamics are uniquely determined by $\theta'$", and it is discussed how this can be exploited. However,  in this case, isn't the main motivation for aleatoric uncertainty as developed in eqn (2) and (3) non-existent?

12. Entire section 6 on Bayes optimality seems very uninformative. The arguments presented are only good for the tabular setting, which is probably not where one would be using curriculum learning or seeing the aleatoric uncertainly problem in practice. With function approximations you will need to additionally constrain policy comparison to policies with the function class, talk about optimality gap between the best policy in your class and $\pi^*$, and also discuss why it is reasonable to think that the proposed method will converge to $\epsilon$-Nash Equilibrium when using these function approximations.

13. While in the ideal setting the proposed method can address the covariate shift in the return _given_ a state, I am not sure how the distirbution shift in observing the specific states itself impacts the test performance. This is not a problem in tabular because states are not aliased, but in the function approximation setting the states that are visited often will tend to have a better policy as compared to the states which are visited less frequently. But if these less frequently visited states tend to get visited more frequently during test time, then there might be problems.

summary_of_the_review: I think the paper falls short because of strong assumptions made by the proposed method on simulators, uninformative theory, and poor presentation of the problem being addressed.
review_title: null

Meta-review: The paper tackles the problem of covariate shift in adaptive curriculum learning.  Unfortunately, the paper lacks clarity and the experiments are insufficient.  The author response clarified the notation and corrected many typos, however, the paper remains conceptually unclear as pointed out by the reviewers.  Hence this work is not ready for publication.
Decision: Reject