Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper introduces a new benchmark for meta-RL research, emphasizing transparency and potential for in-depth analysis and structural richness. The proposed Alchemy environments give the two features: 1) structural interestingness, which features compositionality, causal relationships, and opportunities for conceptual abstraction; 2) structural accessibility, in which the ground-truth parameterization of the distribution should ideally be accessible. The authors also conduct experiments with two meta-learning algorithms in the Alchemy environments. The results show that the agents failed in structure learning and latent-state inference and thus displayed poor performances in the Alchemy environments.
strengths: The structure information is interesting for RL methods. The implementation of Alchemy gives the features via the generating graph topology. The experiments on VMPO and IMPALA demonstrate that the tasks are challenging for the two algorithms. The author also gives ablation studies.
weaknesses: 1. As far as I understand, this paper proposes 1 environment. So how representative the environment is should be clarified, e.g., does it contains properties that many other tasks may share.

2. The task generating method, i.e., the task is resampled at the start of each episode, is only one possible setting for meta-RL. However, such a setting is unfriendly for RL training, since it creates a dynamic environment that can injure the learning. Another setting where the sampled tasks are fixed at the beginning is thus a common choice.

3. As a benchmark, this paper should survey more about related benchmarks, and test more meta-RL methods.

4. The paper show that the environment requires a good latent structure to be learned in order to achieve a good performance. The environment is interesting for validating latent structure learning. So, why not focusing on proposing an environment for structure RL? One reading the paper is hard to feel this paper is related to meta-RL, which addressing questions like what to extract from past tasks and how to transfer to new tasks. These questions seems orthogonal with learning latent structure.
correctness: It seems that there is no hyper-parameter search. Not sure if the empirical conclusions still hold after a well search.
clarity: The paper is easy to follow.
relation to prior work: The prior work of environments designed for meta-learning should be discussed in detail.
additional feedback: none
review_title: Clarifications are needed

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper describes a 3D video game environment developed in Unity. The environment has an underlying latent causal structure that needs to be identified at the meta-level for an agent to perform well. The authors argue that their environment is accessible (ground truth available) and interesting (similar to challenging real-world tasks featuring compositionality, causal relationships, and opportunities for conceptual abstraction with solutions requiring strategic sequencing of actions). The environment can be image-based (3D) or feature-based (symbolic). They show that recent strong algorithms fail to identify the latent causal structure and how augmenting image-based and feature-based versions of one of the algorithms with various forms of "privileged information" gets their performances closer to an oracle's.

strengths: The experiments are very interesting and thorough. The authors study VMPO and IMPALA as baselines and compare their performances with a Bayes ideal observer with a belief state, an oracle and a random heuristic agent. They also augment two versions (image-based and feature-based) of the VMPO baseline with inputs of the belief state/ground truth to analyse the impact of these and show that their performances improve.

They also try out experiments to disentangle whether task sequencing or complexity of visual observations hold back the image-based VMPO agent from utilising privileged information which the feature-based VMPO was able to utilise.

They further try out helping the image-based and feature-based VMPO agents by adding auxilliary tasks of predicting certain features which drastically improve their performances.

weaknesses: 1) **The clarity of the paper could be improved in many places**

I will now use quotes from the text and describe my issues here.
>As shown in Figure 3a, the agent used more potions during episode-initial trials than the ideal observer benchmark.

It is not clear which bar "agent" in the above sentence refers to in Fig. 3a and thus it is difficult to parse the figure. Why not already call the agent by the name used in the legend of the plot? For example, "Input: Ground truth" and "Input: Belief state" haven't been defined in the text yet at this point and I found it hard to make out if "Input: Ground truth" may be the oracle or the ideal observer. This is especially so because the oracle or the ideal observer are not mentioned in the figure caption until the end and I had read the caption only till where it describes sub-figure "a" because the text refers to Fig. 3a. I would recommend to add the ideal observer and oracle also to the graph legend instead of just in the caption.

I think *I* know why the oracle is only shown in sub-figure a in Fig. 3 but it would be nice to add the reasoning behind this to the caption because it took me a while to figure it out.

I did not understand the "Synthetic models" paragraph. Especially, what "fit of these two models" and "was compared for the behavior of the ideal observer reference agent". Additionally, the sentence
>In both models, predictions were based on a posterior distribution over the current chemistry.

could just say that the input was the belief state. Then the terminology would be consistent with the figure caption. in my opinion, scientific papers should try to use consistent terminology to reduce the cognitive load of understanding them.

>When furnished with this side information, the agent’s average score jumped dramatically, landing very near the score attained by the ideal observer reference (Figure 2a ‘Symbolic’).

Did you mean Fig. 2b? I don't see this in 2a (this error is repeated multiple times later). Again, it could have been mentioned here that the reader should look at the "ground truth" bar in the left half of 2b instead of saying "side information". The "ground truth" aspect is only mentioned in the next sentence but I was already trying to parse the figure after reading the quoted sentence above.

>the VMPO agent might have more trouble capitalizing on the given side information in the 3D version of Alchemy because doing so requires composing much longer sequences of action than does the symbolic version of the task.

Why does it require longer sequences?

>compositional latent organization

Compositionality was not discussed in the text. I would like that it is described why Alchemy tasks feature compositionality.

I also find the term "3D Alchemy" misleading because I think the authors mean the image-based task here but according to me even the symbolic version is inherently describing a 3D world and thus I find "3D Alchemy" a misnomer.

What does "individual replicas" mean? Seeds? Why are they 5-8 and not a fixed number?

In line 300, please mention what part of the Appendix.

Sentence beginning in lines 300-306 could be improved grammatically.

>a technique frequently used in cognitive science to analyze human learning and decision making

Citation needed.

Figures 3 and 4 are discussed before Figure 2. And it is hard to keep moving back and forth between the text and the figures.

Why does "None" have 2 colours in Fig. 2?

>Note that to surpass a zero score, both agents required kickstarting [29], with agents first trained on a fixed chemistry with shaping rewards included for each potion use.

It was not made clear that "both agents" meant the image-based agents here.

>Informal inspection of trajectories from the symbolic version of the task was consistent with the conclusion that the agent, like the random heuristic policy, was dipping stones in potions essentially at random until a high point value happened to be attained.

When exactly does an episode end is not clear to me?


2) **The paper could do a better job of describing its limitations**

Appendix E mentions "thorough discussion of limitations" but I did not see this.

In my opinion, the tasks are still some very specific tasks and cannot cover the whole range of tasks we see in the real world. This is fine but should be mentioned in the limitations.

The environment is very expensive. It seems to me that there is no real signal even until one billion timesteps in Figure 7. This should also be mentioned in the limitations.

correctness: I believe the claims are correct.
clarity: I had already read the arxiv version when it came out but it still took me a while to properly understand the many things mentioned above. But apart from those, the paper *is* mostly well-written.
relation to prior work: I think the comparisons with related work could have been more detailed and possibly a section of their own.
additional feedback: I am curious to know how the hyperparameters were set. Given how expensive the environment is, this information is valuable I believe.
review_title: Very interesting work that helps perform in-depth analyses for meta-RL; should improve clarity and describe limitations in more detail

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors propose the Alchemy benchmark, which is an environment with parametric procedural generation, allowing to propose a diversity of complex tasks to meta-RL agents. Using Alchemy, authors conduct experiments with two SOTA Deep RL algorithms, and provide an in-depth analysis of their (low) performance results, showcasing that the meta-RL challenge of Alchemy is relevant.
strengths: First of all, note that I am not an expert on Meta-RL, my domain of expertise is more on Automatic Curriculum Learning (ACL) & Deep RL.

This benchmark seems relevant to the meta-RL community, as it proposes a standardized testbed on which to test meta-RL agents. It is well documented and as such it seems accessible for a large audience. Computation costs seems quite high, but a symbolic version of Alchemy is provided, which is good for research teams with low compute budgets.

Actually, the meta-RL objective studied here is quite different from classical Meta-RL works, e.g. MAML, or the Meta-World paper, which focuses on the problem of meta-learning a new task in a few gradient updates. Here agents must learn over the course of an episode, without updates, which is an interesting setup.

Also, since one of Alchemy strengths is to have an accessible parametric task distribution, i.e. a task space, I'm wondering whether this platform could also be used as an ACL benchmark. It would only require the additional assumption that the ACL algorithm is allowed to control the chemistry sampling procedure. Anyway, this is most likely out of the scope of this paper.
weaknesses: For me a *potential* major weakness is the lack of Meta-RL baselines. VMPO and IMPALA are SOTA Deep RL agents, without meta-RL mechanisms. I am wondering whether there are better meta-RL baselines in the literature ? Maybe not, because the benchmark assumes no gradient updates at test time. If it is the case that there is no alternative meta-RL baseline agent, then I think that the paper would benefit from clearly stating this.

I am looking forward to discuss with authors and other reviewers on this subject. If this problem is clarified, I will gladly increase my score.
correctness: Evaluation methods are appropriate, clearly stated, and performed correctly. Running more seeds per tested conditions would have been better, as it would have allowed to perform statistical significance tests, but the current status of the manuscript is acceptable w.r.t. the current state of the DRL literature.
clarity: Excellent writing quality.
relation to prior work: Related work appears to be properly discussed
additional feedback: What is the Bayes optimal agent exactly accessing to ?
review_title: An interesting benchmark, potentially NeurIPS worthy, but clarifications are needed

Meta-review: This work introduced a Unity-based 3D environment for evaluating the ability of meta-reinforcement learning agents to infer latent causal structures from interactions. At the initial reviews, this work received 5, 7, 7 from three expert reviewers. Reviewer 1Tpo requested clarifications on task design and survey about related benchmarks. Reviewer WAyX and WeQ5 felt positive but also raised several clarification questions about the technical details. The authors provided detailed responses to these questions and addressed the reviewers' comments well. In the end, Reviewer WeQ5 agreed to upgrade their rating from 7 to 8 but didn't get a chance to do so before the review system is closed in the final phase. The AC read the paper, the reviews, and the authors' discussions in detail, and agreed with the reviewers that this paper introduced an interesting new benchmark to facilitate research in the challenging meta-RL domain. Taking all the factors into account, we recommend accepting this paper in this track.
Decision: Accept