Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents a new interactive environment which is both text-based and contains visual simulation which are aligned. The authors also propose a first agent architecture which uses the visual observations as well as the text-based (named BUTLER). The authors tested the generalization capabilities of the proposed BUTLER architecture compared to a seq2seq transformer model.

Strong points:
- novel environment for text-based and aligned visual content (could potentially lead to follow up research) - a significant contribution to the community.
- have demonstrated that visual representation helps to generalize in these kind of environments (text + visual)
- the paper is nicely written and easy to follow
- the figures plots and tables are clear and help to understand the research

Weak points:
- the complex system: e.g.,  a pre-trained M-RCNN, a pre-trained text-agent, and training the text-agent using imitation learning (DAgger) biases the experimentation (makes the results less convincing). Maybe an intermediate naive baseline should have been considered.
review_title: An Aligned TextWorld and Visual Environment

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper describes an approach for training an agent which completes goals specified via a language instruction in the ALFRED environment. The paper proposes a model and training scheme. 

The model decomposes the problem into three steps: (1) a perceptual module which takes as input an environment observation and generates a textual description of it including the objects and their spatial relations, (2) a goal-planning module which takes as input the high-level goal (which may require completing multiple subgoals), the textual description generated from module 1 and generates a textual description of a subgoal, such as an action the agent should take with arguments, and (3) a controller module which takes as input the state of the environment and the subgoal description generated from module 2, and generates a sequence of actions which execute this subgoal. The entire model is ran each time the agent completes a subgoal, using the current environment observation as input to module 1.

The proposed training scheme focuses on pre-training the second module. This module is pre-trained using a proposed environment, ALFWorld, which is modeled after text adventure games like TextWorld (Côté et al. 2018), and is intended to abstract away the perception and control problems from the reasoning module. Training environments in ALFRED are converted into ALFWorld environments, and for each training goal, an oracle is constructed (which is a function that maps from any ALFWorld environment state to an optimal subsequent action that leads to the goal). This facilitates pre-training the second module using DAgger through interactions with ALFWorld.

----------------------------------

Reasons for score:

I vote for accepting the paper. The idea of explicitly training the model to reason about subgoals is intriguing. However, there are several assumptions and limitations of the proposed approach.

----------------------------------

Strengths:

- A model which decomposes high-level goals into low-level action sequences is very valuable and interesting.
- Comparison of single-goal vs. multiple-goal models.
- Good set of ablations, experiments, and comparisons, although I am not sure why Section 4.3 only looked at a single task.

----------------------------------

Weaknesses:

- Because ALFWorld is an abstraction of the problem, it must make assumptions about perception and/or physical control. The paper describes several simplifications it makes: e.g., as far as I can tell, it disregards physical attributes of objects, and only retains the existence of particular object types when BUTLER::Vision processes an observation. Also, ALFWorld does not implement physical/semantic constraints (e.g., what can fit in a microwave). The setup of BUTLER means that losing perceived attributes in the output of the first module may result in the reasoning module being unable to reason about these attributes, and it is up to the designers of ALFWorld to decide which attributes are important to retain.
- As above, ALFWorld is manually designed and requires making decisions about what aspects of the problem to model and which aspects should be abstracted away. This also includes requiring design of an oracle function, or alternatively an effective reward function (i.e., a denser reward function than binary task completion), which was not explored.

----------------------------------

Questions for the authors:

- Can you elaborate more on what it means for a sequence to be prototypical (in the abstract)?
- What is the difference between the high-level goals and low-level step-by-step language in ALFRED? Is this similar to the goal "put a pan on the dining table" vs. "go to the cabinet", "open the cabinet"?
- What exactly are the generalization dimensions for unseen rooms? Novel combinations of object instances that existed during training, novel instances (e.g., color) of object types that existed during training, or completely novel object types? Or something else?
- In the last paragraph of section 3.1, does this refer to playing a game at inference time using ALFRED, or during training with ALFWorld?
- Beam search is only done on the subgoal descriptions output by BUTLER::Brain, not the action actions taken in the ALFRED environment by the controller?
- If the Human Goals setting is using human-written (natural language) goal specifications, then what are the goals in the other evaluation settings? Auto-generated?

----------------------------------

Additional feedback:

- The color coding that begins on the third page is confusing, and the colors are very hard to tell apart (at least for me). 
- In the related work on embodied language learning, the last sentence is a bit confusing as ALFRED is a fully interactive modality, right? If not, what is considered fully-interactive?

----------------------------------

Rating after discussion: lowering to a 6, as I share concerns with R1 about experiments and generalizability of the proposed approach.
review_title: Recommendation to Accept

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work suggests an approach to improve the embodied agents interacting in an environment solving language-specified tasks. The approach introduces a parallel text-based (minus any physical interaction or visual input) environment which can be used to pretrain the agent by learning language-only policies. This text-based environment is an extension of the TextWorld framework.

Strengths:

* The paper is clearly written.
* The premise of the paper that such a parallel environment allows agents to explore/learn in an abstract language-only environment while also allowing transfer of the learned knowledge to physical embodied environments is well motivated.

Weaknesses:

* The main premise is that such a text-based environment can be used to learn abstract (high-level) policies that can then be transferred to an embodied agent to solve language-specified tasks in a physically simulated (with visual input) environment. However, the main experiment to prove this claim in Table 4 falls short of proving this. The results in that table show agents learned with an oracle state estimator which means there is no visual input processing during this mode. It can also be noted that the Controller is also a heuristic module with no learning. Which means under this setting, learning/evaluating in embodied environment is pure text-driven which is same as TextWorld, hence it is not surprising that TW-Only performs better.
* The performance of BUTLER-ORACLE in Table 3 is similar to the performance of BUTLER on TextWorld (on All Tasks) which further proves that using oracle state estimator is essentially reducing the embodied environment tasks to TextWorld task. 
* Furthermore, given that this is the main premise of the paper, Table 4 needs to be complete with all the tasks.
* It is unclear why the components in BUTLER Agent interact using discretized constructs (of natural language) instead of continuous vector spaces. 
* There seem to be several hacks adopted to obtain good results (such as using beam search during evaluation, using rule-based expert for supervision) which make this approach highly specific to the given dataset/environment limiting its general applicability.

Overall: The strong motivation of this work is not supported by empirical results. The results presented in the paper use a number of hacks, are incomplete and lack insights.

Update after author response:

The author response is much appreciated. However, my two main concerns remain unaddressed. The authors may add these additional experiments/results to Table 3 and 4 in further revisions for a stronger submission.
* Table 3 is the main result of the paper which claims policies learned in TextWorld (TW) environment can be transferred over to ALFRED (ALF) environment under zero-shot setting. This result alone has several weaknesses -- (1) Evaluation done on non-human goals in ALF seem to use same template as that used in TW, so it is not surprising that agents will have non-zero success rates on similar language specifications. (2) When evaluation is done on human goals (which seems to be the real test), the agent's performance is very low. Also, there are no baselines (e.g., random) provided to compare those scores against. (3) Why are the experiments only conducted in zero-shot setting? This actually brings me to the second weakness.

* Since the transfer learning is happening from a pure-text TW environment to a physically simulated (with visual input) ALF environment, it is more interesting/relevant to see how the language module pre-trained on text-only TW adapts to multimodal setup in ALF. This adaptation will require further training/fine-tuning on ALF so that visual/control modules can adapt to this pre-trained language module. This experiment was attempted in Table 4, however, as pointed in my initial review, this falls short of proving any claims made in the paper because the agents in Table 4 are learned with an oracle state estimator which means there is no visual input processing during this mode. It can also be noted that the Controller is also a heuristic module with no learning. Which means the setting used in Table 4 reduces learning/evaluating in embodied ALF environment to a pure text-driven environment.
review_title: Lacks clarity on the usefulness of the proposed ALFWorld environment

Meta-review: 
The paper proposes ALFWorld, which combined TextWorld and ALFRED to create aligned scenarios (one that is text-only, and the other in an embodied visual simulator) so that high-level policies in language can be learned in a simpler world, and then transferred to the embodied one (using the proposed BUTLER architecture).  The proposed BUTLER model consists of three components: 1) a perceptual module (converts environment observation to specification of objects and relations using text), 2) goal-planning module for generating textual specification of subgoals (from observed environment state) and 3) controller module which takes outputs from 1) and 2) and generates a sequence of actions.  Experiments show that using the textual specification, it is possible to models pretrained in the text world can generalize better to embodied settings.

Review Summary: The submission received slightly divergent reviews with R2, R3 recommending acceptance (score 7) and R1 recommending reject (score 4).  All reviewers recognized the novelty of the work, and the potential for follow-up work based on the submission.  After considering the author response and discussion between reviewers, both R2 and R3 agreed that there are indeed flaws with the work as pointed out by R1 (R3 lowered their rating to 6).  Despite the concerns, both R2,R3 remained on the positive side.

Pros: 
- The work and proposed framework can stimulate further research on transferring policies from simple text environments to more realistic visual environment. (R2)
- The decomposition of high-level goals into low-level actions sequences is a good direction for future research (R3)
- Good set of experiments and comparisons (R3)
- the paper is clearly written and easy to understand (R2)

Cons:
- The main claim of the work (high-level policies learned in a text-based environment can be transferred to a physically simulated environment) is not properly substantiated by the experimental results. (R1)
- The proposed method is a complex system and simpler baselines should be considered (R2)
- Some assumptions are made in ALFWorld need to be hand designed and may miss important aspects of perception (R3) 
- More experiments and ablations are needed to properly evaluate the framework

Despite the issues pointed out by R1, the AC believe that the work can inspire future work in this area, and thus recommend acceptance.  The paper is also well-written and easy to understand. 
Decision: Accept (Poster)