Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Summary:

The paper proposes a multi-stage directed exploration algorithm, XTX. It first imitates previous high score trajectories and then switches to an exploration policy with novelty bonuses.
Conceptually, XTX is a method that extends Go-Explore which only acts randomly after reaching the frontier of familiar states.
The paper argues that with novelty bonuses, the agent will be encouraged to explore more promising actions. This can especially be helpful when the action space is large like text-based games.
Empirically, XTX shows strong performance on a large set of text-based games.
main_review: Pros:

The paper is generally well-written and easy to follow. 
The novelty of XTX is clearly elaborated.
The method surpasses the existing method with a large margin on text-based games. The ablation studies show the individual components introduced by XTX can bring improvements.

Cons:

One weakness of the paper is these experiments did not clarify why the novel part of XTX (i.e. exploration with novelty bonus on the frontier) is helpful over random actions. The paper hypothesizes that novelty bonuses can encourage the agent to select promising actions in large action spaces. However, the ablation study (Figure 2) casts doubts on this hypothesis. XTX brings significant improvements over Go-explore in Zork1 but not other games. The difference doesn't seem to be correlated to the size of action spaces.

Questions:

I don't fully get why the method is motivated to solve the problems with large action space. How can an agent receive a novelty bonus if it did not enter that novel state by trying random actions? Do the authors assume the generalization of the neural network plays a key role here?

Other Suggestions:

The author might want to try other hard-exploration tasks. For example, minigrid or maze can be tested, if not Atari games like Montezuma Revenge. Since these are environments where existing exploration methods are developed, we can have a better understanding of how exactly XTX compares to other exploration algorithms, rather than the existing text-base game agent without directed exploration.


summary_of_the_review: Reason for the Score:

The write-up and experiments in this paper are of good quality. The method itself is novel and the empirical finding in this paper might be particularly interesting for the audience of text-based RL. I have minor concerns author's claim about why this method works better than existing exploration algorithms while I'm happy to increase the score if they are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new exploration algorithm, eXploit-Then-eXplore (XTX), for text-based games which require extensive exploration. The authors propose an algorithm that explicitly disentangles exploitation and exploration strategies within each episode. XTX first learns the exploitation policy that imitates the promising trajectories from past experiences, then uses exploration policy to discover the novel state-action spaces. Finally, the authors demonstrated the outperforming results in the Jericho environment. 
main_review: This paper is well motivated and most parts are well written, but the main method section is written to be difficult to follow. The results demonstrate empirical gains in the Jericho environment. However, the baselines consist only of simple algorithms without an exploration strategy. The detailed comments and questions are as follows:

1. In the experiment, the performance is compared with DRRN and MPRC-DQN, which lack exploration strategy. XTX seems to be an exploration method very similar to Go-Explore. Moreover, in the paper, Go-Explore and PC-PG are mentioned as the most closely related approaches, but they are excluded from the baseline algorithms. It would be better to demonstrate the results of them together.

2. (Page 5, section 3.1.2, sampling trajectories) It is hard to follow the explanation. Can it be understood as a kind of weighted behavior cloning? Moreover, I understand the motivation of biased sampling towards high scores, but don’t understand the motivation for the length. I think that a shorter trajectory length is not necessarily better. Can you give an intuitive explanation?

3. In the paper, policy $\pi_\text{il}$ is modeled as GPT-2, and policy $\pi_\text{inv-dy}$ is modeled as DRRN. Is there any reason why each policy is modeled differently? Especially, the policy $\pi_\text{il}$ is renormalized over the valid action set, is there any reason or advantage to learn the policy with GPT-2?

4. In the experiments, the results demonstrate XTX underperforms DRRN on ENCHANTER. Is there any intuitive explanation for this result? It would be better if a discussion about what characteristics in the ENCHANTER made the XTX not work would be added.


summary_of_the_review: This paper is well-motivated, written overall, and demonstrates state-of-the-art performance in the Jericho environment. However, there are relevant but missing baseline algorithms (Go-Explore, PC-PG) for the main table of experiments. I think the results of these algorithms should also be included in the main table, and I think this can further support the main arguments of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces an agent with a built-in exploration strategy that is aimed at text adventure games, or more generally, environments with large action spaces and sparse rewards. The exploration strategy is constructed from two independent policies: one trained with self-imitation learning on successful trajectories, and one trained on an inverse dynamics intrinsic reward. The agent plays episodes by starting with the exploitation policy for a number of steps that depends on the experience collected up to that point, and then switching to the exploration policy. The paper is well-written, describes the contributions clearly, and places itself in the context of the existing literature on exploration. It includes results on a number of text exploration games from a recent benchmark, where it shows by and large a significant improvement relative to the baselines included.
main_review: The main contribution is an exploration strategy with an in-episode switch from an exploitation policy to one aimed at exploration. This approach to combining exploration and exploitation is different from much of the existing literature, where typically a single policy is used throughout the episode, and often throughout training, that merges two reward signals. Since the switching policy in this paper is the element that looks most hardcoded, and therefore potentially brittle, it would be valuable to investigate a bit more whether a more flexible solution is also possible here. While different, Agent57 (whose predecessor NGU is cited) might offer inspiration here: it also uses multiple policies, and manages the switching with a learned (bandit) mechanism. A significant difference is that there the switching only happens between episodes, but a similar switching mechanism might be considered here within episodes nonetheless.

The in-episode switch is there to ensure that exploration happens at the edge of the known region of state space, where it is needed and meaningful. That is a very sensible thing for the agent to do, but there are other exploration strategies that effectively also do that, such as random network distillation (Burda et al., 2018) and inverse dynamics (Pathak et al., 2017), which the authors use to train their exploration policy. While the exploration region is less explicitly located at the edge of the known state space region in those algorithms than in this paper, the prediction errors that they rely on for intrinsic reward generation are more likely to occur at that edge. One question I have for the authors here is whether the inverse dynamics reward signal itself can be used to indicate when to switch from explore to exploit. In that case, the two-policy solution can be simplified again to a single policy that merges the two behaviours. I did not see this ablation in the paper, but I believe it would be a good thing to include.

Conversely, it would be valuable to see the performance of the strategy proposed here on other exploration benchmarks, such as the hard exploration games from the Atari suite (Bellemare et al., 2016). While I appreciate that text adventure games are in some ways different from their video counterparts, since they have a different observation space (language, not pixels) and action set (again language, not moves), they are still both RL environments, and general agents should be able to play both. Furthermore a game like Montezuma’s Revenge has a bottleneck aspect similar to the one that many text based games have, as well as the need for exploration on the frontier of the known region of state space. All in all it seems that the proposed strategy here could work on a wider range of environments than addressed in the paper. If that is not the case, it is still a valuable contribution, but if it is, it would be good to know.

A last comment: the agent proposed in the paper has another unusual feature in that its exploitation policy is trained only by self-imitation. While it is important to find the edge of the explored region of state space, and the self-imitation training regime can help with this, the XTX strategy can also be implemented with an exploitation policy that is trained in a more traditional way, with one of the many RL approaches available. Can the authors comment on why they chose the self-imitation approach instead?

summary_of_the_review: The paper is well written, presents a marked improvement over the baselines provided (I’m not sufficiently familiar with the text adventure game literature to be certain those represent state of the art, but I will assume they do unless corrected), and provides an interesting approach to the exploration problem through the two-policy architecture. I recommend acceptance, but I also feel the paper could be strengthened by addressing the questions raised in the main review section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose eXploit-Then-eXplore (XTX), a training strategy for agent solving human-generated text-based games. 

XTX consists of two training phases:

1. In the exploitation phase, the agent samples high quality experiences (in terms of score and trajectory length) from its replay buffer. Using the sampled trajectories, an action generation module is trained. At a certain game step $t$, the action generation module takes the observation $o_t$, as well as the two most recent past actions $a_{t-1}$ and $a_{t-2}$ as input, and generates the new action $a_t$ in a word-by-word auto-regressive manner. This process is referred as self-imitation by the authors. 

2. In the exploration phase, in addition to the Q-learning loss as used in DRRN, the authors use two auxiliary losses to encourage the model to capture useful representations. First, the inverse dynamics loss $L_{inv}$ optimizes a module that predicts an action $a_t$ given two consecutive observations $o_t$ and $o_{t+1}$, where $o_{t+1}$ is resulted by $a_t$ given $o_t$. The second loss $L_{dec}$ is a regularizer that optimizes a module reconstructs an action $a_t$ from its encoding $f_a(a_t)$.

During training, the two phases take control in an (almost) alternate manner, however, there is a coefficient $\lambda$ controls the interpolation between the phases. The authors show that it is beneficial to not having the exploitation take control solely. 

On a subset of games from the Jericho suite, the authors show their agent outperform prior works. 

main_review: **Strengths**

1. The disentanglement of exploration and exploitation makes sense. The phase-alternating pipeline is nicely designed. 
2. The paper is clearly written, it is relatively easy to understand how the model look like (although the intuition of each component isn't too clear). 
3. The set of ablation experiments in Section 4.2 are well designed. 

**Questions and concerns**

1. What's the reason of choosing this subset of 12 games? While the list seems to cover a wide range of game difficulties, but why not using the entire Jericho suite?
2. The authors cited the INV-DY agent (Yao et al., 2021) in their Section 3.1.3, and actually, if I understand correctly, the entire Section 3.1.3 is describing Yao et al.'s model, without any new contribution. Why do not the authors compare their agent with INV-DY in result tables?
3. In Section 3.1.PHASE 1, the authors describe two criteria that switch the agent to exploration phase. Can the authors elaborate on the second criterion, what does it mean if the number of steps in an episode equal to the longest of the $k$ sampled trajectories? If an agent moves back and forth between two locations, which may result a super long steps, but this behavior is not necessarily desired. 
4. In Section 3.1.2, Sampling trajectories, the authors describe the way they use to sample trajectories. However, to my understanding, the loss shown in Eqn. 5 is a game-step-wise loss. Does the authors also sample game steps from the sampled trajectories (if so, how?), or they compute this loss on all game steps within the sampled trajectories?
5. In the paragraph under Eqn. 2, the authors mentioned that "Note that the action distribution over actions $a$ induced by $\pi_{inv-dy}$ is conditioned only on the current observation $o$". However, according to Eqn. 6, it is also conditioned on $o'$, which is the next observation, i.e., $o_{t+1}$. 



**References**
1. Keep CALM and explore: Language models for action generation in text-based games. Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. EMNLP 2020. 
2. Reading and acting while blindfolded: The need for semantics in text game agents. Shunyu Yao, Karthik Narasimhan, and Matthew Hausknecht. NAACL 2021.  


---------------------------- Nov 29, 2021
We had a good discussion among reviewers, let me give the authors some update.

**1.  Increased my score to 6.** This is because the authors have somewhat addressed my comments, I'm relatively satisfied. There are a few concerns remaining, as listed below:

a) on modelling novelty: the novel components are only a) the sampling strategy in exploitation phase and b) the two-phase pipeline. It was a bit weird to (almost) "copy and paste" a subsection from a prior work into the main body of this submission, which may confuse readers by giving a false message about the contribution. However, if other co-reviewers are fine with it, I'm fine too. 

b) After a few paper updates, the main results (in Table 1) is only marginally higher than prior work. The authors can add more discussion addressing this in their cameraready.
 
**2.  We recommend the authors to remove the Dragon row from the result tables** (or rerun when the Jericho team fixes the bug):

As Reviewer PsKh find out, the proposed agent's scores exceed the max score on that game. 

I happen to know some core Jericho contributors, and we tested the [Dragon game](https://ifdb.org/viewgame?id=sjiyffz8n5patu8l). 

Usually, when reaching the goal, this will pop up:

```
Dragon's Treasure Store

The Dragon's secret hoard is open before you. By the flickering light of your little candle, you can make out a heaps of treasure stacked untidily around the floor.
You can see piles of gold and heaps of jewels, many rising higher than the top of your head. The Dragon has told you it has no use for the treasure and it is now yours.

You are rich beyond your wildest dreams!

*** You have won ***

In that game you scored 25 out of a possible 25, in 101 turns.

Would you like to RESTART, RESTORE a saved game, UNDO your last move, give the FULL score for that game or QUIT
```

In that game, the scoring function works like this:
```
1 for buying the box
1 for finding the screwdriver
2 for finding the candle
1 for finding the matches
1 for for opening the castle door
1 for building the hand-glider in the right place
2 for getting the sword/booklet
1 for escaping from the tower using the hang-glider
2 for killing the Troll to get the horn
5 for talking to the Troll to get the horn
2 for killing the dragon
5 for charming the dragon instead of killing him
5 for finding the treasure

= 25 points maximum total (e.g., multiple ways to get the horn)

(minus 2 points for each RESCUE or 'dead' recovery)
```
Given this -2 points for each RESCUE action, an agent can get negative total points. Because the original game did some *short* to *unsigned char* converting, this caused underflow (-128 vs. 128). 

This may because the author of the Dragon game (in 2003) didn't expect machines to play his game, because most humans will give up playing before reaching this underflow point :)

So the weird numbers are not the authors' problem. As I mentioned, they can either remove that row, or to rerun whenever Jericho fixes that. 

summary_of_the_review: While I like this paper in general, my main concern is its novelty and contribution. As mentioned in my questions and concerns (Q2) above, the entire Section 3.1.3 is describing prior work (Yao et al., 2021), the "Learning from trajectories" part of Section 3.1.2 is describing another prior work (Yao et al., 2020). Actually, neither (Yao et al., 2021) nor (Yao et al., 2020) is compared in result table. As a consequence, to my understanding, the contribution of this paper is the two-phase pipeline and the sampling strategies in Section 3.1.2. I am not sure if this paper contains enough contributions to publish at ICLR. 

Please correct me if I understood wrong.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. All reviewers are unanimously leaning towards acceptance of this paper. Reviewers in particular liked that the paper is well-written and easy to follow [186e,TAdH,Exgo], well motivated [TAdH], interesting [PsKh], novel [186e] and provides gains over baselines [186e,TAdH,PsKh] with interesting ablations [186e,Exgo]. I thus recommend accepting the paper and  I encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Accept (Spotlight)