Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new RL agent architecture called Hierarchical Transformer Memory (HTM), which adds a history-chunking mechanism to the base GTrXL/TrXL model. HTM is found to perform well on six memory tasks.
main_review: - Originality

The proposed architecture is novel. But the acronym HTM is too easily confused with the HTM network (Hierarchical Temporal Memory) introduced by Jeff Hawkins in 2004, and still in development today.


- Quality

The paper’s central claim is that HTM’s history-chunking mechanism improves its ability to retrieve information as needed. The most relevant experimental baseline is TrXL (essentially HTM without the chunking). HTM outperforms TrXL on four of six RL domains, but no TrXL results are reported for the other two tasks (Passive Visual Match and Paired Associative Inference). Nevertheless, HTM performs on par with SOTA models on those two tasks.

The case for HTM is undermined by the scarcity of hyperparameter tuning as reported. Reliable experimental practice generally requires that hyperparameters for baseline models be tuned with as much care as those of the new model. Table 1 (in Appendix A) lists the settings of 28 hyperparameters (HPs) for each of the 6 experiments. Four of those HPs pertain only to HTM, one pertains only to TrXL, and the other 23 apply to both models. The paper discusses the tuning of only 6 HPs (in Table 2 and Appendix D.3). How were the values of the other 22 HPs obtained? Most of them vary by task, and so were presumably found through limited testing of HTM on short runs instead of full sweeps. In no case are separate values reported, one for HTM and one for TrXL. Since TrXL is essentially an ablation of HTM, one would be tempted to tune many or most of the shared HPs on HTM, then apply the best HP configuration to TrXL as well. But this would inadvertently introduce a strong bias in favor of HTM over TrXL. Attention is the central transformer operation, and history chunking alters it in a major way, so there is little reason to believe that a given HP setting will prove equally good for HTM and TrXL.

All of the reported experiments were run for billions of training steps, demonstrating that sufficient compute was available to tune HPs for HTM and TrXL independently, if only on shorter runs. In the absence of separate tuning, it’s hard to assess the benefit of history chunking.


- Clarity

In general, the paper is very clear and well-written. Here are a few points for improvement:

Contrary to the answer on checklist item 3.b, section A.1 does not explain how most hyperparameters were chosen.

In Figures 3.a and 3.b, why does the HTM curve start at 100%?

In Figure 5, how could the LSTM do worse than random chance?


- Significance

As noted in “Limitations & future directions”, HTM requires the permanent storage of data from each step. The severity of this limitation strikes me as fundamental, and I don’t expect the potential workarounds to make it a practical approach for most applications of interest in the future. But this is a personal view, and I could be wrong.


- Conclusion

The open questions regarding the choices of hyperparameters lead me to doubt the attribution of HTM’s performance gains to its chunking mechanism. If I have overlooked or misunderstood something, I look forward to being corrected.

**POST-RESPONSE UPDATE**

I commend the authors for their helpful responses. They have clarified several issues related to hyperparameters, and have performed supplementary hyperparameter sweeps that improve the work.

Regarding the prior use of the term HTM, I find no problem with any of the suggested alternatives. HCAM seems the clearest to my mind.

I recommend replacing “chance level performance” with a clearer term, as it can too easily be interpreted to mean the performance of a trivial agent that simply takes random actions. 

I have raised my score by one point.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors proposed a memory network that breaks events into chunks, and retrieve memory hierarchically. They tested their model on a wide range of qualitatively different tasks and showed (sometime dramatic) improvement of performance against many baseline models
main_review: Originality
At the high-level, the idea of storing memories in a tree structure may not be mind-blowing. It’s still important to show that it can work in practice. And the range of tasks tested is rather impressive.

Quality
The quality of the work is high. The network architecture is intuitive and well-motivated. The empirical results are thorough and convincing. 

Clarity
The paper is clearly written.

Significance
This work provides a solid and significant improvement in the use of memory network for RL agents.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an architecture to augment an RL agent’s memory using a novel hierarchical transformer. The main novelties of the approach are 1) that the transformer attends over chunks and 2) that it’s used for long horizon RL. They test their approach in many interesting environments that each test a different component of memory (delayed recall, object permanence, distractors…). The results show that across the different environments the proposed approach outperforms the tested baselines. Moreover, the approach competes well with specially tuned state-of-the-art methods in other environments.
main_review: 
1 Strengths:

The main strength of the paper is the sheer number of different environments that were used to benchmark the method. It really is impressive and shows that the method can potentially handle different scenarios. Taking it a step further and comparing the architecture to state-of-the-art results in other domains is also a big plus.

Another strength is that the approach is fairly intuitive - attending over chunks instead of over individual states makes intuitive sense. 

Many ablations are provided in the appendix.

2 Weaknesses:

Tuning the chunk size for each environment feels a little hacky. As mentioned by the authors – learning the chunk size would be an interesting avenue of future work.

3 Correctness:

From scanning the appendix – Figure 10 shows that the self-supervised loss seems to be crucial to obtain decent performance in the Ballet and Rapid word learning tasks. The model performs close to chance without it. I don’t think this is emphasized nearly enough in the main text. The method doesn’t work without it…

I find it hard to believe that the transformer XL performs so poorly on some of the simpler tasks. I can’t help but wonder if the baseline was properly tuned for these tasks. I scanned the appendix, and it seems like the same sweep over hyper parameters was performed for the main method as well as the baselines but I wonder if something as simple as tuning the learning rate for the baseline (which to my understanding was not done) could have improved its performance.

4 clarity:

For the most part the paper is well written and easy to follow. I think the paper could have benefited from a background section where Transformers and RL could have been introduced. As is, without any background, section 2 feels very dense. In particular, lines 95-109 are extremely dense and should be handled with more care. 

I also think that Figure 11 from the appendix should somehow make its way into figure 5 of the main paper. It really feels strange to just show that the baselines are failing when in reality they do learn properly on the training task.

5 Relation to prior work:

The relation to prior work is handled nicely in the discussion section.

UPDATE: Thanks for the replies to my questions. I leave my score as it stands. (6)
limitations_and_societal_impact: No code and minimal plans to release – this will make it extremely hard for the community to iterate on the approach.
review_title: null

Meta-review: - The proposed method is tackling an important problem, reasonable, and the experiments are well designed and executed.
- The major concerns from the reviewers (e.g., hyper parameter tuning, chunk size, TxXL results, etc.) are addressed well enough by the rebuttal. 
- For reproducibility and accessibility, I suggest to open source the code and make the environment available at your earliest convenience. 
- I suggest accepting the paper. 
Decision: Accept (Poster)