Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper revisits recurrent neural network based model-free RL methods with carefully tuning on various aspects of the learning details. Results on a large number of tasks demonstrate that with a well-tuned RNN implementation of RL methods, it is enough to achieve sufficient performance compared with many state-of-the-art competitors on various types of tasks. 

main_review: Although this paper does not propose any new algorithms or theories, I think these implementation details with RNN in RL are very practically useful experiences for solving RL problems. These introduced tricks, designments, and techniques on the architecture, input space, RL algorithms, context length, etc., are all valuable conclusions for the readers.

Moreover, the perspective viewing meta RL, robust RL, and generalization or transferability in RL as POMDP problem is also very interesting and reasonable. From this view, I can connect many of these different tasks from a principled way.

In the experiments, could the authors provide details on the network structures used for different tasks? Since efficiency of the implementations are discussed and compared with prior works. Also, instead of comparing with some SOTA methods in specific tasks, would the simplest baseline, i.e., the plain network by just removing the RNN layer, be compared as a reference in the curves? This would provide a most straightforward view that how much benefit RNN brings.

For all these tasks, will stacking RNN layers be better than a single layer?

Another question is that as demonstrated in the experiments, a carefully designed RNN implementation can always result in much better performance to solve POMDP problem, does there indeed exist theoretical explanations connecting the specific architecture of RNN and the analysis based on belief in previous POMDP theories?

Overall, although this paper focuses on RNN implementations, instead of proposing new algorithms and methods, I think the contexts discussed in the paper can contribute to the practical RL domain.

summary_of_the_review: The paper provides many useful implementation details for RNN based RL methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper primarily provides an empirical analysis of recurrent model-free RL on several classes of POMDP, showing that if parameters are well-chosen, the basic approach of just applying a recurrent layer is not only competitive with but often outperforms specially-designed methods for those problem classes. The authors consider different architectures, algorithms, types of RNN, and other parameters and fill in a significant gap left by previous papers, which didn't consider most of the important details of getting model-free RNN methods to work well
main_review: Strengths:
  - A few grammatical mistakes aside, the paper is well-organized and understandable. It sets out a clear picture of the current POMDP landscape, making clear the distinctions between different classes of problem (such as stationary vs nonstationary, across episode, policy inputs) and the concerns that each raise. It describes in detail the process by which the experiments were chosen and performed.
  - It's surprising to me how novel the contribution is. This is not because any new methods are proposed, but rather because so little previous work has really considered the question of actually getting recurrent algorithms to work well. I think an analysis like this is long overdue, even if I may have some concerns over the experiments themselves. 
  - A good amount of effort clearly went into reproducibility and efficient, readable code

Weaknesses:
  - I'm not as convinced as I'd like to be by the experiments, which are ultimately what this empirical study rest on. I'll detail a few issues:
    1) In each class of problem, the tuned RNN is only compared against one other method. In general, these appear to be recent, SOTA, and well-chosen, but between that and the use of bar graphs instead of learning curves, it's hard to really sanity check. How would something like PFRNNs or DRBPNs compare?
    2) As a follow-on, I'm not sure that these are really fair comparisons. For the comparisons to other methods, the authors use an off-policy algorithm (SAC) with access to information that is not necessarily common in POMDP literature (rewards and done signals aren't often counted in the stored history). VRM follows a relatively similar training procedure, as does the modified VariBAD, but the other two algorithms are quite different. Can these methods really be compared in terms of simulation steps?
    3) While the set of all possible RNN architecture choices is enormous, I do think that there were a few more that could have been considered. Most importantly, I'm not fully convinced by the shared vs separate argument. The hypothesis of exploding gradient should be relatively simple to empirically log and demonstrate, and it also heavily depends on the shape of the environment rewards. Most of these environments have large, dense reward signals which need normalization. Another consideration would be whether the RNN should be before or after feature extraction. Finally (though this is more something that I've come across in my own work), layer-normalized RNNs often vastly outperform their standard counterparts.
    4) I preferred the learning curves found in the appendix to the bar chart comparisons in the main body. I think it would be more informative to compare the methods with some error bars and a better idea of the process. Also, it seemed like some environments (like Cheetah-Vel) didn't learn? A couple baselines (optimal upper-bound, random policy lower-bound) would give a better idea of how well the methods are actually doing.
summary_of_the_review: I'm recommending this paper with some reservations. Overall, I really think work like this is valuable and illustrates a large gap in knowledge that was previously unaddressed. I've always found it frustrating that the approaches to partial observability are so ad-hoc. I also appreciate the clarity with which this paper addresses the classes of POMDP and the considerations required for each. That said, I have some concerns with the thoroughness of the experiments and would like to see more baselines and fairer comparisons.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is concerned with understanding when RNNs are useful in POMDPs. They should that, for a class of POMDPS (e.g., meta RL, robust RL), standard RNNs can be competitive with solutions that are tailored to the given POMDP structure. The authors discuss four design considerations that they claim are essential for performance: (1) decoupling actor and critic networks, (2) using off-policy instead of on-policy algorithms, (3) context length of RNN, and (4) using rewards as historical input. They should on 4 mujoco baselines that their results are competitive with sota methods whenever these design choices are taken into account.
main_review: This work is quite straightforward and simple - which is good. The authors state that an end-to-end approach can be competitive with sota methods on tailored problems, if only specific considerations are taken into account.

The main drawback of this work is that the results are inconclusive.
1. There is no theoretical reasoning behind the authors claims. No motivating principles. It is hard for me to assess the correctness of their claims solely from their experiments on 4 mujoco tasks.
2. While the authors did run extensive experiments on the 4 mujoco tasks, these are definitely not enough. For the claims of this paper, I would at least want to see extensive experiments on another large domain (e.g., 4 other environments in atari, and perhaps at least 1 hard task in another domain).
3. When do things fail? Results that only show successes usually feel cherry picked. I assume the authors can find environments and/or algorithms for which their design choices fail. It is important to show these as well to understand the tradeoffs better.
summary_of_the_review: Good direction, results are inconclusive.
review_title: null

Meta-review: This paper recognizes that several common sub-problems studied in RL, such as meta RL and generalization in RL, can be cast as POMDPs. Using this observation, the authors evaluate how a straightforward approach to deal with POMDPs---using a recurrent neural network---compares to more specialized approaches. The reviewers agree that the research question studied in this paper is very interesting. However, after careful deliberation, I share the view of reviewer 2WFY that the results insufficiently support the claims made in the paper. In particular, I view the main claim from the abstract "We find that a careful architecture and hyperparameter decisions yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques in their respective domains."  as insufficiently supported. The main issue with the experiments is that only a small number of simple domains are considered. As Luisa points out in the public comments, variBAD dominates recurrent baselines when more complex tasks are considered, while on simpler domains such as the Cheetah-Vel domain considered in this paper, it performs similar to a recurrent model-free baseline. In the rebuttal the authors have added a more complex domain to address this, showing that a recurrent model-free baseline outperforms an off-policy version of variBAD. However, I view these results as inconclusive, as only a single complex domain is considered and they appear to contradict previous results with on-policy variBAD. For these reasons, I don't think the work in its current form is ready for publication at ICLR. But I want to encourage the authors to work out this direction further. In particular, adding more complex domains and also considering the on-policy variBAD method, can make this work stronger.
Decision: Reject