Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper performs benchmarks on partially observable RL environments and a bunch of network architectures on memory. The authors make sure the learning of the POPGym environments is affordable with consumer-grade GPU to ensure the experiments are feasible for researchers. They conclude that GRU is the best general-purpose memory model. 
strength_and_weaknesses: Strength
+ A very good and in-depth summary of the current RL benchmarks, especially on the partially observable environments. After analysis of the shortcomings of the current benchmarks, they motivate the work;
+ The author conducted an extensive set of experiments with 14 partially observable environments and with 13 memory model baselines;
+ Considerations have been made for wide adoption of the proposed benchmark, including low compute cost for a wide community, compatible APIs with the popular training library, etc;

Weaknesses
- I would expect the analysis part of a benchmark paper would take the largest portion of the paper and will have a very detailed analysis of the benchmarks and experiments. However, the major parts of the paper are about the task description and baseline description; 
- Though the authors have run very extensive experiments, the results do not seem dramatically different than what people already understood;
- I  found limited insights from the benchmark results;
- The authors should point out potential directions for tackling the partially observable tasks after the benchmarking. 
clarity,_quality,_novelty_and_reproducibility: The clarity is high, the quality is decent, novelty is on a borderline level. The reproducibility is good.
summary_of_the_review: I found unsatisfactory that a benchmark paper does not provide much content on the in-depth analysis of experiment results, even though the authors conduct an extensive set of experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduced a set of new environments to test RL algorithms on partially observable environments. The authors introduced a diverse set of environments in which different aspects of the hardness of partially observed system was tested. Further, the authors conduced quite extensive study and tested the performance of previously suggested algorithms for the partially observed domain. 

strength_and_weaknesses: Pros:
Partially observed system are a crucial challenge to tackle for applying RL in real-world applications. I tend to agree with the authors claim that currently there are not enough benchmarks to test different RL algorithms for the partially observable settings. I hope that this work will encourage further research into the partially observable setting, and lead to the design of new and practical algorithms for this domain. 

Cons:
Although the authors introduced several new environments there are several aspects of partially observed systems that are not being investigated. First, the problems (in my opinion) are quite basic and still seem quite far from practical applications. Second, the theoretical RL community recently studied several POMDP models that would be interesting to explore from practical perspective (in my opinion):
1. Weakly revealing POMDPs (see https://arxiv.org/pdf/2204.08967.pdf).
2. MDPs with latent context (see https://arxiv.org/abs/2102.04939).
3. \gamma observable POMDP (see https://arxiv.org/abs/2206.03446)


clarity,_quality,_novelty_and_reproducibility: The writing is quite clear and in good quality in my opinion. Below are some minor comments.

1. This it would be worthwhile to add that current benchmarks test the high dimensional observations.
2. Figure 3. Can you normalize the reward such that 1 would indicate the optimal value? if I understand correctly it seems that the normalisation might make it seems the performance is high although it is extremely sub optimal (this can happen due to the normalization).
3. End of section 1.1. should put period "." at the end of each line.
4. page 3, "wall following behavior", what does that mean?
5. page 5 "Like Repeat Fist" -> "Like Repeat First"
6. Page 5. Reference to additional bandit book: https://tor-lattimore.com/downloads/book/book.pdf.
summary_of_the_review: Overall, I think this paper has a clear and nice contribution: it introduced new benchmarks for study partially observed RL problems, This, in my opinion, is very much worthwhile and will hopefully encourage the community to study these type of problems in more depth. Further, I hope that the authors/additional contributers will expand the set of task in the future.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The submission introduces a library containing 14 POMDPs, each with three levels of difficulty, and 13 memory model baselines. The submission benchmarks 12 of these memory baselines.
strength_and_weaknesses: ### Strengths

- I agree with the submission's characterization that there is a lack of suitable partially observable testbeds.
- Because the submission compares existing algorithms, rather than introducing new ones, the experiments are less likely to suffer from experimenter bias.
- From a brief perusal, the code appears easy to read and well organized.

### Weaknesses

- While (again) I agree that there is a dearth of good POMDP benchmarks, I am not sure the submission does a great job addressing the problem:
    - Six of the 14 games are small modifications of fully observable settings
    - Four are described as "diagnostic", meaning useful as "memory introspection and debugging tools"
    - Two are relatively simple card games,
    - Two are navigation based, which, as the submission points out, are one of the few types of POMDPs for which there are suitable existing benchmarks.

    What I am getting at is that the partial observability in POPGym largely doesn't feel *strategically interesting*.
- The other gripe I have is with algorithm benchmarking:
    - First, given that the submission is a benchmarking paper, it seems disappointing that it implements but does not benchmark differentiable neural computers. The submission states that this is because computation cost, despite also indicating that POPGym environments induce relatively quick convergence.
    - Second, the metrics that were reported are not up to date with modern deep RL benchmarking -- see *Deep RL at the Edge of the Statistical Precipice* (NeurIPS 2021). 
        - The most obvious criticism here is that each algorithm is over only three seeds. It is well documented that three seeds may be insufficient to accurately reflect performance. Given that it is a benchmark paper, I'd hope for at least 5 or 10. 
        - Additionally, it would be good to include the metrics endorsed by the statistical precipice paper, such as interquartile mean, optimality gap, performance profiles, and stratified bootstrap confidence intervals.
clarity,_quality,_novelty_and_reproducibility: The clarity of the work is good.

The quality is fair; however, as articulated in the weaknesses section, there is substantial room for improvement.

The originality is good in the respect that there has not been much work organizing POMDP benchmarks for deep RL.

The reproducibility is high in the sense that code is available and appears to be well organized but less high in the sense that outlier-sensitive results were reported over a small number of runs.
summary_of_the_review: While the idea motivating the submission (providing good POMDP benchmarks for deep RL) is one that is very valuable to the community, I think the submission falls short of its potential both in terms of the proposed environments and the benchmarking of existing algorithms.

Still, I think the submission ought to be accepted, as the value it provides to the community is strictly positive. 

I hope the authors will consider addressing the weaknesses listed above in future revisions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper analyzes multiple POMDP algorithms on numerous environments with different aspects. While their experimental and discussion sections could benefit from more extensive comparisons and deeper analysis, the paper has tested and compared various algorithms on different environments. This benchmark of POMDPs is actually quite important and timely -- many real worlds problems almost always require working with partial observation, thus understanding the current states of algorithms adds an important value to the community. I however personally found their results not super surprising, but it is also good to share with the community that many results follow our intuition or word-of-mouth. So, I give a weak accept. Should the paper be rejected, I highly recommend performing more analysis/ablation studies.
justification_for_why_not_higher_score: The paper is clearly missing deep analysis on understanding each given result.
justification_for_why_not_lower_score: With the same reason above, I would be OK if the paper is rejected -- and give one more iteration to deeply analyze and polish.
Decision: Accept: poster