Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an empirical evaluation method to measure of the generalization capacity of an RL agent. It relies on CDPs combining a tabular environment with a supervised learning dataset. Generalization is measured across three axis: state space, observation space and action space. The empirical evaluation is led on DQN and QR-DQN on the four room domain and corridor domain combined with the MNIST dataset in the online and offline settings. The authors find that dropout improves action generalization but not observation generalization while regularisation improves observation generalization. They also find that QR-DQN generalise better than DQN in the offline setting on the observation axis and action axis but not on the state space axis.
main_review: I think the ideas in this paper are presented clearly and it is globally well-written. I like the distinction the authors made to distinguish generalization across their three axis is interesting and although they have been looked at separately in the literature, their juxtaposition and evaluation procedure in these different regimes are as far as I know novel. It also enables to measure generalisation in a single task setting and not across MDPs unlike some other works.

Originally, in statistical learning theory [Vapnik, 1995], the training and test distributions are the same in the definition of the generalisation gap (train error - test error) and the distribution only differs on the test in transfer learning so it would be good if you added a reference for your definition or precise why you are using a different definition. 

Do you have some intuitions about why DQN and QR-DQN generalise similarly in the online setting on the observation space? or an hypothesis why DQN generalise better than qr-dqn on the space axis?

I think the empirical measure of an agent generalization capacity would be more convincing and realistic on more complex environment like some from the Atari game for instance. An empirical evaluation on more environment would also strengthen the paper. It would be interesting to also have results for other agents.

“While generalization of state-value is similar to regression, generalization with quantities related to action, such as policy or action-value, do not have supervised learning analogues and hence require separate consideration. “ could you add more justification for this please?

Here are a few minor points that did not affect my rating. Please use the proceedings links in your references instead of the arXiv links.
summary_of_the_review: This work provides interesting insights comparing the generalization capacities of DQN and QR-DQN and their evaluation method corroborates some findings in the literature (e.g. about regularization) and some somewhat surprising results on the state generalization capacities of DQN vs QR-DQN. I think the paper would be stronger by providing results for other agents and more complex environments which is why I recommend a weak reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach to measure to quantify generalization properties (state generalization, observation generalization, action generalization) in single-task RL in the context of offline RL. The paper discusses the limitations of several existing approaches for measuring cross-environment generalization, then presents their generic measure of generalization (Equation 4), and evaluates generalization when learning from offline data using a DQN and QR-DQN in a contextual decision process (CDP) problem created out of MNIST classification. The results suggest that dropout is effective for action generalization, not state generalization, and L2 penalty is effective, and QR-DQN can generalize better than DQN in the offline setting, but worse in terms state generalization. 
main_review: I like the theme of the paper, and I indeed agree that designing the right metrics for quantifying generalization in single-task RL are needed and are not quite present in the literature. However, I think the paper falls short of delivering on and convincing the reader that the empirical analysis is useful and generalizable, and the proposed metric is a good one. I will explain why:

1. It is unclear how these observations transfer to other RL problems. Would testing generalization (within a single task) on other domains, e.g., Atari get the same results? Why or why not? For example, if the claim is that dropout doesn't help, does it not help on this task, or does the analysis also support why it shouldn't help on other domains?

2. One good point about converting SL problems to RL problems is that it is easy to measure generalization against a groundtruth SL oracle. With that in mind, the performance of RL agents is worse, so one argument is that everything is generalizing poorly on an absolute scale. Comparing with such metrics should have been more informative, compared to just DQN and QR-DQN, and this is a limitation.

3. In the offline setting, it is known that better algorithms that are more uncertainty-aware and do not commit to a single target value, such as ensemble DQN, random ensemble mixture, QR-DQN, etc work much better compared to the DQN, because they can be more accurate on OOD predictions. Does the action generalization experiment say essentially this? What is the new insight behind this high-level known result?

4. What insights should I take away as to why one algorithm is worse than the other in each of the different settings? Does it indicate that some principle is behind the relative performance differences? A detailed empirical analysis of what about QR-DQN and what about DQN makes them perform differently in these different generalization challenges would shed light on this. Similarly, a rigorous empirical study of why dropout helps or why L2 helps, etc, seems necessary to me to take away from this paper.

5. How does the accuracy translate to metrics I can measure during training, such as some kind of loss, validation error, etc? This is essential in understanding how one should modify algorithms to make them generalize better. 


summary_of_the_review: Without presenting any analysis of the **why** question, it seems that the work is not complete in my opinion. Also, the generalizability of these findings to other domains is under question. If these two major concerns can be addressed, especially along the lines of the pointers above, I am happy to revise my score. But for now, I would vote for rejection of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses generalization in deep RL. The key contribution of the paper, from my understanding, is that the authors argue that different from generalization in SL, in RL state, observation and action should be considered separately. A measurement (Eq.4) is proposed to evaluate generalization capacity of deep RL within the contectual decision process (CDP) scheme. Experiments were performed on grid world environments with MNIST image as observations and several results were concluded. 

main_review: Strength:

- The paper discussed an important problem in RL. The writting is clear and easy to follow. 

- The exprimental results are abundunt and there are enough repeats (30) to ensure statistical significance. The plots are well presented.

Major Concerns:

- The largest concern is that the experiments are limited to simple grid world envrionments with an unnatural setting to use MNIST images as observation. In particular, the authors wrote in Chap 2.2 "If the total number of classes K is equal to the number of states |S| in the MDP, then each state can be uniquely identified with a class label". This assumption restricts the scope of this study to discrete and relatively small state space. Since the results are mostly empirical, it remains unclear how these results generalize to more practical and real-world RL tasks.

- The novelty and significance of technical conritbution is limited. Either the concept of CDP or using performance gap between train and test set as evaluation criteria is not original. The being studied L2 regularization and dropout have also been discussed in Cobbe et al, 2018 (actually should be 2019 as the publication year of ICML).

- Although the experimental results are clear and easy to understand, I expect a more comprehensive empirical investigation, e.g.,  visualizing the learned policy in your envrionments (as in Cobbe et al, 2018), and perform experiments in more tasks. 


Other issues:

- In the end of Chap 4.1, "Referring to 14,", what is 14? Do the authors mean Figure 14?
- In "Figure" 7 (should be a table): Relu -> ReLU, ADAM -> Adam.
- The spatial interval between Figure 5 and 6 is too narrow. 
- In chap 4, before 4.1, it was written "some have reported that these techniques improve generalization.". Are there some references?
- References are not professional, please check the venue of publications instead of just citing the arXiv version.


summary_of_the_review: Overall, the paper discussed an significant topic in deep RL and their paradigm for evaluation is clearly presented and empirically investigated. However, the current study restricts itself to simple grid world envrionments with an unnatural setting to use MNIST images as observation. While the main contribution of this paper is empirical, it remains unclear how these experimental results generalize to more practical and real-world RL tasks. Also, the technical novelty and significance is limited. In sum, I consider the current paper fail to meet the acceptance criteria of ICLR and recommend a major revision, probably by performaning more comprehensice investigation on more tasks with preferrably continuous state space.


--------- Post Rebuttal -----------

The authors have addressed most of my concerns, and I increased my score by one level accordingly. However, a core limitation of the current work, "do the results/insights also apply to more realistic environments with continuous/large state space?", is not resolved. While I believe this work has its potential, currently I lean toward weak reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new methodology to test the generalisation performance in reinforcement learning. Authors study two  (non state-of-the-art) methods (DQN and QR-DQN) and analyse how L2 regularisation and dropout affect the ability of the algorithms generalising to new observations, states and actions (each of them independently)  
main_review: This paper has multiple strong points. Authors propose a new perspective to evaluate generalisation in RL agents based not on accumulated rewards as most research usually do now, but through a careful evaluation of the generalisation performance of the different elements of an MDP. From the theoretical point of view, most of the mathematical constructions are drawn from previous works (well referred). Still, this work presents a new empirical formulation to better study generalisation. 
My main concern is that this paper is not the first pointing to change in this manner the way we evaluate generalisation and, the empirical method prosed here is very limited, in the sense that it requires full knowledge of the state space and transition dynamics, e.g., it uses dynamic programming to recover optimal value and policy. Additionally, the empirical approach used in this work, mapping the states to observations of the MNIST dataset, illustrates that this whole evaluation is carried on relatively small hand-crafted environment. It leaves me doubtful if the results obtained here, e.g., that dropout only helps action generalisation, would really translate to more visually rich and complex domains. Note that authors also state that the RL algorithms were memorising the training data, I wonder what would happen if this was not the case.

Minor mistake/suggestions:
* End of page 2: “The MNIST gridworld environment used by Lee et al. (2018) may be considered a CDP, but is not recognized as such” why?
* There where some points of the literature review that became obvious only after reading the whole work, it might be beneficial moving this section to the end
* Section 2, authors don’t present the bellman equations in standard form but as expectations, it would be beneficial for the reader to anticipate here why this is relevant in this work.
*Section 2.1 second paragraph, it is not clear to me why rewards cannot be a good way to measure generalisation in the environment, (if you don’t know if your policy is optimal, only way to reduce uncertainty when you don’t have full access to the MDP dynamics is to keep checking new policies to whether one of them produces better reward)
*Section 4, a figure of the environments showing how they are used with the MNIST dataset, would help here
* Section 4.1, line 4. Figure 10 should be Figure 1
* Figure 1, it is not specified which of the two RL algorithms is used while evaluating the regularisation procedures. 

summary_of_the_review: Authors explore a non-standard method to measure generalisation in RL applying it empirically to two deep RL algorithms. Despite of authors multiple points against using rewards as generalisation metric, I am still not convinced that the struggles of RL with respect to supervised learning, which is the absence of the ground truth answer, is tackled by the proposed method. Authors rely on heavy assumptions of the environment, allowing them to have access to the "ground truth", i.e., optimal policies and value. This is not the case in most problems about RL generalisation.
review_title: null

Meta-review: This work proposes to study the generalization capabilities of RL algorithms using contextual decision processes (CDPs). CDPs allows to study generalization similar to how we are used to studying generalization in supervised learning, and can separate the generalization capabilities of a learned agent wrt observation, state and action space. This proposed measure for generalization is used in an extensive study on grid world domains to evaluate existing algorithms that aim to improve generalization.

**Strengths**
This manuscript is well written and the work is well motivated
A novel perspective and way of measuring generalization of learned agents
An empirical study that compares existing algorithms on how well they generalize in observation, state, action spaces

**Weaknesses**
Some clarity issues existed (missing links to existing literature, experimental details) 
empirical study is (out of necessity) limited to small scale grid worlds
no deeper analysis of the results, why do algorithms perform the way they do from this novel perspective of generalization, which makes it hard to understand how one could choose an algorithm for larger scale settings which don't allow for this type of analysis

**Rebuttal**
The authors updated the paper to improve the parts that were unclear, and had an extensive discussion with reviewers on the intuition of the results and converging on take-aways. Unfortunately, this intuition and take-aways have not been added. 

**Summary**
While I understand the authors wish to not speculate on intuition, I agree with the reviewers that without (experimentally supported) take-aways the provided analysis is incomplete. Understanding why each algorithm achieves the performance they do wrt this novel way of measuring generalization is the only way the proposed method to measure generalization and the evaluation can be used to draw conclusions about more general problem settings. Thus, although this is a very promising direction on an important problem, the manuscript is not ready yet for publication.
Decision: Reject