Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new experimental setting for RL called relay evaluation, which tests the ability of the RL agent to handle the trajectory of other agents (strangers) in the mid-way. Such a situation may be helpful in the application of RL when the agent needs to take control from humans and is seldom considered in the literature. It turns out that current RL algorithms can easily fail in the relay evaluation. Therefore, the authors propose a naive framework and a self-trajectory augmentation framework to improve the generalization ability of the RL agent, with any RL algorithm in theory. They use the trajectories, either trajectories from other stranger agents or historical trajectories to enhance the RL agent. The results on multiple datasets with different RL algorithms show the performance of the proposed frameworks.
strength_and_weaknesses: Strength:

1. To the best of my knowledge, the authors are the first to consider the relay evaluation setting in the field of RL, which does have real-world usage such as in the self-driving system. I think this evaluation could be used as an important check before the application of RL to the real world.
2. The results of the relay evaluation show that existing RL algorithms can easily fail to handle this situation. Figure 1 illustrates the results well and shows a possible explanation for the failure.
3. The authors propose two frameworks to improve existing RL algorithms, both of which better utilize the trajectories, either trajectories from other stranger agents or historical trajectories. In theory, these frameworks can be combined with any RL algorithms and are easy to modify.

Weakness:

1. In my view, the motivation of this work is unquestionable but I have some concerns. First, the authors provide an example situation that "a self-driving system may need to take over the control from humans in the middle of driving and must continue to drive the car safely."  Under such a situation, the algorithm used in the RL agent is known by the users, right? However, in the experiments, agents using different RL algorithms are considered the main part of the experiments (A-B pairs in Table1). I wonder does there exist a gap between the experiments and the claimed application scenarios.
2. If we consider the adaptation of different trajectories, I think the works of meta-RL may be very relevant to this work. In my view, meta RL can handle different distributions of the environment, which could be seen as different trajectories as well. Therefore, could the authors compare their works with several popular meta-RL frameworks?
3. After looking at the results of Table 2 and Table 3, I think the performance of STA is kind of unsatisfactory compared to the naive algorithm. So, I begin to question the performance of STA, maybe the simple way in the naive algorithm can already handle relay evaluation?
clarity,_quality,_novelty_and_reproducibility: In my view, the novelty of this work is unquestionable since the relay evaluation is hardly considered in the literature. But the quality of the work is not so satisfactory due to the weakness I find. As for reproducibility, I can not make a judgment since the authors do not plan to open-source their code.

summary_of_the_review: This paper aims to solve the ray evaluation issue which is hardly considered in the literature. However, except for this point, the other parts of this paper look not good enough to me. As I mentioned in the weakness part, the authors need to clarify their motivation and enhance their experiments. Therefore, I consider this paper is not yet ready for ICLR. Admittedly, I am not an expert in this area, if there are some fatal mistakes, please feel free to tell me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses the challenge of "relay generalization", where a policy trained with RL is evaluated when starting from initial states observed under an independently trained policy.  Relay generalization is intended to measure the out-of-distribution generalization error, by testing how well the policy performs starting from the "reasonable" states found by another good policy.  They demonstrate in some of the MuJoCo environments that policies that perform well from from the environment's default initial state distribution can fail catastrophically when they are required to take over control from other, equally effective policies.

They determine that the issue stems at least in part from the fact that independently trained policies tend to settle into different regions of the state space, and so never learn how to take successfully control the agent from the states that other policies concentrate on.  The propose a simple method to mitigate this issue, in which the current policy is, with some tunable probability, rolled-out from a state encountered previously during training, rather than an initial state of the environment.  They demonstrate that this significantly improves relay generalization when applied to the soft actor-critic algorithm.
strength_and_weaknesses: The main contribution of the paper is the idea of "relay generalization", and empirical demonstration that this issue actually exists.  While it is not entirely surprising that on-policy methods would fail when asked to take control from a state they would not typically encounter during training, this work provides a means of directly measuring this type of out-of-distribution error.  They also provide some qualitative insight into exactly why these failures occur in the MuJoCo environments.

The main weakness of the paper is that the empirical results are limited to four MuJoCo environments.  The failure mode, at least in the humanoid environment, seems to be somewhat specific to this setting, and so it is unclear how large a problem this would be in general.  It is also unclear how much of an issue this would be with fully off-policy methods with large replay buffers, which might have a similar mitigating effect to the initial state sampling method they propose here.  Additionally, the solution they present (STA) is relatively simple, and it is unclear how much it can improve generalization performance overall, for example, from states that no policy would be likely to find during training.
clarity,_quality,_novelty_and_reproducibility: The novelty of the work comes mainly from the explication of relay generalization as a specific form of generalization error in RL.  The article appears to be technically correct, and is clearly for the most part.  A few suggestions for improvements, however:
1. The related works section could be more organized, as it is currently just a long block of text.
2. Table 1 (or an additional table) could show actual returns in addition to failure rates.  Table two provides this, but divided out between "failure" and "success" trajectories, so the actual impact of relay generalization of quantitative performance is not immediately clear.
3. "Failure" in this case is determined by each environments termination criteria, these should be described for each environment, as it is possible the termination criteria in the MuJoCo environments are too strict, and terminate policies that might eventually succeed.
4. Definition 2.1 is not entirely clear, as it does not seem to require that a trajectory be "feasible" given the action space available to the agent (seems to admit trajectories that arbitrarily jump between states).
5. While we assume this is the case from the context, it is not explicitly stated that the "stranger" and "test" policies are trained using the same configuration, for the same number of steps.
5. There are a number of typos, for example "very slow" rather than "very low" in the first paragraph of page 4.
summary_of_the_review: Overall this work empirically demonstrates an interesting phenomenon that is important for RL researchers to be aware of.  It provides a heuristic solution to this specific issue, but does not tackle the problem of out-of-distribution error more generally.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose "relay-evaluation", a method for evaluating the generalization performance of RL agents. Specifically, it forces the agent under test to start from the state s\_0, which is sampled from the trajectory generated by stranger agents. A sharp degradation of performance is observed under relay-evaluation, which proves the limitations of existing methods. Correspondingly, the authors proposes STA, a novel method that can be applied in any modern RL algorithm, to improve the generalization ability.

Furthermore, there is no code or any other supporting material for this paper, and the reproducibility of the paper is unconvincing.

strength_and_weaknesses: #### Strength

Overall, generalization is a popular and essential topic in RL, and many methods have been explored to evaluate or improve the generalization of RL agents from different aspects. It is quite interesting that the authors studied the generalization from a brand-new point of view and proposed a concise but effective method. The experiments are compelling and well-rounded. I think the paper did contribute some new ideas and open up a new direction worth being studied.

#### Weakness

Despite of the outstanding performance illustrated in the experiments, a provable improvement might can not be guaranteed as many components applied in STA are more likely intuitive.   

In relay-evaluation, authors sampled states from trajectories generated by stranger agents, which are trained with only different random seeds. PBT[1] might be of help to find more "controllable" states where test agent fails and thus evaluating the generalization more accurately.

Not really a weakness, more a suggestion. A pseudocode that describes the process of STA overall might help authors understand it more clearly. Would you consider adding it to the paper in the next version, please?

There is no code or any other supporting material for this paper, and the reproducibility of the paper is unconvincing.

clarity,_quality,_novelty_and_reproducibility: #### Clarity

The paper is mainly well-written and is generally easy to follow. The phenomenon that existing methods fail in relay-evaluation is motivating, and the analysis is persuasive. 

#### Quality

The quality of the paper is high, and the experiments are compelling, as far as I can see.

#### Novelty

To my knowledge, the paper is the first to propose a new method for evaluating the generalization of RL agents on "controllable" states and improve it via STA, a new training paradigm.

#### Reproducibility

A/N

summary_of_the_review: Overall, the paper is very interesting as it studies the generalization of RL agents from a novel view, despite some minor defects like the lack of provable improvement. The paper is well organized, and the analysis of the experiments is convincing. The proposed method seems concise but effective enough without introducing extra sample complexity.  

[1] Derek, K., & Isola, P. (2021). Adaptable Agent Populations via a Generative Model of Policies. *Advances in Neural Information Processing Systems*, *34*, 3902-3913.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes “relay-evaluation” a method to asses the generalization capabilities of RL agents when being forced to start from a state that is out of the distribution defined by the agent’s policy. The paper also introduces “Self-Trajectory Augmentation” (STA) a training method to improve generalization in which the agent is randomly set to start from a state that is sampled from one of the agent’s old policies.
strength_and_weaknesses: ### Strengths
This paper addresses an important problem in RL that I believe hasn’t been explored sufficiently. Generalization in RL is normally regarded as the ability of an agent to perform well on similar tasks/environments.  However, this paper evaluates generalization to out-of-distribution trajectories, showing that, even when the environment stays the same agents may fail to perform well if states are sampled from a different policy. The paper is clearly written, for the most part, and the experiments are thorough.

### Weaknesses
The relay-evaluation metric is computationally heavy it requires training multiple other agents to ensure that there is sufficient variety in the policies of the strangers. If the number of stranger policies is small, the method may incorrectly determine that an agent is able to generalize well just because the algorithms used for training the test and stranger agents are very similar.

The definition of “controllable state” is somewhat loose (i.e. what does “high return” mean and how large does L need to be?).

I would have liked to see experiments on other types of environments. The environments used are all of the same nature. Agents need to learn to walk/move and keep their balance. These require a lot of coordination and thus it is natural for the agents to overfit to a particular trajectory.

Can the STA training scheme negatively impact the agent’s performance? As explained in Section 3.1 STA is equivalent to training in an augmented MDP with a larger initial state set. Is it possible that the optimal policy for this augmented MDP is different from that of the original MDP? 

As far as I can see, a simulator that can be reset to any state is needed for STA.

I found Section 3.2 particularly unclear: 

  * The score used by STA to evaluate if a state is controllable depends on the policy being followed. Hence at the beginning of training when policies perform badly, the inclusion of states in the initial set seems arbitrary. That is, some states may not be included just because these bad policies can't perform well when starting from them.

 * Is the score of a state averaged over multiple runs? Or is it just a single sample estimate? The latter is a poor estimate of the true score if the policy and/or the environment are stochastic. Why don’t you use the Q network to estimate the score?

 * I am not sure why the states need to be controllable to be added to the initial set. What happens if you just add all states to the starting set regardless of whether they are controllable? Wouldn’t that also improve generalization?

 * Why is $Q(s, \pi(s))$ a good metric to determine if the agent is familiar (not familiar) with a particular state? A high (low) Q value may be due to the agent being overconfident (underconfident) about that particular state.

 * The use of a parameter $\lambda$ to truncate the return is a bit hacky. Can't you just divide the score by the number of remaining timesteps?

 * How is $\eta_{STA}$ computed?
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper is generally clear although it could benefit from proofreading. The second half of the paper where STA is introduced is hard to follow. In particular, Section 3.2 is quite convoluted. It is difficult to parse how the method works especially given that STA uses so many new hyperparameters.  Perhaps adding bullet points or pseudocode explaining step-by-step how states are added to the initial set may improve clarity.

Typos/grammar mistakes:

Abstract “generalizatin failure” -> “generalization failure”

“pretraining additional models are time-consuming” -> “pretraining additional models is time-consuming”

“based the experiment of Table 2” -> “based ON the experiment IN table 2”? 

“which are very slow” -> “which are very low”?


### Novelty 
I found the paper novel, although I must admit I am not very familiar with the literature on generalization in RL.

The idea of “relay-evaluation” seems similar in nature to that of “Tandem RL” (Ostrovski et al., 2021). Could the authors comment on how these two papers are related?

Ostrovski et al., 2021: The Difficulty of Passive Learning in Deep Reinforcement Learning.

### Reproducibility

Source code is not provided. However, the method should be relatively easy to implement.

summary_of_the_review: I am overall in favor of accepting this paper. I really liked the first part of the paper where the problem of generalization to out-of-distribution trajectories is evaluated using the relay method. I am a bit less excited about STA since the method makes lots of choices that seem rather arbitrary and very domain dependent. I am tentatively giving the paper a score of 6 but I am willing to increase it if the authors can clarify some of my concerns in their rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new measure for generalization in reinforcement learning - the performance of a policy when starting from states generated by another trained policy. The scenario is motivated by autonomous driving where the agent has to take over from a human. The paper shows that even if both policies perform well in isolation, the newly learned policy might be poor at this generalization as it focuses on a different state space. Including states that have occured during training as intial states is shown to solve this issue.

Strength and weaknesses: The paper presents a new and very intuitive measure of generalization for reinforcement learning. In addition, the presented work-around, i.e., including initial states that occurred during early training is intuitive and simple (in a good way) and seems to be effective. In the downside, the experiments could be more exhaustive as only 4 mujuco environments are tested in an "agent takes over from an agent" setting instead of testing the "taking over from a human" setup. Yet, all reviewers agreed that the presented idea is intruiging and could lead to many follow up work.
justification_for_why_not_higher_score: Experiments are too limited.
justification_for_why_not_lower_score: Paper motivates a new way to measure generalization in RL which is very well motivated. Moreover, the solution strategy to get a better generalization regarding this measure is intuitive and effective.
Decision: Accept: poster