Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an approach to handling the challenge of applying RL via supervision methods to stochastic environments. The method is empirically demonstrated to outperform two recent baselines across three environments.
strengths_and_weaknesses: At the bottom of page 2, the paper claims three contributions:
1) the observation that RL via supervision agents "can only reliably achieve outcomes that are independent from environment stochasticity."
2) the proposed method ESPER
3) the stochastic offline RL benchmark tasks used to evaluate ESPER

However:
- observation (1) was previously made in "Shaking the foundations: delusions in sequential models for interaction and control" Ortega et al. [Arxiv 2020]
- the environments (3) contribution cannot be fully evaluated as code is only available for the simplest of the three environments. Open sourcing all environments would improve the significance of this contribution.

ESPER (the proposed method) remains a novel contribution that significantly outperforms two recent related baselines. The empirical evaluation of this method could be improved by including CQL in figures 4-6 and quantifying the correlation between performance and target return currently visualized in Figure 6. I agree with the conclusion the authors reach from this plot, but in this case believe the raw value would be more informative than the plots currently used. The space saved could then be used to include the more insightful results currently in Appendix A.3 on the representations learnt.
questions: 1) Please clarify how the observation regarding the fundamental problem of applying RL via supervision agents to stochastic environments differs from or compliments the findings of "Shaking the foundations: delusions in sequential models for interaction and control" Ortega et al. [Arxiv 2020]. This could improve my rating of the significance of contribution (1).

2) Why was CQL not included in Figures 4-6? Does including it provide further insight into the relative strengths and weaknesses of these approaches? This could improve my rating of the significance of contribution (2).

3) On lines 243-244, the paper notes "ESPER can both win and lose with perfect accuracy depending on the target performance" but misses an opportunity to discuss why it doesn't do well at drawing (0 target return). Is this because this return is not included in the training distribution due to the policies used to generate the offline dataset of trajectories? This question would help clarify my current understanding of the empirical results.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work. I would be very interested in seeing the follow up study proposed regarding testing the performance of ESPER on other types of goals as I agree this may require further innovation.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper first identify that the existing offline "reinforcement learning via supervised learning" (RvS) methods that conditioned on returns/outcomes (e.g. Decision Transformers) can only reliably achieve outcomes that are independent from environment stochasticity. In other words, they do not consider whether the offline demonstration results are achieved by pure luck. 

Based on this observation, the authors proposed learning learn environment-stochasticity-independent representations of trajectories, by adversarially clustering offline experiences by data collection policy, to disentangle actions from environment stochasticity. Trajectories in each cluster can be annotated with the average return for trajectories. These pieces together give ESPER, environment-stochasticity-independent representation. RvS with ESPER achieves substantial improvements on several simple stochastic environment benchmarks: Gambling (an illustrative single-step stochastic environment), Connect Four with a stochastic opponent, and 2048.
strengths_and_weaknesses: Strengths:
* I appreciate the insightful observation that existing RvS can only reliably achieve outcomes that are independent from environment stochasticity. This paper presents a clear illustration to explain why this could be an issue. I believe future RvS research could potentially benefit a lot from this work.
* The proposed solution is theoretically sound. Empirically, it shows decent improvements, compared to return conditioning, on several simple benchmarks, including Gambling, Connect Four, and 2048. I think these are enough proof that the issue does exist, and that the proposed method ESPER achieve some initial successes.

Weaknesses:
My concerns are about whether this paper has discussed enough limitations of the proposed solution (i.e. clustering by policy) and enough future challenges. Besides high-dimensional observation space, I think clustering by behavior/policy implies that, when the data is collected with many behavioral modes, the number of clusters need to be able to scale with that. For that reason, I think we will perhaps see that it struggles with high-dimensional and continuous action space. Potentially, (state-dependent) no-ops and near-no-ops could also lead to unnecessary complexity. Furthermore, my haunch is that clustering by behavior will go against behavioral compositionality, making it difficult to interpolate desired outcomes, as opposed to return-conditioning, and thus probably go against the authors' desire to "pave the way for creation of more general agent" as stated in the conclusion section. 

Given the early position of this paper (which focus on a limited set of empirical evaluations), I would like to see much more discussion about the potential limitations and future challenges of the proposed approach, to help future works to extend this contribution. If these could be sufficiently addressed or clarified during rebuttal, which I think would not be very difficult, I will increase my rating to 6 or 7 to recommend acceptance. 
questions: As mentioned in the above section, I wonder if we can have more discussion around potential limitations and future challenges of the proposed approach.

nitpicks
* In the Figure 2 caption, it's unclear what "... such that the dynamics within each cluster matches the environments dynamics ..." means. I suggest rephrasing it.
* I recommend spelling out "ESPER" at least in introduce. It doesn't appear until Sec 2.3, which makes it difficult to follow.

limitations: The potential negative societal impact is addressed. However, the limitation is not sufficiently discussed, which I think is also my main concern about his paper and discussed in the "Strengths And Weaknesses" section.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Offline Decision Transformers trained with supervised learning do not account for the fact that some favorable outcomes are due to stochastic dynamics and not to the quality of the policy. 

Therefore, by design, some probability mass may be distributed to sub-optimal actions. The solution proposed is to focus on trajectory statistics that are independent of uncontrollable randomness in the environment. From cluster assignments based on these statistics, useful conditioning quantities such as expected returns can be learnt.

Three predictors are optimized alternatively with two losses:

- On the one hand, a transition predictor is trained to predict the next state conditioned on the previous state, action, and the trajectory cluster assignment.
- On the other hand, the cluster assignment predictor is trained to maximize the loss of the transition predictor while maximizing the likelihood of a policy predictor trained to predict the action conditioned on the current state and cluster assignment.

Finally, another predictor is trained to predict the return conditioned on a cluster assignment. It is then used to label trajectories for policy learning conditioned on expected returns.

The authors propose a benchmark of three stochastic environments and build offline datasets with various policies ranging from random policies to experts. Empirically, their approach outperforms naive return-conditioning and a Conservative Q-Learning baseline.
strengths_and_weaknesses: # Strengths

- To the best of my knowledge, this method is new. It is simple to understand, well motivated, and easy to implement.
- Claims made at the end of the introduction are supported by experimental results.
- The paper is mostly easy to follow and understand.
- The authors highlight a potential key limitation of Decision Transformers trained with offline trajectories in stochastic environments. As Transformers become more popular as agents, such insights are beneficial to the community.

# Weaknesses

The paper is titled “Why Decision Transformers Fail in Stochastic Environments” but not a single Decision Transformer is trained nor evaluated in this work. While the authors provide evidence that MLP-based behavioral cloning agents conditioned on returns fail in toy stochastic environments, they do not illustrate this phenomenom in more complex environments with recent architectures. For instance, Decision Transformers in [Atari with sticky actions](https://arxiv.org/abs/1709.06009) or [Crafter](https://arxiv.org/abs/2109.06780) could be considered.
questions: On a simple Atari environment considered in the original [Decision Transformer](https://arxiv.org/pdf/2106.01345.pdf) paper (e.g. Pong or Breakout), would it be possible to run your set of experiments with an actual Decision Transformer instead of an MLP? Their [code](https://github.com/kzl/decision-transformer) and dataset of offline trajectories are available.


limitations: The main limitation of this work is the lack of empirical evidence beyond toy environments and multi-layer perceptrons. While I agree that MLPs can be used instead of Transformers in the MDPs considered, the question of whether Decision Transformers fail in complex stochastic environments is not answered.
review_title: null

Meta-review: metareview: The authors explore a fundamental limitation of Decision Transformers and related RL via supervised learning approaches when applied to stochastic environments. They propose a new and simple approach that clusters experiences to disentangle action quality from environment stochasticity. Their ESPER approach achieves large improvements on a number of simple, stochastic environments including Gambling, Connect Four and 2048.

The reviewers were all satisfied by the novelty, technical soundness and relevance of this work for the NeurIPS community, but were initially of mixed opinions about the selection of challenge domains. Having discussed this point at length with the reviewers, I am satisfied with the authors' choice of environments for two reasons: (1) they allow the specific shortcomings of previous methods to be isolated and addressed directly, and (2) they are consistent with the environments used by previous related work published in top-tier conferences. I am recommending this paper for acceptance accordingly.
confidence: Certain
Decision: Accept