Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the issue of inconsistency in return-conditioned supervised learning (RCSL), such as Decision Transformers. Specifically, when RCSL in highly stochastic environments is conditioned on the highest dataset return, the resulting expected return could be lower as the environment randomness is not under the agent's control. To this end, this paper proposes to capture only the agent's controllable factors and minimize the learning dependence on future rewards and transitions — which are environment characteristics. Under reasonable assumptions, the paper proves that their latent variable approach with Mutual Information constraints leads to policies consistent with their conditioning input reward.
strength_and_weaknesses: ## Strengths
- The paper identifies an important problem and solution for RCSL with inconsistent conditioning, especially in stochastic environments.
- The paper is clearly written with all the necessary details and flow to understand everything.
- A good set of simple environments are chosen, including a didactic Bernoulli Bandit environment and MuJoCo benchmarks.
- The paper has theoretical results on the consistency of their proposed method.


## Weaknesses
- **Missing crucial comparisons**
    + There are two key differences of DoC from prior approaches (i.e., VAE) using latent embedding of future to mitigate inconsistency: (a) Mutual Information (MI) constraint and (b) Inference from a learned prior and value function. The importance of these two components must be ablated on all the environments. Specifically, the following comparisons can be added:
        * VAE + Inference with learned prior (a separate copy with stopgrad) and value function
        * DoC w/o MI constraint (= DoC - a)
        * DoC + conditioning on highest return (= DoC - b)
- **Comparison against VAE**
    + Since future-VAE also regularizes the z to the learnable prior conditioned on the past, this ensures that the latent z is incentivized not to use future information. Therefore, DoC's key benefit must come from utilizing the controllable part of the future in z while ignoring the environment transitions. Why is it expected that encoding the controllable information in z will improve the performance? What are example environments with this property?
    + The difference between the ideology of VAE and DoC being seemingly small is reinforced by the results on the mujoco environments, where the empirical performance of VAE and DoC are pretty similar. Even on reacher, where KL beta is not tuned well, VAE first reachers almost the optimal performance before falling. Therefore, it is not clear that DoC is necessarily better than VAE.
        * Is more stochasticity in mujoco environments expected to increase the difference in performance between DoC and VAE?
        * Is any other experiment (quantitative or qualitative) possible to show that the importance of encoding controllable future information in z is helpful?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written with good quality and novelty. However, certain experiments are needed to justify the novelty.
summary_of_the_review: I like the paper's writing, problem, and proposed approach. However, it is missing some crucial justification as to why it is expected to be better than the best baseline (VAE), which prior work has proposed to mitigate inconsistency. I would happily reconsider my rating if the above issues are addressed, resulting in a clear demonstration of improvement over baselines and ablations.

----  
[Post-rebuttal]
The author response addressed most of my concerns, except "DoC does not convincingly outperform VAE in control tasks." I have increased my score to reflect this.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: After reviewing the limitations of the current RCSL methods in capturing reward stochasticity, this paper has proposed a new Learning algorithm DoC that solves the control problem in a stochastic environment by separating what is controllable from what is not. The former is formulated as the actions at hand at each timestep, while the latter is interpreted as the stochastic environment transition probabilities. They have learned a new latent variable to represent future information and aid inference and included mutual information constraints in the optimization problem. Experiments comparing DoC with mainly RCSL/DT and future VAE methods proved the novelty of the paper in terms of the recovery of the optimal policies.
strength_and_weaknesses: Strength:
1. The paper has clearly pointed out the problem sets they are trying to improve on. And the way they came to their choices on mutual information sounds natural. 
2. Most of the notations and the derivations are clear to follow
3. The experiment setup and resulting figures are clear

Weakness:
1. Some part of the derivation mentioned in the next section is confusing to me
2. Some of the phrasings are very abstract and can be more rigorous, such as "By only capturing the controllable factors in the latent variable, DoC can maximize over each action step without also attempting to maximize environment transitions as shown in Figure 1".
clarity,_quality,_novelty_and_reproducibility: The paper elaborates most of the conclusions clearly and presents a new and novel view of the stochasticity in the RL environment, but with few confusions left, leaving out derivation details without any reference to the appendix. Most of the notations, theories, and assumptions are accessible. I have listed a few points I am confused about below:
1. How do you convert the problem of minimizing the mutual information to maximizing the same equations derived from mutual information?
2. How to calculate the corresponding value function for a give z during inference is ambiguous

summary_of_the_review: Regardless of the derivation confusion, it causes me, it is a novel and original paper to me from both points of view of optimization goal formulation and empirical results. I would consider this an inspiring direction to which I would like to contribute.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new regularizer for future-conditioned supervised learning that specifically prevents the learning procedure from being biased by lucky rollouts (in terms of environment stochasticity). On highly stochastic environments, the proposed method achieves higher performance compared to the baselines.
strength_and_weaknesses: Strength
- Empirical performance on environments with high stochasticity is strong compared to other future-conditioned supervised learning baselines.
- The paper presents various examples and experiments to backup their claims.

Weaknesses
- DoC inference requires sampling latents from the learned prior and choosing the latent with the highest (estimated) value. I worry this process adds randomness to the policy's performance and thus is unstable. Also, if there exists only few expert quality trajectories on the dataset and other trajectories are low quality, much more sampling will be required to include a latent with good returns. In addition, the paper does not discuss the protocol for setting K (number of samples hyperparameter).
- The authors did not provide the source code, which makes the credibility of the paper questionable.
- The paper lacks results on hyperparameter sensitivity. For example, how does the return curve change if we change \beta and K?

Questions
- On Gym MuJoCo, why is time-correlated noise used instead of simple Gaussian noise?
- Can the authors provide results on applying the proposed regularizer to other future-conditioned supervised learning baselines, for example RvS [1]?

[1] Emmons et al., RvS: What is Essential for Offline RL via Supervised Learning?, ICLR 2022.

######## Post-rebuttal comment ########

The updated manuscript addresses my concerns mentioned above. Thus, I am raising my score.
clarity,_quality,_novelty_and_reproducibility: Mentioned above
summary_of_the_review: The paper claims that current future-conditioned supervised RL methods lack the ability to distinguish environment randomness from policy randomness and introduces a new regularizer to address this. While the proposed regularizer seems to help boost performance on highly stochastic environments, more experiments and clarifications of the proposed method is required.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper is in the field of "Future- or return-conditioned supervised learning". Here the main idea is to learn, from a batch of data, a policy that is conditioned on a latent variable z, which is often the return of a given trajectory.  However, in stochastic environments, the return is often not driven by the performance of the actions executed,  but by the inherent stochasticity of the environment. 

Here, the authors introduce a novel objective function, by penalizing the mutual information between the latent variable and the (environment-driven) stochastic rewards. (The method is (loosely) inspired by the stoic concept of dichotomy of control, which they used to name the method and the paper).  

The authors  test this method on a set of stochastic benchmarks  (often deterministic, but adjusted to be stochastic) and show better stability and performance
strength_and_weaknesses: Pros
- The paper is written very well and good to follow
- The paper is focused on a relevant question in the field
- Concepts are well introduced and explained
- Experiments make sense and are executed nicely: testing in stochastic scenarios of different scale -- visualization, experiment repetition etc.


Cons:
My main critique is around the point of universality:  I feel a bit unsure about how universal the approach is for different forms of stochasticity in the environment and how well it would perform if no stochasticity would be present:

 1. I feel a bit unsure about Eq. (7) (especially in continuous cases): to calculate the mutual information between the reward r_t and z (and s_{t+1}) you say you "we set ρ to be the marginal distribution of rewards in the dataset" and that is has to be a  "fixed sampling distribution of rewards"  But this sounds like you will have to make simplifying assumptions here about the stochasticity of the s_t and r_t: 
    - are you limited to Gaussian and empirical (histogram) distributions? What happens if the environment is multi-modal or long-tail in its state or reward transitions? 
    -  do you learn the parameters of the conditional distribution \omega?
    -  what happens if the environment would be stochastic in some regions and deterministic in others?
    -  Looking at the experiment, I noticed the algorithm is only tested in situations where randomness is either Bernoulli (6.1, 6.2) or Gaussian (6.3).  


2. The authors evaluate their method solely on stochastic tasks (or deterministic tasks, like MuJoCo but made stochastic). I do not know how well the method performs under standard (deterministic) settings. E.g. perhaps it will behave a bit more conservative because it assumes some form of stochastic behavior






clarity,_quality,_novelty_and_reproducibility: The writing is very clear and foundations are well explained. However,at the key contribution: the addition of constraints in the objective using mutual information and the derivation of the objective, here the authors could have made the steps and key assumption more clearer.
summary_of_the_review: The methodology is novel and its performance is empirically shown.   I have some doubts regarding the universality of the approach, which may be cleared up during the rebuttal and subsequent discussions. 

Therefore, for now I vote for acceptance with some reservations regarding empirical significance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
Summary:


The paper reviews the limitations of the current RCSL methods in capturing reward stochasticity. Then it proposes a new Learning algorithm DoC that solves the control problem in a stochastic environment by separating what is controllable from what is not. The former is formulated as the actions at hand at each timestep, while the latter is interpreted as the stochastic environment transition probabilities. They authors learn a new latent variable to represent future information and aid inference and included mutual information constraints in the optimization problem. Experiments comparing DoC with mainly RCSL/DT and future VAE methods proved the novelty of the paper in terms of the recovery of the optimal policies.

Strengths:

- Empirical performance on environments with high stochasticity is strong compared to other future-conditioned supervised learning baselines.
- The paper presents various examples and experiments to backup their claims.
- The paper has clearly pointed out the problem sets they are trying to improve on. And the way they came to their choices on mutual information sounds natural.
- Most of the notations and the derivations are clear to follow
- The experiment setup and resulting figures are clear
- The paper is written very well and good to follow
- The paper is focused on a relevant question in the field
- Concepts are well introduced and explained
 Experiments make sense and are executed nicely: testing in stochastic scenarios of different scale -- visualization, experiment repetition etc.
- The paper identifies an important problem and solution for RCSL with inconsistent conditioning, especially in stochastic environments.
- The paper is clearly written with all the necessary details and flow to understand everything.
- A good set of simple environments are chosen, including a didactic Bernoulli Bandit environment and MuJoCo benchmarks.
- The paper has theoretical results on the consistency of their proposed method.

Weaknesses:

- Some of the phrasings are very abstract and can be more rigorous.
-  unsure about how universal the approach is for different forms of stochasticity in the environment and how well it would perform if no stochasticity would be present:
- DoC does not convincingly outperform VAE on the chosen MuJoCo tasks

Recommendation:

All reviewers agree on acceptance. I have therefore decided to accept the paper. I recommend the authors to look at the feedback provided by the reviewers and use that to improve the paper for the camera ready version.
justification_for_why_not_lower_score: The reviewers provide high scores. The reviewers point out numerous strengths of the paper, while the weaknesses are very few.
Decision: Accept: notable-top-5%