Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers learning a good representation for offline reinforcement learning algorithms with pixel-based visual observation space. The authors aim to extract the representation, which ignores any control-irrelevant information. The authors choose multi-step inverse models to learn the observation representation because the multi-step action prediction for learning exogenous-invariant representations has been shown useful in  RL theory community.

The proposed method ACRO first trains the image encoders so that the features can be used to predict multiple actions between two observations. With the well-trained encoder, the downstream offline RL algorithm TD3+BC takes in the learned representation as policy input to output actions. 

In order to evaluate the robustness of the proposed method for pixel observations, the authors introduce new datasets with temporally-correlated noise and diverse noise in the observations. The dataset is similar to D4RL but with some exogenous images or videos incorporated in the observations as noise.

In comparison with other representation learning methods, the proposed one significantly outperforms the baselines.
strength_and_weaknesses: Strengths:
The authors introduced the multi-step inverse model in offline RL problems and proved that Bellman completeness can be achieved via the representations without exogenous noise.
The experimental results support the claims well. In the datasets of noisy observations, this paper demonstrates the significant advantage of the proposed representation learning approach.
The newly introduced datasets for offline RL can be useful for the community.

Weaknesses:
The major concern is about the limitation of the representation only containing control-relevant information. How about there are other moving entities in the observations, which are not controllable by the agent but are still very important for the success of the policy? For example, a robot navigates busy streets. Pedestrians are not controllable by the robot, but the robot should consider the status of the pedestrian to avoid collisions. Will the proposed representation learning approaches fully ignore pedestrians and make errors in the decision?

Another example is robotics manipulation tasks. The robot hand should manipulate the objects, so the object position and orientations are important information for the control policy. But the proposed method may tend to ignore the object information. This is problematic and limits the scope of the proposed method.


clarity,_quality,_novelty_and_reproducibility: Clarity:
Overall, this paper is written clearly. The tables and figures are easy to read and convey information efficiently. I especially like table 1 which helps to compare different representation learning approaches.
Also, it will be great to explain the importance of the Bellman completeness more clearly and intuitively.

Quality:
The quality is good with the technically solid method and impressive experiment results.
However, the experiments are only conducted on locomotion tasks. It will be great to show results in D4RL kitchen tasks, especially because these tasks require object manipulation, and the proposed representation approach may fail in this scenario. I'm curious whether the proposed method can be modified or extended to the manipulation tasks.

Novelty:
The novelty is okay but not very great, because the multi-step inverse model has been studied for a well in the RL area, and this work is just to adapt it from standard RL to offline RL.

Reproducibility:
The code is provided but the new datasets have not been released yet. Considering there is detailed information about hyper-parameters in the Appendix, the reproducibility is fine as long as the datasets can be released later.
summary_of_the_review: This paper presented a well-motivated representation learning method for offline RL, with great experiment performance and some theoretical foundation. But the proposed method may be only suitable for locomotion tasks, due to the limitation of the learned representation. This seems not general and widely useful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-step inverse model to learn the representation for RL problems with offline data. The proposed method ACRO predicts the current action based on the current state and a future state, in a reward-free manner. ACRO can learn controllable states while keep invariant to exogenous information. Experiments in various settings show that ACRO outperforms prior methods. Theoretical analysis shows the benefits of ACRO. The paper also provides several benchmarks to evaluate representation learning in RL.
strength_and_weaknesses: ### Strengths

1. The idea is neat and easy to implement, intuitive and theoretical grounded.
2. The empirical results are promising, covering multiple scenarios.
3. Good presentation and visualization of the ideas and results.
4. Interesting theoretical insights.

### Weaknesses and Questions:

1. The novelty of the proposed method is relatively limited, as a continuous-control extension of prior work (Lamb et al. 2022). 
2. Following bullet 1, I am wondering whether the authors can comment more on the theoretical comparison with Lamb et al. 2022. Without the bottleneck constraint, will the learned representation be guaranteed to discard exogenous information? If not, the claims of Section 3 do not hold for the proposed algorithm, right?
3. The algorithm adopts a process of offline representation pretraining + policy finetuning with frozen representation. However, such representation learning may highly depend on coverage of the offline dataset. I hope there could be more discussion on the requiremenet of data/exploration, both in theory and in experiments.
4. The representation is only tested with offline RL finetuning, while it is not clear whether the learned representation is sufficient for learning optimal policies in an online manner. Intuitively, the latter would be harder and thus more interesting (the representation should be not only good for modeling optimal behaviors, but also good for exploration). If the claims hold that the learned representation is controllable, it should be able to learn a policy by interaction. Combined with bullet 3 above, I am worried that if the offline dataset does not have good coverage, downstream online policy learning will be hard.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written with reasonable quality. 

The novelty is relatively limited since the proposed algorithm is directly extended from prior work.

The authors have provided code and detailed experiment settings.
summary_of_the_review: This is an interesting paper with good empirical results. But I am hesitating to recommend it for acceptance, because 1) the method is a direction extension from prior work (and in the meanwhile it may lose the theoretical guarantees of prior work), and 2) there is a lack of in-depth discussion on the assumption and limitation of the current method, including requirement for offline dataset. I may missed some information in the paper and appendix. I would consider increase the ratings if the authors can provide convincing analysis/results for the above points.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: By defining exogeneous information as information irrelevant for control, this work works to learn a representation that removes these features in the context of offline RL. These representations are learned by taking a latent space learned through inverse dynamics modeling. It also provides a set of benchmarks for offline RL by adding background videos to Mujoco tasks. The key insight of this method is to predict the first action of the sequence connecting the first state and a state k steps in the future. 

strength_and_weaknesses: Strengths: 

This work investigates an important question of how to construct representations based on controllability and introduces a method for doing so.

This work implements a large number of baselines, which would be useful if released as code. 
It would have been nice to include in the introduction some indication of how this work innovates over existing representation learning methods since inverse dynamics models have been heavily used in RL and model-based RL.

Weaknesses: 
The description of an MDP actually describes a PO-MDP, with an exogenous block context. In addition, the typical formulation of the agent carries \pi(a|x_{1,...t}), or the history of past observations. Policies conditioned on the current observation require the "rich observation" assumption, which should be stated in 2.1, not 2.2.

There is an unstated assumption that the policies from which the data is collected are also invariant to the exogenous information, otherwise, information about the action between two states would result in improved performance. This is actually a significant issue in the offline RL setting since we don't have control over the collected data.

The work claims that the multi-step objective allows for more diverse representations because it forces the representation to encode long-range dependencies. However, only the first action in the sequence is added, so it is entirely unintuitive as to why the representation would have to be different. In the example of the action being stored in the observation, this would still be the case even if t+k is given. Furthermore, this representation introduces its own issues, since the problem itself is now ill-defined: suppose that there are multiple action sequences between s_t and s_{t+1}. In this case, the inverse dynamics model would have to output an ambiguous probability which would be dependent on the frequency one path was taken over another in the dataset. It is not clear what representation this would encode. Figure 2 also shows x_t and x_{t+k} with the same image.


A clear weakness of this work is that even though it equates exogenous with irrelevant, it is actually likely to remove information that could be task-vital. This is because the inverse dynamics model only needs to capture sufficient information to predict the first action between a pair of states, and can lose any information once it can make that prediction. Suppose that we are in an object manipulation domain where the agent uses an arm to grasp a block. The representation has no reason to capture the block because only the arm is necessary to predict the action. None of the experiments appear to capture these kinds of relationships
clarity,_quality,_novelty_and_reproducibility: The usage of exogenous does not match the typical usages this reader is familiar with. In particular, in control/RL literature exogenous is typically used to describe elements that cannot be controlled (but might still affect control), and in causal literature, it describes elements that have no incoming causal links from the variables of interest. It seems like "irrelevant" or "distractor" information would be a more appropriate term. 

This work repeatedly cites Guaranteed Discovery of Controllable Latent States with Multi-Step Inverse Models (Lamb et al 2022), which appears to approach the same problem with the same solution. If this work is by the authors, then this is disingenuous since the authors' claims are being supported by citations of the author's own work. Furthermore, the claims being supported, such as the resolving of single-step issues by using a multi-step predictor, appear to be made without strong support.

The authors contrast this work from others using inverse dynamics (ID) models by saying that this is the first which uses ID for representation learning instead of exploration. Not only is this not the case after a short search (Integrating State Representation Learning Into Deep Reinforcement Learning), but just because the ID representations are used for exploration, and often with more complex and refined techniques. If the authors do not write this piece of prior work, it would appear that this work introduces nothing new to the community.

The proposed offline RL datasets are the same (at least in conceit) as those described by Learning invariant representations for reinforcement learning without reconstruction (Zhang et. al. 2020), except that the prior work was introduced outside of the context of offline RL.
summary_of_the_review: I propose to reject this work due to lack of novelty since inverse dynamics models are a common choice for state representation learning. It also has some concerns with the blind format, since it repeatedly cites works that are likely by the same authors. Last, the writing and definitions are not entirely consistent with the motivation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn a multi-step inverse model [Lamb et al.] to learn a representation that ignores exogenous information (i.e., uncontrollable information) for offline RL. In addition, this paper introduces several temporally-correlated and diverse visual distractors on top of the v-d4rl dataset to investigate the quality of the representation in RL. The results show that the proposed method outperforms several self-supervised learning methods including CURL and DRIML.
strength_and_weaknesses: [Strength]
* The idea is technically sound.
* The results look good. 
* Introduces interesting visual distractors.


[Weakness]
* The technical novelty is not significant, as the main objective is essentially the same as Actor-Controller State from [Lamb et al.].
* The novelty compared to the prior work [Lamb et al.] is not clearly explained.  
* No empirical comparison to the closely related prior work [Lamb et al.]. 
* Some of the empirical results are not convincing. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

This paper is easy to follow. However, the contribution/novelty in comparison to the relevant prior work was not clearly described. For example, when the paper introduces the main idea, it says "Our proposed method, which we call Agent-Controller Representations for Offline-RL (ACRO), optimizes the following objective based on a multi-step inverse model: [Equation 1]", without referring to [Lamb et al.] which proposed the equivalent objective called Agent-Controller state (AC state). As someone who is less familiar with [Lamb et al.], I had entirely misinterpreted the statement and thought this objective is original until I checked [Lamb et al.]. Similarly, Lemma 1 is in fact from [Lamb et al.], but there is no explicit statement about it. I highly encourage the authors to move the majority part of Section 2.2 (Proposed Method) to Section 2.1 (Preliminaries) and only highlight new ideas in Section 2.2.

**Novelty**

The main objective (multi-step inverse model) is essentially the same as [Lamb et al.] except for a few details such as continuous representation in this paper as opposed to discretized representation in [Lamb et al.]. However, the application of this idea to offline RL setting is novel. In addition, the new visual-temporal disctractors introduced by this paper are new and interesting. 

**Quality**

* A comparison to [Lamb et al.] should be included, because the paper claims that learning a continuous representation is the main distinction (hence novelty) to [Lamb et al.]. Without this, the use of continuous representation is not well-justified. 

* It would be more comprehensive to include a noise/distractor-free setting to get an idea of how much each method suffers from visual disctractors.  

* It is unclear what "normalized performance" means, given that Figure 6 has various ranges from 100 to 500. It would be informative to present performance normalized w.r.t. demonstration performance across the entire main paper (assuming that raw returns are included in the appendix).

* Although I appreciate the analysis through reconstruction (Figure 7), it's unclear whether the proposed method is clearly better than DRIML. It looks like the reconstruction from the proposed method is just blurrier than DRIML not only for the background but also for the controllable part. 

**Reproducibility**
The paper provided details of hyperparameters, architectures, datasets, but not the code.
summary_of_the_review: Although this paper demonstrates good results on offline RL with exogenous information, the main idea seems to mostly come from the prior work [Lamb et al.], which is not clearly stated in the current manuscript. The paper would benefit from making a clear distinction from the prior work and adding a comparison to the prior work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for learning representations that are  invariant to task-unrelated information in the offline RL setting.  Theoretical results are presented which show that such representations  allow accurate value function approximation and that using  representations which are not invariant to task-unrelated information  can cause value function approximation to fail. Experiments and ablation studies support these claims.

strength_and_weaknesses: **strengths**

* The paper's about a relevant question (how do we ignore unimportant/distracting information in RL?).

* The paper is rigorous and argues its case convincingly. The approach is motivated by theoretical arguments. The experiments are extensive and show an edge over strong baselines. Ablations empirically motivate the decisions that the authors make.

**weaknesses**

* The block assumption (that the emission distributions of different states are disjoint) is limiting. It amounts to a full-observability assumption.

* Another limiting factor is that the learning problem requires data from an exo-free policy.
clarity,_quality,_novelty_and_reproducibility: * The presentation is a bit confusing at first. The description of the setup appears like a POMDP. Eventually it becomes clear that we are in a setup where the observation uniquely identifies the state. Here, I would advise using the term EX-BMDP more prominently (it is mentioned, but appears more like an aside) so that it is clear early on that we are in a special case.
* It is interesting that ACRO outperforms other method even when there are no distractors (if I understand Table 2). This would imply that the representation learned by ACRO is a better representation than alternatives, even when there is no exogenous information. This is interesting, since the theory used to derive ACRO is all based on the existence of exogenous information. Do the authors have some thoughts here?
* What is the difference in setup between Table 2 and Figure 5? As far as I understood, Table 2 contains no distractors, while Figure 5 contains the distractors used by Lu et al. [1]. What I don't understand is what "easy", "medium" and "hard" refer to in Figure 5. Also, the captions reads "Normalized results across two domains [...]": which domains?
* Where are the numbers in Table 3 coming from? Are these averages across the tasks? In which distractor setting?
summary_of_the_review: The paper makes solid experimental and theoretic contributions to the problem of discarding irrelevant information in RL. The experimental results are both extensive and in support of the proposed method. The main drawbacks I see are the requirement for an exo-free policy to learn from and the block assumption, which excludes POMDPs.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This papers aims to find representations for reinforcement learning that are not influenced by irrelevant superfluous information. While the paper and its results were deemed interesting by some reviewers, there was strong concern repeated by several reviewers: the novelty compared to the prior work of Lamb et al., 2022 seems to be quite limited. As such, I recommend this paper to be rejected.
justification_for_why_not_higher_score: There was a major concern repeated by several reviewers.
Decision: Reject