Kati8/ICLR_2023_2025 huggingface dataset:
authors: Morgan Byrd
authors: Maks Sorokin
authors: Robert Wright
authors: Sehoon Ha
reviews: 
Comments: This paper tackles the challenge of enabling intelligent agents to generalize to unseen environments by focusing on changes solely in observation spaces. The authors introduce a novel approach that trains policies using only the history of rewards and actions, excluding direct observations. They hypothesize that dense reward signals can facilitate zero-shot transfer across diverse environments. The proposed framework leverages temporal histories of reward-action pairs and expert guidance to train reward-based policies. Experiments in Pointmass, Cartpole, and Car Racing demonstrate that these policies achieve 60% to 90% of the performance of standard observation-based policies while maintaining transferability to environments with significant observation shifts.
Strengths: - **Novel Approach:** The paper presents an innovative method for training policies using only reward and action histories, offering a fresh perspective in reinforcement learning.

- **Intuitive Explanation:** The authors clearly and convincingly explain why reward-based policy learning can be effective, especially in navigation tasks. For example, learning a goal-conditioned policy remains feasible by only incorporating the reward signal.
Weaknesses: - **Limited Real-World Applicability:** Training without direct observations restricts the method’s applicability, as real-world scenarios often involve more complex changes beyond observation shifts. This limitation raises concerns about the generalizability of reward-conditioned policies in diverse settings. What are the possible scenarios in the real world where only observation changes within the MDP? 
- **Scalability Issues:** While the reward-based learning approach shows promise in simpler environments like Cartpole, its effectiveness in more complex tasks such as locomotion, manipulation, or humanoid control remains uncertain. Can reward-based policy handle complex locomotion or manipulation tasks? Since this work focuses on understanding transfer benefits, there is concern that it might struggle to learn necessary behaviors before transfer.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper introduces a method called reward-based policy. It makes the assumption that rewards are observable in a reinforcement learning problem, and attempts to learn a policy that outputs actions based only on previous rewards and actions, and no states or observations. The hope is that such a policy will be transferable between environments where dynamics (and reward functions) are the same.
Strengths: The paper is very well-written and very easy to follow and understand.
Weaknesses: I think the major problem with the paper is that the studied setting is very unrealistic. The paper provides some discussions, which also acknowledge how difficult it is for their assumptions to be satisfied. At the end, it is still difficult to think of any application where the proposed approach is going to be useful.

Specifically, I am not worried about the assumption that the reward is observable. But what are some real-world problems / environments where an RL agent can perform reasonably well just by seeing the previous actions and rewards? As the paper acknowledges, this is exactly like trying to play a game by looking only at the score and no other parts of the screen. Other than environments where it is possible to memorize the solution, it seems to me that these conditions are extremely hard to satisfy. And this is not even the only assumption. In addition to these, there should also be some transfer concern where the robot's transition and reward functions stay the same but the observations change. I don't see any useful application.

A somewhat promising part of the paper is the section where it tries to estimate rewards from observations. However, that section requires access to a state-based policy that solves the task. Again, this is very unrealistic. If the problem is a POMDP, how would one have access to a state-based policy? The states are not even known to the agent.

Finally, the main promise of the paper is not very interesting. When the learned policy does not depend on the observations, of course the observation function can be changed in any arbitrary way. Under that setting, that function is completely irrelevant. So the experiments are completely unsurprising and obvious.

Below are my other comments that are more minor:
- et al. is plural, so those citations should be thought as "they" instead of "it". There are some grammatical errors about this.
- The POMDP definition is missing the function that maps the states to observations.
- The so called reward-based policy is defined as $R\times A \to A$ but my understanding is that a full history of rewards and actions are inputted, not just the most recent ones. In that case, this definition of the policy function is incorrect.
- Incorrect capitalization in line 248.
- Again, the second hypothesis/question (line 253) is validated by the definition of reward-based policy. Why is it even a hypothesis?
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 4
reviews: 
Comments: The authors propose a method to learn transferable policies by treating rewards as the observation space on which a learned policy operates. Specifically, they learn behavior policies that utilize as input, a history of observations and actions and generate future actions. These inputs are aggregated using LSTMs which generate actions. This model is trained using PPO online along with an offline behavior cloning loss on a demonstration dataset. The method is evaluated on 3 simple environments - Cartpole, pointmass and Racing Car.
Strengths: 1. The paper is well written and easy to understand
2. The method is principled and intuitive. 
3. The authors perform ablations and provide statistically significant results.
Weaknesses: 1. **Lack of a justifiable motivation:** It's unclear why one would use only rewards as the observation space in practice. For any complex dynamical system, having a policy dependent on an observation is required. Theoretically, this would only work in stationary bandit-like settings, where a single action optimal action exists from the ones available and a history of past actions and rewards would suffice to take the right action. However, in all other “RL” settings where an action taken changes the state of the world, i.e., environments with a transition function, this method would break. Also, it’s hard to motivate this method from a practical perspective - in most real applications, one would utilize all possible information available to learn a behavior. In short, the authors claim they study a much more complex problem - that of reinforcement learning, while applying a primitive set of assumptions - those of a multiarm bandit. I don’t believe this method would work under the advertised conditions. Additionally, I question the claims of generalization in the paper - but of course, the method would generalize across visual observation perturbations. It is not conditioned on these perturbed inputs.  
2. **Unclear why this works:** I suspect the method is able to solve tasks due to the demonstrations available to it - not due to the online PPO. This is clear in the ablations also - for tasks that actually have a transition function like Car Racing, the demonstrations yield appreciable improvements over not using them. 
3. **Lack of Novelty:** It’s unclear to me whether this paper adds to existing knowledge in any way. Multiarmed Bandits are well studied and memory based agents are well studied also. The paper currently does not present any new theoretical insights, nor does it show massively scaled experimentation.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 3