reviews: 
Comments: This paper explores a reward-only approach to RL, where policies are trained using rewards and actions rather than observational data, enabling potential zero-shot transfer across environments with different visual representations. The authors propose a method using LSTM-based temporal history and expert-guided behavior cloning to learn reward-based policies in simple environments. They demonstrate that reward-only policies perform reasonably well compared to observation-based ones, especially in tasks with consistent dynamics. The method is evaluated in popular tasks like Pointmass, Cartpole, and Car Racing, with 2D-to-3D transfer shown in a reconfigured AirSim environment.
Strengths: **S1. Thorough Challenge Analysis**: The authors provide a useful in-depth examination of the difficulties inherent to reward-based policies, such as poor observability, difficulty in estimating value functions, and limited exploration. This highlights the limitations of using a scalar reward signal rather than high-dimensional state information. They emphasize the necessity of dense rewards, showing how the method struggles as dimensionality increases.

**S2. Expert Guidance Ablation.** An ablation study on the behavior cloning loss component shows how important it is, especially in the more complex task like Car Racing.

**S3. Simplicity and Comprehensibility.** The method is straightforward, with minimal components and no additional tunable hyperparameters, making it easy to implement and understand.

**S4. Robustness to Observation Quality**: Once trained, the model is independent of observation quality, allowing it to perform well even with noisy or degraded visual observations.

**S5. Effective Reward Estimation for Transfer.** In Section 4.4, the authors demonstrate the transfer capabilities by estimating rewards directly from images, allowing the reward-based policy to perform well even without access to the true reward at inference. They further show that transferring from a 1D reward signal is easier and more reliable than transferring from high-dimensional observations, requiring minimal data and proving more resilient than state estimation.
Weaknesses: **W1. Numerous Assumptions.** The method makes many assumptions for successful implementation, limiting its adaptability and real-world applicability.

1. **Dense and Accurate Rewards**. The authors state that rewards must be *sufficiently dense* and *give a good value throughout the state space*, yet there is no analysis of how sparse they need to be. This lack of clarity makes it uncertain how sparse or dense rewards can be before training fails. In practice, it likely means that transitions without an informative reward signal contribute minimally to training, making the approach highly sample-inefficient. In realistic settings, feedback is often imperfect or delayed. For example, in robotics, sensors might provide inaccurate readings due to interference or hardware limitations, leading to noisy or inconsistent rewards. Similarly, in environments where rewards are human-generated (like feedback in recommendation systems), subjective or inconsistent responses can introduce noise.
2. **Domain Compatibility**. The source and target environments should share the same transition dynamics, action and observation spaces, and reward structure. This overlooks many practical cases, where slight discrepancies in dynamics or observation structures are the norm. The authors don’t discuss where these conditions might realistically apply, leaving practical feasibility unexplored.
3. **Expert Guidance**. The ablation study shows that expert guidance significantly boosts performance, particularly on complex tasks. However, this dependence undermines the claimed benefits of a “reward-only” approach, as it reintroduces standard RL processes the authors aim to bypass. Moreover, training an observation-based expert policy adds a computational overhead, and since training is done online, inference also needs to be run on the expert model. 

**W2. Loss of Spatial Context.** By relying solely on rewards and actions, the method lacks spatial awareness, which is critical in tasks requiring an understanding of position or orientation. For example, in navigation tasks like maze-solving, an agent without spatial context will struggle to differentiate between distinct but similarly rewarding areas, such as identical-looking corridors or dead ends. In manipulation tasks, the lack of positional feedback leads to incorrect actions, like reaching for objects without adjusting for their relative location. Without such spatial cues, the method is limited to simple tasks where positioning doesn’t play a role. The agent’s ability to keep the car centered on the track in the Car Racing environment hinges on the overly-engineered reward function that promotes this behavior. Relying on such finely-tuned rewards is impractical in real-world applications, where crafting reward functions to this level of precision is rarely feasible.

**W3. Lack of methodological novelty**. The approach lacks innovation, relying on a simple combination of an LSTM and behavior cloning to train PPO. This simplicity offers little advancement over existing techniques and contributes minimally to the field, as both components are well-established in RL. Moreover, the explanation of the method is scarce. Neither the text nor the caption of Figure 3 adequately explains the diagram, making it difficult to interpret. For example, it’s not immediately evident that the ‘options’ in the testing phase of Figure 3 depict the modes where the environment provides the reward and where the reward is learned.

**W4. Weak Experimental Evaluation.**
1. The experiments are limited to simple environments (Pointmass, Cartpole, and Car Racing), with the authors suggesting that more complex environments with higher state dimensions or sparse rewards are "impossible" for this method. This raises questions about practical utility—if the method can't handle harder, more realistic tasks, its applicability remains unclear.
2. The authors only compare their reward-based policy to a regular observation-based policy, which is supposed to serve as an upper bound. However, they include no comparisons with other established techniques whatsoever, offering no context on how their approach measures up to alternative methods in this setting.
3. While evaluation is done over 50 trials, the number of seeds used for training isn’t specified. Without this, it’s hard to assess the robustness of the method.
4. The authors state that *it is essential that some recurrent network is used*, and that that *a single reward/action pair is not sufficient*. However, they provide no analysis or ablation study to clarify the LSTM’s specific contribution. Alternative approaches for maintaining temporal history, like stacking previous rewards and actions, are not explored, leaving it unclear whether the LSTM is genuinely necessary or if simpler methods could achieve comparable results.

**W5. Transfer Limitations**. The authors attempt 2D-to-3D transfer by using the Car Racing and AirSim environments, as shown in Figure 1, considering a scenario where changes only occur in pixel-based observations. This leads them to circumvent the realistic challenges of transfer by manually reimplementing the kinematics to match across environments. This undermines the notion of true zero-shot transfer, as manual alignment is rarely viable in real-world applications and reveals the method’s limited applicability. Furthermore, no quantitative results are provided for the 3D transfer; it’s merely claimed that the policy shows “reasonable driving performance,” leaving the success of this transfer largely unsubstantiated. The authors’ statement that “our focus is not on dealing with the dynamics shift” dismisses the real complexities of 2D-to-3D transfer, where adapting to dynamic differences is unavoidable. This approach fails to demonstrate genuine transfer.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 3