Gaze on the Prize: Shaping Visual Attention with Return-Guided Contrastive Learning

## Abstract

Abstract Visual Reinforcement Learning (RL) agents must learn to act based on high-dimensional image data where only a small fraction of the pixels is task-relevant. This forces agents to waste exploration and computational resources on irrelevant features, leading to sample-inefficient and unstable learning. To address this, inspired by human visual foveation, we introduce Gaze on the Prize . This framework augments visual RL with a learnable foveal attention mechanism ( Gaze ), guided by a self-supervised signal derived from the agent’s experience pursuing higher returns (the Prize ). Our key insight is that return differences reveal what matters most: If two similar representations produce different outcomes, their distinguishing features are likely task-relevant, and the gaze should focus on them accordingly. This is realized through return-guided contrastive learning that trains the attention to distinguish between the features relevant to success and failure. We group similar visual representations into positives and negatives based on their return differences and use the resulting labels to construct contrastive triplets. These triplets provide the training signal that teaches the attention mechanism to produce distinguishable representations for states associated with different outcomes. Our method achieves up to 2.4 × \times improvement in sample efficiency and can solve tasks that the baseline fails to learn, demonstrated across a suite of manipulation tasks from the ManiSkill3 benchmark, all without modifying the underlying algorithm or hyperparameters.

## 1 Introduction

One of the characteristics of intelligent behavior is the ability to perceive a complex visual world while focusing only on what matters itti2001computational. Through a combination of gaze and foveation, humans naturally discount irrelevant visual information, focusing only on the small subset of cues needed to complete a task hayhoe2005eye; land2009vision. For artificial agents (e.g., robots) however, replicating this natural human ability of selective, task-directed attention remains a fundamental challenge. Modern Reinforcement Learning (RL) has achieved great success when provided with structured state representations or carefully engineered features mnih2015human; silver2016mastering. However, when learning directly from raw pixel inputs, these same algorithms struggle with sample efficiency and robustness. The core issues arise from high-dimensional visual observations that contain vast amounts of task-irrelevant information. Unlike recent studies in supervised learning, where human demonstration can guide where to look, either through active vision chuang2025active; xiong2025vision or human gaze supervision chuang2025look, RL agents must learn to distinguish relevant and irrelevant features from pixels through their own trial and error. The result is a sample-inefficient process, in which high-dimensional inputs make it difficult to identify the visual cues critical to task success, and cause agents to waste extensive exploration and computational resources.

To address this, we propose Gaze on the Prize, a framework that learns visual attention based on simple, yet powerful insights: 1) at any instant of time, task relevant visual cues are localized in one or more contiguous regions within the robot’s field of view, 2) when similar states lead to different returns, their distinguishing features are likely to contain task-relevant information jonschkowski2015learning; blakeman2020selective; kim2024make. In our approach, the RL agent leverages gaze, a visual attention mechanism to identify task-relevant features, trained via a contrastive signal derived from the agent’s reward. For this purpose, we first isolate visual inputs that are similar in the feature space but lead to different reward outcomes. By contrasting such inputs, the attention mechanism can be guided to discover task-relevant features. Our method is designed to be compatible with any visual RL algorithm and augment their performance. While it adds an auxiliary contrastive objective that affects both attention and policy/value learning, it preserves the core structure and hyperparameters of the base RL algorithms.

Our visual attention provides a lightweight focus mechanism using just five parameters to represent gaze as a Gaussian region, offering human-like inductive bias and also improving the explainability of the agent’s actions.

Our contributions can be summarized as:

- 1.
Learnable Foveal Attention for RL: We adapt a parametric attention model from human gaze research to visual RL. This design provides an inductive bias that is well-suited for manipulation tasks and provides explainable insights into the agent’s actions.
- 2.
Return-guided Attention Learning: We devise a contrastive learning method that shapes attention patterns by comparing how an agent’s gaze influences the task outcome. Using a triplet loss, the attention mechanism learns to focus on image regions that distinguish success from failure.
- 3.
Plug-in Compatibility: Our approach serves as a plug-in enhancement that is agnostic to the underlying RL framework. It adds a gaze module to improve visual RL algorithms while preserving their logic and structure.
- 4.
Experimental Evaluation: We demonstrate the utility of the proposed method on RL schemes such as off-policy SAC and on-policy PPO.

## 2 Related Works

### 2.1 Gaze in Computer Vision and Robotics

There is growing interest in leveraging human gaze and gaze-inspired mechanisms in computer vision and robotics. Many works focus on predicting dense saliency maps jiang2015salicon; kummerer2016deepgaze; kummerer2022deepgaze; liu2020picanet, which identify image regions likely to attract human gaze. Complementary research investigates how saliency maps can enhance downstream performance across a wide range of vision applications sugano2016seeing; liang2024visarl; crum2025mentor. Other approaches explore parametric attention models that compactly represent gaze and saliency maps via Gaussian or Gaussian mixture models song2024learning; reddy2020tidying.

In robotics, gaze has been studied in both imitation learning (IL) and reinforcement learning (RL). Early IL approaches used Gaussian mixture models to estimate human gaze, foveating a robot’s vision by cropping its input around the predicted gaze kim2020using. A more recent work leverages foveated vision transformers, allocating high-resolution patches near the gaze and coarser patches toward the periphery chuang2025look. In RL, VisaRL liang2024visarl pretrains a vision encoder using human-annotated saliency maps, improving downstream performance for robot control. Eye-Robot kerr2025eye trains an RL agent to control a mechanical eyeball, with rewards guided by the behavior cloning loss of a co-trained IL policy. In contrast to these approaches, we explore how gaze and attention can emerge naturally within standard visual RL frameworks using contrastive learning, without relying on explicit gaze supervision or specialized hardware.

### 2.2 Contrastive Learning in RL

Contrastive learning has emerged as a powerful approach for representation learning in RL. CURL improves sample efficiency in RL by applying an instance-level contrastive loss to augmented image observations alongside the standard training objective laskin2020curl. M-CURL extends CURL by adding a masked contrastive objective over video frame sequences, using a transformer to reconstruct features that match the ground truth zhu2022masked. eysenbach2022contrastive reinterprets goal-conditioned RL as a contrastive learning problem, aligning state-action embeddings to approximate a goal-conditioned value function. TACO introduces a temporal contrastive objective to align state-action features with future state representations zheng2023taco. Most closely related, liu2021return uses reward returns to define a contrastive loss, aligning state-action pairs with similar returns. Our work complements and extends this idea by investigating how return-guided contrastive learning can instead shape visual attention, improving both performance and explainability.

## 3 Gaze on the Prize

### 3.1 Overview

The goal of Gaze on the Prize is to enhance visual RL by constraining visual representation learning to a specific region(s) within an image. This is achieved through an attention mechanism trained with return-guided contrastive learning. Our framework is designed as a versatile plugin, the main additions being a simple gaze module and an auxiliary contrastive loss that enhances the base RL algorithm. It is fully compatible with any vision-based RL architecture including those with standard CNN-based vision encoders.

The method consists of four components:
(1) a learnable foveal attention formulated as a 2D Gaussian function,
(2) a contrastive buffer storing CNN features and task returns,
(3) a triplet mining procedure to identify similar features that lead to different outcomes, and
(4) a return-guided contrastive loss that trains the representation to better distinguish similar features with different returns.

Figure: Figure 1: a) A CNN backbone encodes observations into feature maps. Instead of passing them directly to the RL algorithm (baseline), our method refines them with a gaze module that predicts Gaussian attention weights parameterized by $\mu_{x},\mu_{y},\sigma_{x},\sigma_{y},\sigma_{xy}$. Multiplying features by these weights ($\odot$) creates a human-like, foveated representation for the RL algorithm. b) During training, we store CNN features and returns in a buffer. Triplet mining groups together similar features that yield different returns. c) The attention is applied on each triplet and a contrastive loss on cosine distances (anchor $z_{a}$, positive $z_{+}$, negative $z_{-}$) guides the module to adjust its attention to better distinguish features by reward.
Refer to caption: x1.png

### 3.2 Gaze Module

The gaze module is a component that attaches to a standard visual backbone (e.g., NatureCNN mnih2015human) without modifying its architecture (see Figure [1](https://arxiv.org/html/2510.08442v1#S3.F1)). It generates spatial 2D attention weights by processing the backbone’s final feature maps, allowing extraction of weighted information from visual inputs. Inspired by exponential modeling of human foveation lehky2011population; roth2023objects; li2025adgaze, we implement a parametric attention mechanism modeled as an anisotropic Gaussian distribution, which we call foveal attention. This introduces a strong inductive bias in RL agent training. The foveal attention mechanism is parameterized by position ($\mu_{x}$, $\mu_{y}$) and covariance ($\sigma_{x}$, $\sigma_{y}$, $\sigma_{xy}$) (see Figure [1](https://arxiv.org/html/2510.08442v1#S3.F1)). This foveal attention rescales the backbone’s feature map element-wise.

This design offers some key advantages: (1) strong inductive bias that we hypothesize aligns with manipulation tasks and constrains the solution space for a more data-efficient learning process, (2) explainable visualizations of the cues driving the agent’s actions, and (3) a lightweight module that adds minimal overhead to existing architectures.

### 3.3 Contrastive Attention Learning

Consider a visual observation $o$, and its spatial feature map $f=\text{Enc}(o)$. We assume that $f$ comprises of task-irrelevant features $f_{\text{irr}}$ (e.g., background) and task-relevant features $f_{\text{rel}}$ (e.g., objects of interest). An effective RL policy should be primarily driven by $f_{\text{rel}}$. Hence, Gaze on the Prize aims to help the RL agent in more effectively extracting $f_{\text{rel}}$ via an attention mechanism $A_{\theta}$ with parameters $\theta$. In other words, we aim to find $\theta$ such that $z=f\odot A_{\theta}(f)\approx f_{\text{rel}}$, where $\odot$ is element-wise multiplication.

The key insight is that when similar visual states lead to different returns, their distinguishing features are likely task-relevant. We formalize this through a contrastive learning framework that shapes attention to highlight these discriminative features.

#### 3.3.1 Contrastive Attention Loss

Given an anchor observation $o_{a}$, we identify its $k$-nearest neighbors in the feature space and partition them into two groups based on their returns. We partition them into high-return (positive) and low-return (negative) groups, using the 50th percentile ($\tilde{R}$) as the dividing threshold with an adaptive delta ($\Delta$) based on return spread. States with returns above $\tilde{R}+\Delta$ become positive samples $\mathcal{P}(o_{a})$, while those below $\tilde{R}-\Delta$ become negative samples $\mathcal{N}(o_{a})$.

For triplets $(o_{a},o_{+}\in\mathcal{P}(o_{a}),o_{-}\in\mathcal{N}(o_{a}))$, the contrastive loss can be formulated as:

$$ $\mathcal{L}_{\text{con}}(\theta)=\mathbb{E}_{(o_{a},o_{+},o_{-})}\left[\max(0,D(z_{a},z_{+})-D(z_{a},z_{-})+\alpha)\right]$ (1) $$

where $D(\cdot,\cdot)=(1-\text{cosine similarity})$ computed on L2-normalized features, and $\alpha$ is a triplet margin. In practice, we approximate this expectation using batches of $N$ triplets:

$$ $\hat{\mathcal{L}}_{\text{con}}=\frac{1}{N}\sum_{i=1}^{N}\max(0,D(z_{a_{i}},z_{+_{i}})-D(z_{a_{i}},z_{-_{i}})+\alpha)$ (2) $$

When attention highlights regions that fail to separate anchor-positive from anchor-negative pairs (i.e., when $D(z_{a},z_{+})\approx D(z_{a},z_{-})$), the triplet margin $\alpha$ ensures the loss remains positive, generating gradients that push attention parameters toward different spatial locations. Conversely, when attention successfully focuses on discriminative regions ($D(z_{a},z_{-})-D(z_{a},z_{+})>\alpha$), the loss reaches zero, stabilizing the current attention parameters. Through optimization over many such triplets, attention converges on features that reliably distinguish different outcome levels.

For Gaussian attention mechanisms like our foveal attention (Section [3.2](https://arxiv.org/html/2510.08442v1#S3.SS2)), we add a regularization term to prevent attention from collapsing to a single point or overly diffusing its coverage:

$$ $\mathcal{L}_{\text{spread}}=\sum_{i\in\{x,y\}}(\log(\sigma_{i})-\log(\sigma_{i}^{\text{target}}))^{2}$ (3) $$

This log-space formulation regularizes the foveal spread and improves training stability. The complete attention learning objective combines the contrastive loss with regularization:

$$ $\mathcal{L}_{\text{attn}}=\hat{\mathcal{L}}_{\text{con}}+\lambda_{\text{spread}}\mathcal{L}_{\text{spread}}$ (4) $$

This attention loss is then integrated with the base RL algorithm’s objective:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{RL}}+\lambda_{\text{attn}}\mathcal{L}_{\text{attn}}$ (5) $$

where $\lambda_{\text{attn}}$ and $\lambda_{\text{spread}}$ are each hyperparameters controlling the relative importance of attention learning and spread regularization.

#### 3.3.2 Contrastive Buffer and Return-guided Triplet Mining

The contrastive loss (Equation [2](https://arxiv.org/html/2510.08442v1#S3.E2)) requires triplets of similar features with different returns. We first describe our buffer for storing vision features, then detail how we mine the triplets from this buffer.

During training, we maintain a buffer (separate from the on-policy batch or off-policy replay buffer) storing three elements from each observation: (1) detached feature maps from the vision backbone’s final layer, (2) flattened feature embeddings for efficient similarity search, and (3) associated episode returns. This circular buffer stores diverse historical visual features rather than just recent features. The vision features are stored detached from the computation graph for gradient isolation, preventing the contrastive loss from participating in vision backbone training. During training, stored features pass through the current attention head, creating gradient paths that update attention while preserving the learned vision representations. This design also reduces computational cost and memory requirements compared to backpropagating through the full network.

We implement triplet mining using FAISS douze2024faiss; johnson2019billion for efficient $k$-nearest neighbor search. FAISS enables sub-linear search complexity even with large search space, making our approach practical for long training runs. During each mining iteration, we sample anchor features from the buffer and retrieve their $k$-nearest neighbors based on cosine similarity of L2-normalized features. These neighbors form candidate pools for triplet construction. We partition them into positive (high-return) and negative (low-return) groups and randomly sample one from each group to pair with the anchor, forming triplets for contrastive learning.

## 4 Experiments

We design our experiments to validate two core contributions: (1) whether parametric foveal attention improves visual RL performance, and (2) whether return-guided contrastive learning enhances this attention mechanism. Specifically, we investigate the following research questions:

RQ1: How do different attention mechanisms affect visual RL performance?
We compare baseline CNN, patch attention (using 1x1 convolution to generate per-patch attention weights), and our foveal attention on on-policy PPO to understand the impact of attention structure on manipulation tasks.

RQ2: Does return-guided contrastive learning enhance attention?
We evaluate foveal attention with and without contrastive learning to isolate the contribution of our contrastive learning.

RQ3: Does contrastive learning improve robustness to visual clutter?
We train RL agents in cluttered environments to test whether our attention can learn to filter irrelevant visual information despite distractions.

RQ4: Is our approach applicable across different RL algorithms?
We validate our complete framework with off-policy SAC to demonstrate compatibility beyond on-policy methods.

### 4.1 Experimental Setup

We evaluate our approach on seven robotic manipulation tasks from the ManiSkill3 benchmark taomaniskill3. The chosen tasks require diverse manipulation skills ranging from simple pushing to object reorientation, providing a comprehensive test of our attention learning approach. For a subset of the tasks, we additionally train and evaluate on variants with random visual clutter to assess whether contrastive attention learns to focus on task-relevant features despite the clutter. Details of each task are provided in Appendix [A](https://arxiv.org/html/2510.08442v1#A1).

### 4.2 Implementation details

We build on widely used PPO schulman2017proximal and SAC haarnoja2018soft implementations provided in ManiSkill3 and CleanRL huang2022cleanrl. For baselines, we either use the recommended hyperparameters from the benchmark or lightly tuned variants to ensure stable training. We do not perform heavy task-specific tuning, since our goal is to test whether the foveal attention and return-guided contrastive learning provide consistent improvements to the base RL algorithm *without* altering the base RL training behavior. This ensures that our comparisons reflect the effect of our method rather than differences in baseline optimization. For our foveal attention and return-guided contrastive components, we adopt a single set of default hyperparameters across all tasks: the number of anchors is 1024, the top $k$=16 nearest neighbors are used for sampling positives and negatives, the contrastive buffer holds 100,000 samples, the contrastive loss weight of $\lambda_{\text{attn}}$=0.1, and the contrastive objective is applied every iteration. We emphasize that these values were chosen as reasonable defaults and may not be individually optimal for each task, though we include ablation on key hyperparameters to examine their influence on performance. A full hyperparameter table for both RL algorithms and our contrastive components is provided in Appendix [B](https://arxiv.org/html/2510.08442v1#A2).

## 5 Results

We evaluate our method on a suite of robotic manipulation tasks from ManiSkill3, organized around the research questions in Section [4](https://arxiv.org/html/2510.08442v1#S4). Experimental results are averaged over three seeds, with shaded regions in learning curves indicating standard error across runs.

Figure: Figure 2: PPO results on ManiSkill3 environments. Top: Comparison of attention architectures without contrastive learning (RQ1), showing baseline CNN, patch attention, and foveal attention. Center: Effect of return-guided contrastive learning on foveal attention performance (RQ2), comparing foveal attention with and without contrastive supervision. Bottom: Comparison of foveal attention with and without contrastive learning when trained in the presence of visual clutter (RQ3), demonstrating the robustness benefits of return-guided supervision.
Refer to caption: x2.png

### 5.1 How do different attention mechanisms affect visual RL performance? (RQ1)

We evaluate PPO agents on six ManiSkill3 tasks, comparing three architectural variants: (1) a baseline without attention, (2) patch attention, and (3) our foveal attention, all without contrastive learning. This comparison is to isolate the effect of attention architecture independent of the contrastive learning objective. As shown in the top row of Figure [2](https://arxiv.org/html/2510.08442v1#S5.F2), both attention mechanisms increase sample efficiency (measured by steps to 50% success rate) in five out of six tasks compared to the baseline without attention. Additionally, we observe that our foveal attention consistently reaches the 50% success rate faster than patch attention across all tasks. These findings suggest that while spatial attention generally adds value when training visual RL, the architectural choice matters. We observe that structured focus provides a better inductive bias for object-centric manipulation, which usually requires focused localization throughout the task. Interestingly, we noticed that on PushT task, patch attention underperforms the baseline, suggesting that unconstrained weights can deteriorate training. Without structural constraints, attention may focus on misleading features or shift too rapidly during training which leads to unstable training. On the other hand, our foveal attention appears to provide essential regularization, preventing these failure modes while maintaining flexibility to focus on task-relevant regions. See Figure [3](https://arxiv.org/html/2510.08442v1#S5.F3) for the visualization of the two attention variants on an example task.

Figure: Figure 3: Attention visualization on PlaceSphere task across training steps. Left: patch attention produces scattered focus across multiple regions. Right: foveal attention maintains consistent, concentrated focus.
Refer to caption: figures/patch_initial.png

### 5.2 Does return-guided contrastive learning enhance attention? (RQ2)

We now add our full method (foveal attention + contrastive learning) to the comparison. The result shown at the center row of Figure [2](https://arxiv.org/html/2510.08442v1#S5.F2) exhibits two modes of improvement depending on task difficulty. For simple tasks (e.g., PushCube and PickCube), the performance benefits of added contrastive learning are marginal. However, for challenging tasks, contrastive learning provides stronger impact, where for PushT, contrastive learning provides a 1.48$\times$ improvement in sample efficiency to reach 50% success, and for LiftPegUpright, only the contrastive variant reaches 50% success within reasonable steps. Notably, PokeCube shows the highest improvement, with 2.4$\times$ better sample efficiency compared to the baseline.

### 5.3 Does contrastive learning improve robustness to visual clutter? (RQ3)

To further investigate whether contrastive learning helps the attention mechanism to filter irrelevant visual information, we train and evaluate on cluttered variants of a subset of the tasks. As shown in the bottom row of Figure [2](https://arxiv.org/html/2510.08442v1#S5.F2), the performance gap is more apparent. For example, while foveal attention without contrastive learning is unable to solve the PushTClutter task, even underperforming the baseline, contrastive learning provides the necessary guidance to find critical cues from the cluttered environment. Also for PokeCubeClutter, the foveal attention with contrastive learning reaches a robust performance of  90% success rate, while without contrastive learning, the success rate plateaus below 80%.

Across both clean (RQ2) and cluttered (RQ3) environments, we observe that return-guided contrastive learning provides more value as visual complexity grows. Whether the complexity comes from task difficulty (challenging manipulation requiring fine discrimination) or environmental factors (visual clutter), the contrastive objective helps attention focus on genuinely task-relevant features. This demonstrates that our method enhances performance by learning robust attention patterns that generalize across different sources of visual complexity.

### 5.4 Is our approach applicable across different RL algorithms? (RQ4)

Figure: Figure 4: SAC results on ManiSkill3 environments (RQ4). Foveal attention with contrastive learning consistently improves off-policy learning across five manipulation tasks, demonstrating the generalizability of our approach beyond on-policy methods.
Refer to caption: x5.png

To test whether the benefits of return-guided contrastive attention extend beyond on-policy PPO, we evaluate our method with off-policy SAC (Soft-Actor-Critic) haarnoja2018soft on five Maniskill3 tasks. As shown in Figure [4](https://arxiv.org/html/2510.08442v1#S5.F4), we observe improvements over the baseline, either faster convergence or higher final success rates. The trend is similar to that of PPO, demonstrating that our approach is not tied to a single RL algorithm, but can be applied to different RL methods without heavy modifications.

### 5.5 Ablations

We conduct ablations on the PushT task under PPO, as this setting shows the largest gap between the foveal attention and our full method with contrastive learning. This makes it the most informative testbed for isolating the effects of each design choice of our method. We focus our ablation on three core hyperparameters that characterize the general contrastive learning pipeline: (1) buffer size determines the data regime for contrastive learning, (2) loss weight $\lambda_{\text{attn}}$ controls the balance between auxiliary and the main RL objective, and (3) update frequency affects computational cost. These represent fundamental design decisions that apply across any vision-based RL task, rather than task-specific tuning parameters. Figure [5](https://arxiv.org/html/2510.08442v1#S5.F5) illustrate how these hyperparameters affect performance, showing the method’s robustness to these critical implementation choices.

Figure: Figure 5: Ablation studies on PushT examining key hyperparameters and contrastive learning contribution. Left: Buffer size comparison including no-contrastive baseline. Center: Contrastive loss weight $\lambda_{\text{attn}}$. Right: Update frequency. Results demonstrate robustness to hyperparameter choices within reasonable ranges.
Refer to caption: x6.png

Buffer size: Contrastive buffer size is a notable design choice of our method, as PPO typically uses only the most recent rollout data unlike off-policy methods. Prior work has shown that augmenting on-policy algorithms with replay buffers can improve performance wang2016sample; novati2019remember, motivating our use of a buffer for contrastive learning. Interestingly, using only on-policy batch ($\approx$20k) performs comparably to the larger 100k buffer, suggesting that our method adapts effectively to different data availability constraints without requiring extensive historical samples in the buffer. However, we noticed that excessively large buffers (200k, 10x on-policy batch) can slow early learning, likely due to mining triplets from stale representations that no longer reflect the current features, highlighting the importance of maintaining a reasonable buffer size that provides sufficient data for contrastive learning without introducing staleness issues.

Contrastive Loss Weight: Balancing auxiliary objectives with the main RL objective typically requires careful tuning to prevent training instability jaderberg2016reinforcement. Our method exhibits minimal sensitivity to the contrastive loss weight ($\lambda_{\text{attn}}$) with all values from 0.05 to 0.25 achieving similar convergence trajectories and final performance. We observe that higher weights (0.25 vs 0.1) produce steeper learning curves, which suggest stronger contrastive supervision accelerates convergence. However, excessive weights can be detrimental, as can be seen with $\lambda_{\text{attn}}$=0.5, suggesting that beyond a certain threshold, the contrastive objective interferes with policy learning. The trade-offs within the acceptable range are minimal compared to the gain over the baseline without any contrastive objective, which indicates that the return-guided signal is inherently beneficial regardless of precise weighting within a reasonable range. This finding makes the method practical for deployment without excessive tuning.

**Table 1: Throughput reduction and performance trade-offs. Steps Per Second (SPS) is measured during training on PushT task using a single NVIDIA RTX 3090 GPU, representing wall-clock throughput. Sample efficiency is measured as the ratio of steps required to reach 50% success rate (SR) compared to baseline. The values are averaged across three seeds.**
| Config | SPS | Throughput | Sample Efficiency | Wall-time to 50% SR |
| --- | --- | --- | --- | --- |
|  |  | Reduction | (higher is better) | (lower is better) |
| Baseline (CNN) | 3693.8 | - | - | - |
| Foveal Attention | 3272.8 | 11% | 1.15$\times$ | 0.98$\times$ |
| + Contrastive (every 8) | 2991.8 | 19% | 0.97$\times$ | 1.23$\times$ |
| + Contrastive (every 4) | 2715.6 | 27% | 1.58$\times$ | 0.87$\times$ |
| + Contrastive (every 2) | 2478.6 | 33% | 1.76$\times$ | 0.85$\times$ |
| + Contrastive (every 1) | 2335.9 | 37% | 1.71$\times$ | 0.93$\times$ |

Contrastive Frequency
Since triplet mining and contrastive learning add computational overhead, we analyze different update frequencies jaderberg2016reinforcement to determine whether less frequent updates maintain performance benefits. The contrastive frequency plot shows a non-monotonic relationship where the contrastive update of every 2 iterations (yellow) provides the best trade-off, while update every 8 iterations (red) underperforms the baseline despite the added overhead. This suggests that the attention may deviate from task-relevant patterns if they are not updated frequently enough, emphasizing the necessity of providing consistent corrective signals during training.

The throughput reduction and performance trade-offs are organized in Table [1](https://arxiv.org/html/2510.08442v1#S5.T1). The results show that our approach not only improves sample efficiency but also achieves faster wall-time convergence despite the added computational overhead. This, along with other results presented in this work demonstrates that the computational investment in learning visual attention is well-justified, leading to practical speedups for visual RL training.

## 6 Limitations and Future Work

A key limitation of our current approach is the reliance on dense reward signals. Our return-guided contrastive learning requires sufficient return variation to generate informative training signals, but in sparse reward settings where most trajectories yield identical returns, lacks informative supervision for our contrastive learning. Future work could explore additional auxiliary signals like value estimates or curiosity rewards to provide supervision when explicit rewards are scarce burda2018exploration; pathak2017curiosity; badia2020never. Additionally, studies show human visual attention naturally incorporates temporal dynamics through saccades and fixations land1999roles. In contrast, our single feature based contrastive loss cannot capture motion patterns or sequential dependencies that may be critical for some complex tasks. Extending our work to address scenarios where task-relevant information spans multiple timesteps might be an exciting future direction, better mirroring the dynamic nature of biological vision systems.

## 7 Conclusion

In this work, we introduced Gaze on the Prize, a framework that guides attention to focus on task-relevant visual features in RL through return-guided contrastive learning. By contrasting similar states with different outcomes, our method guides attention toward the features that matter for task success. Experiments across multiple robotic manipulation tasks and validation on two RL paradigms (on-policy and off-policy RL) show that our method can enhance standard visual RL with learned attention, improving sample efficiency and performance without requiring architectural changes to the base RL algorithms. To enable visual RL agents to tackle complex, cluttered manipulation tasks, we need methods that can discover task-relevant visual cues without human supervision. We believe this work is a step toward more sample-efficient visual RL, where agents learn not only what to do, but also where to look and focus, similar to the gaze mechanism that makes human vision so effective.

## References

- [1]
Laurent Itti and Christof Koch.
Computational modelling of visual attention.
Nature reviews neuroscience, 2(3):194–203, 2001.
- [2]
Mary Hayhoe and Dana Ballard.
Eye movements in natural behavior.
Trends in cognitive sciences, 9(4):188–194, 2005.
- [3]
Michael F Land.
Vision, eye movements, and natural behavior.
Visual neuroscience, 26(1):51–62, 2009.
- [4]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.
Human-level control through deep reinforcement learning.
nature, 518(7540):529–533, 2015.
- [5]
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
nature, 529(7587):484–489, 2016.
- [6]
Ian Chuang, Andrew Lee, Dechen Gao, M-Mahdi Naddaf-Sh, and Iman Soltani.
Active vision might be all you need: Exploring active vision in bimanual robotic manipulation.
In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 7952–7959. IEEE, 2025.
- [7]
Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jeannette Bohg, and Shuran Song.
Vision in action: Learning active perception from human demonstrations.
arXiv preprint arXiv:2506.15666, 2025.
- [8]
Ian Chuang, Andrew Lee, Dechen Gao, Jinyu Zou, and Iman Soltani.
Look, focus, act: Efficient and robust robot learning via human gaze and foveated vision transformers.
arXiv preprint arXiv:2507.15833, 2025.
- [9]
Rico Jonschkowski and Oliver Brock.
Learning state representations with robotic priors.
Autonomous Robots, 39(3):407–428, 2015.
- [10]
Sam Blakeman and Denis Mareschal.
Selective particle attention: Visual feature-based attention in deep reinforcement learning.
arXiv preprint arXiv:2008.11491, 2020.
- [11]
Kyungmin Kim, JB Lanier, Pierre Baldi, Charless Fowlkes, and Roy Fox.
Make the pertinent salient: Task-relevant reconstruction for visual control with distractions.
arXiv preprint arXiv:2410.09972, 2024.
- [12]
Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao.
Salicon: Saliency in context.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1072–1080, 2015.
- [13]
Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge.
Deepgaze ii: Reading fixations from deep features trained on object recognition.
arXiv preprint arXiv:1610.01563, 2016.
- [14]
Matthias Kümmerer, Matthias Bethge, and Thomas SA Wallis.
Deepgaze iii: Modeling free-viewing human scanpaths with deep learning.
Journal of Vision, 22(5):7–7, 2022.
- [15]
Nian Liu, Junwei Han, and Ming-Hsuan Yang.
Picanet: Pixel-wise contextual attention learning for accurate saliency detection.
IEEE Transactions on Image Processing, 29:6438–6451, 2020.
- [16]
Yusuke Sugano and Andreas Bulling.
Seeing with humans: Gaze-assisted neural image captioning.
arXiv preprint arXiv:1608.05203, 2016.
- [17]
Anthony Liang, Jesse Thomason, and Erdem Bıyık.
Visarl: Visual reinforcement learning guided by human saliency.
In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2907–2912. IEEE, 2024.
- [18]
Colton R Crum and Adam Czajka.
Mentor: Human perception-guided pretraining for increased generalization.
In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7470–7479. IEEE, 2025.
- [19]
Peipei Song, Jing Zhang, Piotr Koniusz, and Nick Barnes.
Learning gaussian representation for eye fixation prediction.
arXiv preprint arXiv:2403.14821, 2024.
- [20]
Navyasri Reddy, Samyak Jain, Pradeep Yarlagadda, and Vineet Gandhi.
Tidying deep saliency prediction architectures.
In 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 10241–10247. IEEE, 2020.
- [21]
Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi.
Using human gaze to improve robustness against irrelevant objects in robot manipulation tasks.
IEEE Robotics and Automation Letters, 5(3):4415–4422, 2020.
- [22]
Justin Kerr, Kush Hari, Ethan Weber, Chung Min Kim, Brent Yi, Tyler Bonnen, Ken Goldberg, and Angjoo Kanazawa.
Eye, robot: Learning to look to act with a bc-rl perception-action loop.
arXiv preprint arXiv:2506.10968, 2025.
- [23]
Michael Laskin, Aravind Srinivas, and Pieter Abbeel.
Curl: Contrastive unsupervised representations for reinforcement learning.
In International conference on machine learning, pages 5639–5650. PMLR, 2020.
- [24]
Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Tie-Yan Liu, and Houqiang Li.
Masked contrastive representation learning for reinforcement learning.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3421–3433, 2022.
- [25]
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov.
Contrastive learning as goal-conditioned reinforcement learning.
Advances in Neural Information Processing Systems, 35:35603–35620, 2022.
- [26]
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang.
$\texttt{TACO}$: Temporal latent action-driven contrastive loss for visual reinforcement learning.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- [27]
Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Jian Li, Nenghai Yu, and Tie-Yan Liu.
Return-based contrastive representation learning for reinforcement learning.
arXiv preprint arXiv:2102.10960, 2021.
- [28]
Sidney R Lehky and Anne B Sereno.
Population coding of visual space: modeling.
Frontiers in computational neuroscience, 4:155, 2011.
- [29]
Nicolas Roth, Martin Rolfs, Olaf Hellwich, and Klaus Obermayer.
Objects guide human gaze behavior in dynamic real-world scenes.
PLOS Computational Biology, 19(10):e1011512, 2023.
- [30]
Duantengchuan Li, Shutong Wang, Wanli Zhao, Lingyun Kang, Liangshan Dong, Jiazhang Wang, and Xiaoguang Wang.
Adgaze: Anisotropic gaussian label distribution learning for fine-grained gaze estimation.
Pattern Recognition, 164:111536, 2025.
- [31]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.
The faiss library.
2024.
- [32]
Jeff Johnson, Matthijs Douze, and Hervé Jégou.
Billion-scale similarity search with GPUs.
IEEE Transactions on Big Data, 7(3):535–547, 2019.
- [33]
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su.
Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai.
Robotics: Science and Systems, 2025.
- [34]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.
- [35]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
In International conference on machine learning, pages 1861–1870. Pmlr, 2018.
- [36]
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo.
Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.
Journal of Machine Learning Research, 23(274):1–18, 2022.
- [37]
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando De Freitas.
Sample efficient actor-critic with experience replay.
arXiv preprint arXiv:1611.01224, 2016.
- [38]
Guido Novati and Petros Koumoutsakos.
Remember and forget for experience replay.
In International Conference on Machine Learning, pages 4851–4860. PMLR, 2019.
- [39]
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.
Reinforcement learning with unsupervised auxiliary tasks.
arXiv preprint arXiv:1611.05397, 2016.
- [40]
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.
Exploration by random network distillation.
arXiv preprint arXiv:1810.12894, 2018.
- [41]
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.
Curiosity-driven exploration by self-supervised prediction.
In International conference on machine learning, pages 2778–2787. PMLR, 2017.
- [42]
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al.
Never give up: Learning directed exploration strategies.
arXiv preprint arXiv:2002.06038, 2020.
- [43]
Michael Land, Neil Mennie, and Jennifer Rusted.
The roles of vision and eye movements in the control of activities of daily living.
Perception, 28(11):1311–1328, 1999.

## Appendix

## Appendix A Task/Environment Details

In our experiments, we evaluate seven manipulation tasks from the ManiSkill3 benchmark [33]. All tasks use single-camera RGB observations and are solvable by PPO within reasonable training time, ensuring fair comparison across methods.

### A.1 Task Descriptions

PickCube: Pick up a cube and lift it to a target (target provided as state). Tests basic grasping and lifting.

PushCube: Push a cube to a target location on the table surface. Tests object manipulation without grasping.

PullCube: Pull a cube to a target location on the table surface. Tests object manipulation without grasping.

PokeCube: Poke a cube with a peg to a target location on the table surface. Tests tool use and indirect manipulation.

PushT: Push a T-shaped object to match a target pose. Tests manipulation requiring rotation and translation.

LiftPegUpright: Reorient a peg from horizontal to an upright position. Tests complex reorientation.

PlaceSphere: Pick up a sphere and place it in a slot. Tests precise placement and object handling.

### A.2 Observation and Action Spaces

All tasks share consistent observation and action specifications:

- •
Observation: 128×128 RGB images (64x64 for SAC) from a fixed camera position
- •
Action: 7-DOF continuous control
- •
Episode length: 50-100 steps depending on task

### A.3 Reward Structure

Tasks use dense reward signals combining distance-based rewards for approaching intermediate targets, and success bonuses for achieving goals. This dense reward structure enables meaningful return differences for contrastive learning.

### A.4 Cluttered Environments

Figure: Figure A1: Visualization of cluttered environments for robustness evaluation. Visual distractors include objects similar in shape and color to task-relevant items.
Refer to caption: x7.png

To evaluate robustness to visual clutter (RQ3, Section [5.3](https://arxiv.org/html/2510.08442v1#S5.SS3)), we create cluttered variants of four tasks by adding randomly placed distractor objects to the scene. These cluttered environments are designed to challenge the agent’s visual processing by introducing objects that are visually similar to task-relevant items but functionally irrelevant.

## Appendix B Hyperparameters

The hyperparameter used in experiments are organized in Table [B1](https://arxiv.org/html/2510.08442v1#A2.T1).

**Table B1: Hyperparameters used in experiments. All values are fixed for main experiments across tasks unless marked with *. Most hyperparameters are adopted from the original baseline implementations [33, 36] with modifications for attention and contrastive learning components.**
| HYPERPARAMETER | VALUE | DESCRIPTION |
| --- | --- | --- |
| Gaze Module |  |  |
| Architecture | [64→32→5] | Conv→ReLU→MaxPool→Linear |
| $\sigma_{\text{target}}$ | 0.1 | Target spread in normalized coords |
| $\lambda_{\text{spread}}$ | 0.1 | Spread regularization weight |
| Contrastive Learning |  |  |
| $k$-neighbors | 16 | Neighborhood size |
| Triplet margin ($\alpha$) | 0.5 | Minimum separation distance |
| Contrastive buffer size | 100k | FAISS-indexed buffer |
| Update frequency | 1 | Every N iterations |
| Anchor samples | 1024 | Per triplet mining |
| $\lambda_{\text{attn}}$ | 0.1 | Contrastive loss weight |
| RL Training (PPO) |  |  |
| Learning rate | 3e-4 | Adam optimizer |
| Discount factor ($\gamma$)* | 0.8-0.99 | Task-specific |
| GAE $\lambda$ | 0.9 | Advantage estimation |
| Num environments* | 1024-2048 | Parallel environments |
| Num steps* | 4-16 | Rollout length per environment |
| PPO clip range | 0.2 | Policy update constraint |
| PPO epochs | 8 | Updates per rollout |
| Minibatches | 32 | Gradient updates per epoch |
| Target KL | 0.2 | Early stopping threshold |
| Input resolution | 128×128 | RGB images |
| Control mode | pd_joint_delta_pos | Joint-space control |
| RL Training (SAC) |  |  |
| Learning rate | 3e-4 | Adam optimizer |
| Discount factor ($\gamma$)* | 0.8-0.99 | Task-specific |
| Num environments | 32 | Parallel environments |
| Batch size | 512 | Samples per update |
| Replay buffer size | 300k | Experience storage |
| Update-to-data (utd) | 0.5 | Gradient steps per env step |
| Target update rate ($\tau$) | 0.01 | Soft update coefficient |
| Input resolution | 64×64 | RGB images |
| Control mode | pd_ee_delta_pos | End-effector control |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 4
questions: - How does the method compare to RAD, DrQ-v2, SVEA, and CURL under their recommended settings? Please report both ID performance and ID and OOD gaps, for example you can use the Distracting Control Suite.
- If reward spread is small, could you form positives/negatives using value estimates or TRP targets? Any preliminary results?
- Please provide a compute table (SPS and wall-clock) for baseline CNN, +gaze (no contrast), and full method, including buffer maintenance and mining time, across several tasks.
soundness: 2
strengths: - Simple, interpretable, drop-in attention module with few parameters and human-readable heatmaps.

- Clear optimization split: contrastive loss updates the gaze head; RL gradients train the encoder/policy.

- Works with both off-policy algorithm and off-policy algorithm

- Useful ablations (e.g., auxiliary buffer, update frequency) and some wall-clock considerations.
summary: The paper introduces a lightweight "gaze" head that predicts an anisotropic Gaussian to reweight spatial CNN features, trained with a return-guided triplet loss mined from a separate buffer of detached features, and combined with a standard RL loss (PPO/SAC). On ManiSkill3 manipulation tasks, the method learns faster and shows larger gains under cluttered visuals.
weaknesses: 1. Missing strong pixel-RL baselines. The paper omits standard data-augmentation and contrastive baselines (RAD, DrQ-v2, SVEA, CURL) that are now expected in visual control comparisons. Without these, it is unclear whether the gains stem from the proposed attention or simply from adding any robust representation trick.
2. Insufficient OOD evaluation. Claims about distraction robustness/generalization are not tested on canonical benchmarks with held-out domains and graded shifts (DCS; DMC-GB/GB2). Results limited to ManiSkill3 and custom clutter do not substitute for these protocols. 

3. Sensitivity to reward structure. The return-guided triplet objective relies on reward spread; sparse-reward regimes may weaken positives/negatives. A value- or TRP-based auxiliary could help, but this is not explored. ([ojs.aaai.org][3])
5. The narrative hints at generalization, but the experiments emphasize sample-efficiency on manipulation. Without in-distribution to OOD gap reporting, distraction-intensity curves, or cross-background transfer, the generalization claim is under-supported.
6. Related-work positioning. Attention/masking approaches targeting distraction robustness: MaDi; saliency-guided methods like SGQN and SGFD; segmentation-assisted FTD are not compared and deserve deeper discussion. ([ifaamas.org][4])


References: 
* Laskin, M., et al. (2020). Reinforcement Learning with Augmented Data (RAD). NeurIPS.[1]
* Stone, A., Ramirez, O., & Jonschkowski, R. (2021). The Distracting Control Suite. [2]
* Wang, S., Wu, Z., Hu, X., Wang, J., Lin, Y., & Lv, K. (2024). What Effects the Generalization in Visual RL: Policy Consistency with Truncated Return Prediction (TRP). [3]
* Grooten, B., et al. (2024). MaDi: Learning to Mask Distractions for Generalization in Visual Deep RL [4]
* Laskin, M., Srinivas, A., & Abbeel, P. (2020). CURL: Contrastive Unsupervised Representations for Reinforcement Learning. ICML. [5]
* Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021). Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (DrQ-v2) [6]
* Hansen, N., Su, H., & Wang, X. (2021). SVEA: Stabilizing Deep Q-Learning with ConvNets and ViTs under Data Augmentation. NeurIPS. [7]
* Hansen, N., & Wang, X. (DMC-GB) and Almuzairee, A., et al. (DMC-GB2). DMControl Generalization Benchmarks. [8] 
* Bertoin, D., Zouitine, A., Zouitine, M., & Rachelson, E. (2022). Look where you look! Saliency-guided Q-networks for generalization in visual RL (SGQN). NeurIPS. [9]
* Chen, C., Xu, J., Liao, W., Ding, H., Zhang, Z., Yu, Y., & Zhao, R. (2024). Focus-Then-Decide: Segmentation-Assisted Reinforcement Learning (FTD). [10]
* Huang, S., Sun, Y., Hu, J., Guo, S., Chen, H., Chang, Y., Sun, L., & Yang, B. (2023). Learning Generalizable Agents via Saliency-Guided Features Decorrelation (SGFD).  [11]
* Tao, S., et al. (2024). ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI. arXiv:2410.00425. [12]

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I'm curious about why the episode return instead of step reward is used to guide the contrastive learning, since one-step reward seems a better indicator for the one-step observation.
2. What will happen if the task involves two objects? For example, if the task is to pick a cube into a box, will the attention mechanism always capture both cube and box, or first focus on cube and then box?
soundness: 3
strengths: 1. The paper introduces a biologically inspired, learnable foveal attention mechanism that enhances both the interpretability and efficiency of visual reinforcement learning. 
2. The paper provides a flexible, plug-and-play framework that can seamlessly integrate with various RL algorithms to consistently improve performance across manipulation tasks.
3. The paper is well-written, presenting a straight-forward idea with conceptual clarity and a logical narrative that is easy to follow.
summary: The paper proposes Gaze on the Prize, a visual reinforcement learning framework that enhances sample efficiency and stability by integrating a learnable foveal attention mechanism inspired by human visual focus. Instead of processing all image features equally, the method guides attention toward task-relevant regions using a self-supervised signal derived from return differences—comparing states that lead to high versus low rewards. Through return-guided contrastive learning, the system learns to emphasize visual features that distinguish success from failure, forming contrastive triplets based on outcome differences. Without changing the base RL algorithms or hyperparameters, this approach yields up to 2.4× higher sample efficiency and enables agents to solve manipulation tasks in the ManiSkill3 benchmark that standard methods fail to learn.
weaknesses: 1. One potential drawback of the proposed method lies in that it assumes that there are a sufficient number of clearly distinguishable positive and negative samples. However, in some difficult or sparse-reward tasks, there can be very few positive samples in the replay buffer, which may affect the performance of the policy.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - Can you explain the choice of positive and negative sampling in your contrastive learning setup (refer to my first comment under weaknesses)?
- how would your method perform when augmenting stronger vision-based RL baselines such as CURL?
soundness: 3
strengths: - The paper is well-written and enjoyable to read
- The proposed approach makes a lot of sense and is sound
- The proposed approach is very simple and can be easily plugged into most vision-based RL frameworks
- The results are promising, the proposed approach seems to yield a consistent improvement over SAC and PPO without strongly impacting wall-clock time, which could have been a concern given the use of retrieval methods such as FAISS
- Ablations illustrate the role of different components and the sensitivity to different hyperparameters that were introduced in this work
summary: The paper proposes an approach to learn a foveated attention mechanism where attention is encouraged to focus on task-relevant information. To achieve this, the authors propose using a Gaussian parametric attention mechanism paired with a contrastive loss where positive and negative samples are mined based on visual similarity and return differences. For that the authors use FAISS to store tuples of observations and returns, and then for each anchor select the positives and negatives based on their corresponding return classification based on a threshold. In simulated experiments, the approach is shown to improve sample efficiency and performance over model-free on-policy and off-policy algoriths such as PPO and SAC.
weaknesses: - On the method' side, one detail does not make full sense to me. I do not understand why the positives are chosen to have the higher returns, the anchor might belong to the low-return set and in that case the latter should be treated as positives.
- The main weakness of this work lies in its evaluation. Specifically, the baselines considered in this work fail to position the contribution well. Let me elaborate. While the proposed framework seems to be quite helpfull when using vanilla PPO or SAC, it would be interesting to see if it has any effect on stronger visual RL methods such as CURL or more recent methods. Without such a baseline, it is unclear whether the proposed framework is even needed. In theory, the proposed method should help no matter what baseline is used, but this needs to be validated and well-studied to understand what level of task complexity is needed for this to become apparent.
- Another missing experiment is to understand the role of this plugin  if the encoders are pretrained either large-scale on images or in a first stage with random data from the target environment
- (optional) what would strengthen the paper a lot and increase its impact is to also test the method on a model-based RL baseline, but I understand this can be time-expensive

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: see comments above.
soundness: 2
strengths: 1. The paper is engaging, presenting an intuitive yet effective idea that is both interesting and easy to follow.

2. The authors conduct extensive experiments, demonstrating the strong performance of the proposed framework and its high flexibility across different algorithms.
summary: The paper introduces a framework for visual reinforcement learning that trains a foveal attention mechanism using triplet data, where two similar feature representations correspond to different returns. The foveal attention module is optimized through return-guided contrastive learning.
weaknesses: The authors may benefit from providing additional implementation details and a more thorough discussion of the framework’s limitations.

1. Clarification on implementation and data mining: I would appreciate a more detailed explanation of Sections 3.3.2 and 4.1–4.2, where the authors discuss return-guided triplet mining and the model implementation. The proposed method appears to rely heavily on datasets that provide dense reward signals and a large number of feature pairs with similar representations but different returns, which may not be readily available in many practical settings. More details on the data-mining process, dataset characteristics, and implementation specifics would help readers better understand the method’s applicability. 
Additionally, the experiments include a baseline model (“Foveal attention” without contrastive loss). It would be helpful to elaborate on this model’s setup to clarify the standalone impact of the contrastive loss component.

2. Extension of the gaze module: The current gaze module focuses on a single focal point, which may limit its generalizability. The authors are encouraged to discuss whether the framework could be extended to handle multiple focal locations. If such an extension is feasible, it would be useful to describe the required adaptations; if not, clarifying the bottlenecks or constraints preventing this would strengthen the discussion.

3. Data availability and robustness: A key limitation, also noted by the authors, concerns the availability of return signals or, more broadly, the scarcity of triplet-mining data. It would be valuable to provide insights into how much effective data is required for training, and how data scarcity might influence performance or stability. This discussion would help readers understand the framework’s robustness under less ideal data conditions.


Decision: Reject