Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation
Authors: Xuening Zhang
Qi Lv
Xiang Deng
Miao Zhang
Xingbo Liu
Liqiang Nie
Venue: ICLR 2026 Poster (2026)
Keywords: robotic manipulation
view transformer
3D perception
dynamic-view perception
Primary area: applications to robotics, autonomy, planning

Abstract: View transformers process multi-view observations to predict actions and have shown impressive performance in robotic manipulation. Existing methods typically extract static visual representations in a view-specific manner, leading to inadequate 3D spatial reasoning ability and a lack of dynamic adaptation. Taking inspiration from how the human brain integrates static and dynamic views to address these challenges, we propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation that jointly reasons from static-view and dynamic-view streams. The static-view stream enhances spatial understanding by aligning features of geometrically consistent keypoints extracted from a pretrained 3D foundation model. The dynamic-view stream achieves adaptive adjustment through position-aware pretraining of an egocentric gaze estimation model, computationally replicating the human cortical dorsal pathway. Subsequently, the complementary view representations of both streams are integrated to determine the final actions, enabling the model to handle spatially-complex and dynamically-changing tasks under language conditions. Empirical evaluations on RLBench, the challenging COLOSSEUM benchmark, and real-world tasks demonstrate that Cortical Policy outperforms state-of-the-art baselines substantially, validating the superiority of dual-stream design for visuomotor control. Our cortex-inspired framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.

Published as a conference paper at ICLR 2026
CORTICAL POLICY: A DUAL-STREAM VIEW TRANSFORMER FOR ROBOTIC MANIPULATION
Xuening Zhang', Qi Lv', Xiang Dengi, Miao Xingboo Liu', Liqiang Nie Haariin Institute of Technology (Shenzhen), Shenzhen, Guangdong 518055, China "Shandong Jianzhu University, Jinan, Shandong 250101, China
ABSTRACT
View transformers process multi-view observations to predict actions and have shown impressive performance in robotic manipulation. Existing methods typ- ically extract static visual representations in a view-specific manner, leading to inadequate 3D spatial reasoning ability and lack of dynamic adaptation. Tak- ing inspiration from how the human brain integrates static and dynamic views address these challenges, we propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation that jointly reasons from static-view and dynamic-view view streams. The static-view stream enhances spatial understand- ing by aligning features of geometrically consistent keypoints extracted from a pretrained 3D foundation model. The dynamic-view stream achieves adaptive adjustment through position-aware pretraining of an egocentric gaze estimation model, computationally replicating the human cortical dorsal pathway. Subse- quently, the complementary view representations of both streams are integrated to determine the final actions, enabling the model to handle spatially-complex tasks under language conditions. Empirical evalua-
and Jynamically-changie tions on RLBench, the challenging COLOSSEUM benchmark, and real-world tasks demonstrate that Cortical Policy outperforms state-of-the-art baselines sub- stantially, validating the superiority of dual-stream design for visuomotor control. Our cortex-inspired framework offers fresh perspective for robotic manipulation and holds potential for broader application in ussoonbbsee robot control.
INTRODUCTION
Enabling robots to handle the uncertainty of unstructured, non-stationary environments remains fundamental challenge for robotic manipulation (Liang et al.) 2024: Chu et al. 2025 Li et al.) 2025a). Critically, this challenge requires coherent scene perception through robust fusion of multi- modal inputs, including vision, language, and proprioception (Lv et al. 2024). To achieve this, view transformers (Goyal al. 2023] (024) provide an efficient solution by leveraging multi-view images to predict actions, showing competitive performance while offering greater scalability than explicit 3D representation-basee approaches.
Leveraging set of static camera views around the robot workspace, previous view transformers typ- ically extract visual representations through naively fusing view-specific 2D information. Despite demonstrated competence in stationary environments (Goyal et al. [023 2024 Zhang et al. 2024] Qian al. 2025), this paradigm fails to model cross-view relationships, hampering 3D spatial under- standing beyond 2D images. More importantly, the static camera configurations lack dynamic-vien view perception essential for numan-like manipulation with unpredictable object displacements (Hallquist et al., 2024). These limitations manifest as two frequent failure modes in robotic manipulation: inadequate spatial reasoning and dynamic adaptation failure. As shown in Fig. (top), when placing an object in between the others, the SOTA view transformer (Goyal et al., 2024) exhibits sig- nificant error, failing to place within the right region. This lack of robustness to 3D scene structure further supported by findings that view transformers like texture, lighting, and table color variations (Qian et al.) 2025). Furthermore, when the target
sensitive environmental disturbances
object is moved during approach, existing methods (Goyal et al. 2023 2024 Qian et al. 2025) fail
Corresponniig author; dengxi ang@h en

---

Published as a conference paper at ICLR 2026
to adjust trajectories as expected, persisting with the segrinally-plannde trajectory until task failure (Fig. [ bottom). These empirical findings underscore that the scene perception provided by current view transformers remains incomplete, which hinders robust manipulation performance.
Expected
Final Statu
"Stack 2 blocks in hetween the bottles"
Right Region
By namics
Expected
Final State
"Stack 2 blocks (scene dynamics)"
Figure 1: Deficiencies of prior view transformers for robotic manipulation. (Top) A task re- quiring the robot to understand the spatial relationships between two bottles before deciding on the placing position. Previous method RVT-2 fails to merge different camera views correctly in 3D, causing wrong block placement. (Bottom) Dynamic adaptation failure during object displacement.
To bridge this gap, we draw inspiration from how the human brain organizes view-based visual guide behavior, and in particular, two cortical pathways for visual processing. The tral pathway utilizes static views for scene understanding, while the dorsal pathway specializes in dynamic-view perception, leveraging real-time visual feedback adjust trajectory (Rossit & McIn- tosh [02T] Chen et al., (025). Translating this cortical principle into computational framework, propose Cortical Policy, dual-stream view transformer for endowing robots with integrated 3D spatial understanding and dynamic adaptation. Our method enhances robotic perception through two separate, complementary pipelines: a static-view stream that encodes enduring environmental structures and a dlynamic-view stream that derives actions from motion cues.
To enhance spatial comprehension, the static-view stream learns 3D-aware features by enforcing cross-view geometric consistency, which is supervised by 3D foundation model. To facilitate adaptive re-planning under task dynamics, the dynamic-view stream extracts action-oriented fea- tures and heatmaps from a pretrained, position-awee aware transformer adapted from an egocentric gaze estimation model. By integrating view representations both streams, Cortical Policy generates actions that are simultaneously geometrically grounded and dynamically adaptive. Extensive evalu- ations on RLBench, COLOSSEUM, and real-world tasks demonstrate that our cortex-inspired pol- icy, with enhanced 3D awareness and adaptive motion control, substantially outperforms state-of- the-art baselines. Specifically, it exhibits superior robustness against environmental perturbations and boosts the interactive capabilities of embodied agent in dynamic physical environments. The main contributions of this work are summarized as follows:
Different from prior view transformers that perform single-stream processing static views, we propose Cortical Policy, a dual-stream view transformer that integrates static and dynamic views for robotic manipulation, mirroring the two human cortical pathways advance visuomotor imitation learning.
Unlike view-independent processing in prior methods, introduce cross-view geometric con- sistency learning objective. This objective leverages a pretrained 3D foundation model (VGGT) align cross-view features shared 3D space, significantly enhancing the spatial reasoning robustness of the static-view stream.
novel dynamic-view stream absent in prior work designed to emulate the human dorsal path- way. This stream extracts action-oriented representations from position-aware, pretrained gaze estimation model, thereby enabling adaptive trajectory adjustment.
2 RELATED WORK
This work extends view transformers for robotic manipulation by enhancing static-view 3D percep- tion and introducing dynamic-view processing. We review the relevant work in this section.
2

---

Published as a conference paper at ICLR 2026
View Transformers for Robotic Manipulation. View transformers have become prevalent chitecture for language-conditonn gregate multi-view visual inputs with language instructions and propriccepiion predict 6-DoF gripper poses, states, and collision indicators. RVT (Goyal et al. 2023) establishes a five-camera paradigm (back, front, top, left, right) to render virtual static views, using view transformer to predict view-specific heatmaps, which are back-projected to 3D to estimate gripper translation; multi-camera features are concatenated to predict the remaining action components. To improve precision, VIHE (Wang et al. 2024) and RVT-2 (Goyal et al. (2024) adopt multi-stage refinement: VIHE iteratively renders virtual in- hand static views, while RV localizes regions of interest with three static views (front, top, right) before predicting poses from refined regions. Recent meth- ods enhance static-view visual representations with visual foundation models (Zhang et al. 2024 Fang et al.] 2025) or 3D multi-view pretraining (Qian et al. 2025). Although these methods have advanced static-view perception, their inherent reliance on pre-defined viewpoints limits the adapt- ability in dynamic scenarios. In contrast, Cortical Policy jointly leverages static and dynamic views for action prediction to overcome this limitation.
manipulation (Guhur et al. 2022 Ma et al. 2024). They ag-
3D Perception in Robotics. To enhance robots' understanding of the physical world, extensive ef- forts have been made to integrate 3D representations into robotic manipulation (James et al. [022]
Goyal al., 2023; Lv et al. 2025). Existing approaches, however, face distinct challenges. Voxel-
based methods (James et al., 2022. Shridhar et al. (023) are computationally expensive. Point cloud methods handle occlusion and sim-to-real transfer well, yet require fine-grained semantic alignment (Zhen et al. 2024: Cui et al. 2025) or use inefficient backbones (Chen et al., (024)) Aulti-perspective projection offers an efficient alternative by projecting point clouds onto virtual orthographic views to generate multi-camera RGB-D images, and has been widely adopted in recent al.) 20256). Unlike existing multi persppppreeeeevvee policies that struggle to capture cross-view rel relations, Cortical Policy addresses this limitation by ex- plicitly modeling inter-view relationships, enhanced by geometric priors from VGGT (Wang et al.] 2025). powerful 3D foundation model whose spatial knowledge remains novel in robotic manipu- lation (Lin et al. [0255] Tang et al. 20255 Abouzeid et al.) (2025). We introduce a novel integration of VGGT within the static-view stream, using its predictions to enforce view-invariant feature learning.
work (Goyal al. 2023 2024 Fang et al. 2025; Li 2025)
3 METHOD
3.1 PRELIMINARIES
Research on the human visual system and neuroscience reveals several cortical principles, which could guide the development of manipulation policies and enable robots to achieve human-like pro- ficiency. These principles include:
1. Parallel streams with separable and complementary structures and functions. The dorsal and ventral streams emerge from distinct regions of the early visual cortex, processing dynamic and static visual signals respectively (Chen et al.) (025). Separate processing channels support generalization to novel scenes and adaptability dynamic tasks.
2. Dual-stream visuomotor control. Consistent with visual processing, human visuomotor con- trol follows dual-stream pattern (Rossit & McIntosh 202T): the ventral stream handles scene perception and object identification, while the dorsal stream translates retinal input into adaptive motor signals. Both streams are indispensable for precise motor control.
3. Enduring representations in the ventral stream. The ventral stream encodes stable visual
stimuli for cognitive processes (Kravitz al., [011] Becker et al. (025). Using allocentric (world-centered) frame of reference (Milner & Goodale (008), it forms enduring representations that facilitate recognition, long-term memory, and action planning.
4. Adaptive action reasoning in the dorsal stream. The dorsal stream encodes spatiotemporal dynamics to guide actions (Kravitz al., 2013] HHllquistt et al., 2024). Using an egocentric (boteccentered) frame of reference (Gheihman et al.) (025), it estimates properties of the target object real time and adjusts movement trajectories accordingly.
Building on these cortical principles of visuomotor control, we present CORTICAL POLICY, an im- itation learning framework for robotic manipulation. As illustrated in Fig.
2 the proposed policy
centers on dual-stream view transformer that integrates parallel streams: (1) a static-view stream encodes 3D spatial structures of the task scene through geometrically consistent representation learn-
3

---

Published as a conference paper at ICLR 2026
ing, which is supervised by pretrained 3D reconstruction model, i.e., VGGT; (ii) dynamic-view stream predicts adaptive actions through position-aware pretrained model. This model processes dynamic wrist-view frames estimate end-effector locations, generating action-oriented features that facilitate overall visuomotor reasoning. The complementary representations from both streams fused by action head, generating precise actions for robot control.
Dynamic View
Gaze Encoder
Allention
Decoder
Position-aware Pretraining
Policy Learning
render
Egncentre videos
Dyuaume Views
Gaze Model
Proprioception
Action Head
Euu-efflectrr position
Action-orintee features
Instruction
VGGT
RVT Encoder
"slack 2 blue blocks"
Static Views
RVT Encoder
Decoder
Lege
Attention
3D Supervision
Conn-Consistent F Features
Overview of the proposed cortical policy. Inspired by the dorsal- pathways in visual neuroscience, this architecture implements dual processing streams: a static-view stream for 3D spatial understanding and dynamic-view stream for end-effector position awareness.
Figure 2:
3.2 STATIC-VIEW STREAM
An enduring visual representation in the brain generally exhibits a unified and compact understand- ing of the 3D world, allowing easy generalization across environments, objects, and time. However, most off-the-shelf vision encoders fall short of comprehensive 3D) understanding as they are trained solely 2D images. To extract 3D-aware representations from image inputs, additional priors must be injected, typically via depth modality integration (Wu et al. 2025), cross-view consistency (You let al. 2025), or equivariance constraints (Howell et al. (023). Notably, You et al. (2025) and Lee et al. 2025 demonstrated that incorporating view equivariance into 2D foundation models signifi- cantly boosts 3D task performance.
Motivated by these findings, our static-view stream reinforces cross-view feature consistency to leamn 3D-aware semantic representations. We adapt RVT-2 backbone (Goyal al. 2024), preserving its core mechanisms including two-stage processing, intra-view self-attention and vision-language co-attention, while augmenting its feature extractor (RVT Encoder) with geometric constraints.
3D Supervision Generation. We leverage geometrically consistent keypoints as 3D supervision signals, which represent identical 3D points across viewpoints. This design anchors cross-view consistency directly in 3D geometry. Using spatial reasoning capabilities of VGGT, we predict depth map, confidence map, camera parameters for N static-view images. These predictions enable unprojection to camera-cove coordiiat point maps [.). which are then transformed into the world coordinate system to identify co-visible 3D points. We apply non-maximum suppression the first viewpoint's co- co-visible points, selecting the M K,, These candidates are tracked across viewpoints to yield geometrically consistent keypoint sets [.C.).2 As shown in Fig. (a), VGGT-derived keypoints primarily lie on the surfaces of objects or robots, providing geometric cues for task-relevant 3D structures.
points as candidate keypoint set
Given geometrically consistent keypoints K. (KK)))>> where keypoint k"' represents 3D point p, in viewpoint Vy, we supervise RVT Encoder to align cross-view features these keypoints. To enable fine-grained 3D supervision, incorporate a trainable 3x3 convolutional laver after the RVT Encoder, which enhances feature res- olution through local patch interactions You et al. 2025). For each keypoint k"', extract its fea- ture f via bilinear sampling from the view feature map. The training objective adopts SmoothAP loss (Brown et al., 2020). which optimizes cross-view view feature rankings by prioritizing similarity for geometrically consistent keypoints. For query k"" and target viewpoint ug. the positive and negative
Feature Consistency Optimization.
4

---

1 15p<N, Published as a conference paper at ICLR 2026
3D Supervision Generation
Train-Ouly
Train & Test Action
Stage
geowetr icolly consision heypponnts
Encoder
VGGT
RVT
bilinear sampling
high similarity
Reducrd
Stage 2
low simblarity
view-varia prseiccionoiiin
Conv Layer
view-lnvariaat prediction
Encoder
RVT
View Erhon
Views
View Features
Keypoint
Lege
Features
View 1 Evror
Feature Consistency Optimization
(b)
Figure 3: Static-view stream. (a) Visualization of geometrically consistent keypoints. (b) Pipeline.
sets are defined KC(i) = { and N(i) = [k) li i, lp P,ll2 > {} respectively, where is a tunable 3D distance threshold. The SmoothAP loss enforces ranking K(ii) above N(i):
K
1 EkyEx()) G(Dij)
SmoothAP (1 + x1))
(1)
Pcp
1+ E((() ((D,)) + EK)EN()) (DD,))
where Di, = f. fm f.r ((x) = (1 + e -/)== is sigmoid function. To suppress error accumulation in sequential view matching, propose cyclic geometric consistency loss:
Lego
SmoothAP(y, - Upa1)
p=1
where
Up+1, U1,
Up
N= N.
Optimizing Lege minimizes ranking loss over closed loop 11 + U2 UN U1. reducing
cumulative action estimation errors by aligning features from identical 3D location (Fig. 3 (b)). This mitigates view-specific biases and promotes viewpoint-invariin representation learning.
3.3 DYNAMIC-VIED VIEW STREAM
Unlike static-view stream that relies on enduring and holistic scene comprehension, the dynamic- view stream prioritizes adaptive egocentric action reasoning. This visuomotor pipeline requires im- mediate exploitation of transient visual cues, emphasizing action-centric perception from egocentric viewpoints (Milner& & Goodale 2008).
Diverging from existing egocentric action prediction work (Dai et al. 2024 Plizzari et al. 2024) that focus what actions occur, infer how actions are executed by predicting kinematie param- eters, including gripper translation, rotation, state (open or close) and collision indicator. Among these, gripper translation specifies 3D coordinates of end-effector, forming the geometric founda- tion for precise action proposals. In light of this, our dynamic-view stream achieves adaptive action reasoning by directly predicting end-effector position from dynamic wrist-mounted camera view (i.e., robot egocentric view). Accordingly, action reasoning is modeled attention map generation, analogous to egocentric gaze estimation (Lai et al., (2024) that predicts human visual attention maps from first-person videos. This shared formulation enables seamless extraction of view-specific fea- ture maps and saliency maps from gaze models. Both maps can be integrated into RVT--2, serving dynamic-view features and heatmaps, respectively.
Fig. illustrates the pipeline of dynamic-view stream, where a state-of-the-un egocentric gaze es- timation model (GLCC (Lai et al. 2024) is utilized feature extractor, coupled with RVT-2 action

---

Published as a conference paper at ICLR 2026
head for action reasoning. First, we construct egocentric video dataset through dynamic cameras, annotating each frame with ground end-effector positions. Subsequently, we perform position- aware pretraining this dataset, enabling GLC to infer positions from dynamic-view frames. Dur- ing training of Cortical Policy, the pretrained GLC model remains frozen, while its intermediate representations (including feature maps and saliency maps) extracted and fused with static-view counterparts for action decoding.
Egocentric Video Rendering. Preparing pretraining data requires addressing three critical issues: (1) Domain gap minimization: since human gaze provides localization cues for camera wearer ac- tions (Li et al., 2018 Huang et al., 2020). the field-of-view (FOV) discrepancy between human hean-mounted cameras (Grauman et 20022 Schaumlöffel et al. 2025) and robotic wrist-mounted cameras (James et al. 2020| Khazatsky et al. 2024) should be bridged, so as to transfer spatiotem- poral localization priors from human gaze to end-effector position. (2) Positional invariance miti- gation: the original wrist camera view produces invariant end-effector projections due to its fixed spatial relationship with end-effector, yielding non-informative annotations that increase overfitting risks and impede position-aware feature learning. (3) Cross-view alignment: the pretraining egocen- tric data serves as dynamic-view observations, thus needs to enable feature distribution consistency with static views to facilitate cross- view generalization. We resolve these issues by constructing dynamic virtual cameras within RVT renderer using real-time wrist camera extrinsies. Compared to raw wrist cameras, these virtual cameras adapt FOVs to match human egocentric data, diversify end-effector projections, and align processing with static viewpoints while preserving egocentric motion dynamics. The renderer associates each frame with its end-effector position via projection, generating annotated RGB-D sequences to constitute the final egocentric videos (see supplementary material for examples). In total, the dataset comprises 3,600 position-labeles videos (18 tasks x 100 episodes x 2 stages) at 224 x 224 resolution, exclusively used for position-awart pretraining.
Position-aware Pretraining. To enable knowledge transfer from human gaze estimation to end- effector position prediction, we initialize the GL.C backbone with Ego4DD-pretraine weights (Grau- man et al. 2022), then fine-tune it on our egocentric video dataset. Each video is segmented into 5-second clips and resized to 256 256 resolution. Following Lai et al. (024)), we randomly sample 8 frames per clip to form input sequences. Each sequence is fed into GLC to generate spatiotem- porally coherent saliency maps. For each frame, the end-effector location is determined by the most salient pixel in its saliency map. GLC ensures robust position localization through two core mechanisms: (i) capturing the temporal attention transition by leveraging egocentric motion cues in dynamic-view frames; (ii) explicitly modeling the spatial correlations between global and local tokens via its dedicated Global-Local Correlation module. Trained with KL.-divergence loss for 15 epochs, we select the final GLC checkpoint for feature extraction in the dynamic-view stream.
Dynamic-view Feature Extraction. We extract intermediate representations from the pretrained GLC as action priors for training dynamic-view stream pipeline. For clarity, the GLC architecture is partitioned into Gaze Encoder (comprising Visual Token Embedding, Transformer Encoder and Global-Local Correlation modules) and Transformer Decoder. The Gaze Encoder outputs visual to- kens that are projected view feature maps via a trainable linear projection layer. The Transformer Decoder generates saliency maps as view heatmaps. Formally, given patch size P, batch size B, and GLC embedding dimension D, the dynamic-view feature map F is acquired by concatenating FSA E R BP PPxDP from the last Transformer Encoder block and FGLC E TR. Bx(xxx) from the Global-Local Correlation module:
F = LP(EFSA FGLC1.) E (XXXXXX)
(3)
where the operator le implements concatenation along the channel dimension, LP(-) denotes lin- ear projection that aligns the 2D-dim GLC embeddings with RVT-2's C-dim token space, enabling integration of F into action decoding. With D- 768 and P = 16, our dual-stream transformer produces dynamic-view feature maps (B x 256 x 1536) and saliency maps (Bx x x2 x 128 x 128) via 2x2x2 downsampling. For compatibility with static-view heatmaps, the saliency map is resized to B x 1 x x 224 224, then temporally compressed to B x x24 224 via 3D convolution.
Dual-stream Action Prediction. Cortical Policy merges dual-stream outputs to determine gripper actions, where 3-DoF translation selects the highest-scoring 3D point from back- view heatmaps. For predicting 3-DoF rotation, gripper state and collision indicator, we follow Goyal ct al. (2024) in leveraging both global and local features. In our implementation, four viewpoints incorporated represent the scene at time I, including three static views and one dynamic view.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with SOTA methods on RLBench. The "Avg. Success" and "Avg. Rank" columns report the average success rate (%) and the average rank across 18 tasks. Best results are highlighted in bold, and the second best are underlined.
Models
Referenec
Avg. Success
Avg. Rank
Clese
Drag
Insert Pog.
Meat
Open
Place Cups
Place Wine
Buttoas
Push
Hiveformer
CoRL (2022)
45.3
8.1
52
76
100
52
0
80
84
CORL(2022))
7.6
92.8
RVT
CoRL (2023)
62.9
5.7
52.0s25
99 241.6
11.2e3.0
88 03.5
71.2
4.0c25
91.0s52
100.0e0.
Li-ggent
15.2s19
3.8at
90.4235
SAM-E
ICML (2024)
70.6
3.8
82.463.6
100,0g00
18,4
95.2
95,2-52
0.0gat
94.4
100.0e0.0
VIHE
IROS (2024)
77
3.6
48
100
76
88
100
RVT-2
RSS (2024)
77.5
3.5
93.3119
97. 9
28.0
100.0.00
92.0.ss
32.0152
84.0.98
100.0.0.
3D-MVP
CVPR (2025)
67.5
4.3
76.0
100.0
20.0
84.0
4.0
100.0
96.0
Ours
81.0
1.8
96.0cto
100.0g00
38.7
100.0.00
84.0a6s
24.0a13
94.7a38
100.0e0.0
Models
Reference
Cupboard
in
Drawer
Put
Safe
Bulh
Block
Slide
Shape
Blocks
Stack
Cups
Stack
Sweep to
Dustpan
Tum
Tap
Hiveformer
(2022)
28
PerAct
CoRL (2022)
28. 0pk4
51. 21.2
84.0a3.6
17.6220
74.0a130
16.8
26.4232
2.422
52,0
88 0s4.4
RVT
CaRI.(2023)
88.0a5.
91.2a3.0
28.863.9
E-agent
CoRL (2024)
66.45445
70.4238
98.4e1.9
73 2e22
74.4445
36.0e3.2
$1.2c54
33 6e6.
80.8e1.
95 2+1.3
SAM-F
(2244)2024)
64.0-28
92.0-s7
95.241.
78.4.56
95.2-18
34.4
26.4-46
0.0wam
100.0-00
100.0-03
VIHE
IROS (2024)
60
96
92
92
96
52
68
68
64
92
RVT-2
RSS (2024)
44.0-65
98.741
92,0c3.
66.741
74,7450
26.7419
80.0-MT
72.0.00
98.7419
94.7+18
3D.MVP
CVPR (2025)
60.0
100.0
92.0
60.0
48.0
28.0
40.0
36.0
80.0
96.0
Ours
65.3
100.0s000
98.72119
81.3a19
86.721.9
37.3+1.9
81,3c19
76.0233
100,0.00
94.7a50
Each viewpoint predicts a feature map Fý and a heatmap H, that indicates the end-effector pixel coordinate. Local features are pooled from Fj these coordinates, while the global feature vector is formed by concatenating the following components:
(F @ &(F @ H2); O(F3 ④ H3); @ O H.): (F2)
(F4))
static views
dynamic view
static views
dynamic view
where @ is element-wise multiplication; o(.) and v-()) represent sum and max max-pooling, respec- tively. The Gl.C-generated representations ensure that H4 effectively highlights task-relevant ego- centric cues (e.g., end-effector positions), therehy producing highly focused global features through teatmmap-eiggting rule F, 3 H,, The total loss combines action prediction loss and cross-view geometric consistency loss in Eq. 2):
L == Laction + ALages
(4)
where Laction is defined as the sum of cross-entropy losses for each action component, and À is trade-off parameter set to Through optimizing Eq. (4) Cortical Policy unifies dynamic-view action cues and static-view spatial knowledge, enabling the policy to robustly adapt to environmental perturbations and dynamic scene changes.
EXPERIMENT
This section evaluates Cortical Policy by answering the following questions: (1) How well does Cortical Policy perform in manipulation compared to state-of-the-an policies? (2) What impact do geometric consistency loss and various design choices in dynamic-view stream have on overall per- formance? (3) How robust is Cortical Policy against environmental perturbations (e.g., distractors, changes in camera pose, and object properties)?
(4) Does Cortical Policy work in real-world tasks? To this end, conduct experiments in both simulation and real-world scenarios, reporting results respectively.
in Sections
4.1 EXPERIMENTAL SETUP
We begin with an overview the datasets, baselines and evaluation metrics. For more detailed experimental settings, please refer Appendix B
Benchmark Datasets. For fair comparison, we adopt a standard multi-task manipulation bench- mark that contains 18 RLBench (Jaess al. 2020) tasks with 249 language-specified variations
7

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on dual-stream view transformer. All designs contribute to improving performance of Cortical Policy. "Arch.", "Pre.". "Heat." denote model architecture, position-aware pretraining, dynamic-viers view heatmap, respectively. "Single" means single-stream model with only static viewpoints, "Dual" means dual-stream model integrating dynamic and static viewpoints.
Models Arch, Loge
Pre. Heat. Success
Avg.
Rank
Avg.
Close
Jar
Drag Stick
Insert
Peg
Meat off Grill
Open Drawer
Cups
Place
Wine
Buttons
A
Single
77.5
93.3-19
37.3
28.0a1.
100.0.00
92.0.13
32.0.57
84.0
100.0-00
B
Single
V
80.1
2.4
94.7a19
100,0a00
25.3
100,0aa
94.7419
311.
86.7250
100.0
C
Dual
77.6
97.3-n0
98.7400
30.7-100
100.0aa0
88.0433
28.0
93.3
100.0aa0
D
Dual
73.3
4.8
90.7-19
90.7-112
26.7.33
100.0.00
94.2419
20.0.00
82.746
98.7-19
E
Dual
79.5
3.1
90.721.
97.3x1.
29.3
100,0.00
100.0.00
46.7.25
88.0a57
100.0.00
(Ours)
Dual
81.0
96.0-n0
100.0.00
38.7-63
100.0.00
84.0
24.0.13
94.7-s8
100.0-00
Models
Arch. Loge
Pre.
Cuphourd
Put in
Drawer
Put
Safe
Screw
Bulb
Block
Slide
Shupe
Sort
Blocks
Stack
Stack
Cups
Sweep to
Dusspmn
Turn
Tap
Single
44.0a0.
78.7
92.0433
86.7419
74.7
26.7419
80.0+57
72.0am
98.7419
94.7=19
B
Single
61.3.s0
98.2419
92.3.19
92.0i53
82.7.18
18.7
92.0
81.3.s0
100.0.00
94.7.19
C
Dual
x
73.3-50
96.0enn
92,0gu0
85.366
78.7s50
6.7c38
81.3st9
50.7550
100.0.00
96.0-0
48.0+143
97.3xx8
88.0-86
85.3.19
65.3.50
18.7-10
86.2-18
44.0+142
94.7450
88.0-1a
E
Dual
50.7a68
88.0a33
89.321.
88.0us5
84.0.33
22.7u3a
82.7.38
81,3
98.7a19
93,3as8
((uns)
Dual
65.3.n.4
100.0.00
98.721.
81.3a1.
86.7219
37.3a1.
81.3a19
76.0a3
100.0.00
94.7=5.
simulated in CoppeliaSim (Rohmer et al. 2013). The tasks are performed by a Franka Panda robot
arm with a parallel jaw gripper. Raw visual observations are captured by four 128 x 128 RGB-D cameras mounted at front, left shoulder, right shoulder, and wrist of the robot. The gripper poses are executed by sampling-basee motion planner. Each behavior-cloning agent is allowed up to 25 steps to complete a task. Following PerAct, we use the same training- -est split, training all models on 100 demonstrations per task and evaluating single checkpoint on all tasks.
Baselines and Evaluation Metries. We benchmark Cortical Policy against 8 state-of -fte--f--thhaaan--eee manip- ulation policies: Hiveformer (Guhur et al. 2022), PerAct (Shridhar et al., 2023), 3 RVT (Goyal al., 2023). VIHE (Wang et al. 2024), RVT-2 (Goyal et al. 2024), Zaagent (Ma et al., 2024). SAM- E. (Zhang et al. 2024), and 3D-MVP (Qian al. 2025 which are predominantly based view transformer architectures and have demonstrated effectivvness in 3D object manipulation. For vi- sual imput, PerAct uses 3D voxels, Hiveformer utilizes raw cameras positioned on the wrist and both shoulders, whereas the other baselines employ multiple static virtual cameras. We report success rates for individual tasks, along with average success rate and rank across all tasks.
4.2 PERFORMANCE COMPARISON ON RLBENCH
Table summarizes the comparison results on RLBench. Cortical Policy achieves the highest aver- age success rate, outperforming the best- best-performing baseline (RVT-2) by absolute improvement of 3.5%. In terms of individual tasks, our model achieves top-1 or top-2 performance in 14 out of 18 tasks. These results demonstrate the efficacy of Cortical Policy for robotic manipulation, ad- vancing toward human-like visuomotor control. For tasks where RVT and RVT-2 already achieve success rates above 90%, our dual-stream framework generally yields better performance, seen in "close jar", "sweep to dustpan",, and "put in safe". We observe that our model outperforms existing methods in multi-object tasks, such as "stack cups" and "stack blocks", with margin of 1.3%- 4.0%. These tasks implicitly require understanding spatial relationships among objects, validating the effectiveness of 3D prior injection in the static-view stream.
4.3 ABLATION STUDY
To evaluate the impact of key design choices in Cortical Policy, we conduct ablation experiments on RLBench, with results summarized in Table 2 The ablated variants are implemented as: (A) Re- moving the entire dynamic-view stream along with cross-view geometric consistency loss Lege- (B) Using only static-view stream. (C) Ablating position-awant pretraining and instead fine-tuning the gaze model jointly with manipulation policy in an end-to-end manner, excluding Lege. (D) Em- ploying only view feature maps without heatmaps during dynamic-view feature extraction, also excluding Lege- (E) Removing Lege while retaining all components of dynamic-view stream. An identical training configuration is maintained for all ablation studies. The discussion follows.
Effects of cross-view geometric consistency. Lege leads to consistent improvements across archi- tectures, e.g., variant B outperforms A by 2.6%, the full model F surpasses E by 1.5%, demonstrat-
8

---

Published as a conference paper at ICLR 2026
Table 3: Results on THE COLOSSEUM. The "Avg. Success" and "Avg. Rank" columns report
the average success rate (%) and the average rank across all perturbations on 4 COLOSSEUM tasks.
Models
Arch.
Cege
Pre.
Heat.
Avg. Success
Avg. Rank
Perturbations
Color
Color
Texture
Texture
MO-
Size
7.7
D.tann
5.3a5
2.0u10
4,0a3.
RVT RVT-2
37.7
6.0
3.0a55
27. 27.0x27.3
36.021550
50.0238.0
57.3222.7
50.7:296
Single
60.5
15.0
64.0115.
64.9.27.2
93.4
66.2.313
80.4.175
Variant B
Single
63.8
3.3
10.3es.
69.7.18.1
70.2+28.0
95.4
7164300
84.0:142
Variant D
Dual
x
V
66.4
2.9
10,0s9.2
69.7s27.5
72.9x29.9
94.0e20
74.7a245
83. 631671
Variant
Dual
68.7
8.7e15.
75 75.0
73.8429.5
96.7au7
11.1
2-13
Ours
Dual
69,9
1,9
10.0a15.1
78.0k266
76.9228.9
100.0a00
66.7a27
86,7216.1
Models
Arch.
Coge
Pre.
Heat.
Size
RO-
Color
Light
Color
Table
Texture
Table
Distractor
Background
Texture
RLBench Variations
Canera
Pose
PerAct RVT
9.3x1.9
7.0s4A
8.0a5.3
3.0a33
2.723.
8.0
25 25.0a24.7
9.0a7.1
22.7.22.3
52,0.307
42.01.08
48.0u219
13.31182
40.0.329
41.0:256
45.0.31.
RVT-2
Single
44.4-292
42.3a317
54.42344
60.4
72.0
63.7a200
62.7:286
Variant B
Single
V
44.0m33.1
73.7+264
47,3e307
61.0s15.2
64 64.0231.2
67.0224.5
68 02252
66.7-314
Variant
Dual
40.0 0-:7
73.2-202
49.4-31
69.2435.
79.1-32.
70.0=766
74.7-32
68.3-312
Variant E
Dual
x
53.8-22
78.3-23.7
60.0c25
70.7e25.9
74.2+185
76.7.22.3
71.42999
Ours
Dual
51. 6:33.
69,3c540
46.7e32.
75.0s27.5
83.12201
77.7x269
82.3s174
74.0=327
ing the effectiveness of Lege for both single-stream and dual-stream policies. This validates that our cewppint-invariiit representation learning method benefits robotic manipulation.
Effects of position-aware pretraining. Compared to end-to-end training (variant C), freeezing position-aware pretrained gaze model (variant E) obtains 1.9% higher average success rate and sta- bility across tasks. This demonstrates the superiority of our pretraining approach.
Choice of gaze model representations. Our dynamic- viwww stream utilizes both feature maps and heatmaps from the gaze model. Without heatmaps, variant D underperforms single-stream variants, confirming that the heatmaps' explicit action cues are crucial to dynamic-view stream.
Dual-stream versus single-stream architecture. Both the static-view and dynamic-view streams boost performance, with gains of 2.6% (variant B vs. A) and 0.9% (variant F vs. B). respec- tively. This demonstrates the effectiveness of incorporating dynamic-view perception for action prediction. Notably, the dynamic virtual camera breaks the strict orthographic constraints of multi- camera setups in existing view transformers, while it still improves performance. We also record computational time for each component (Fig. 4 (a)), showing that our dual-strean design enhances performance without sacrificing efficiency.
4.4 ROBUSTNESS EVALUATION ON COLOSSEUM
We further evaluate the robustness and generalization capabilities of our method on the COLOS- SEUM benchmark (Pumacay et al. 2024), which is an extension of RLBench. The models trained on the original RLBench tasks are evaluated in environments spanning diverse unseen perturba- tions, encompassing changes in object color and size, lighting, distractors, and camera poses, etc. As shown in Table 3 Cortical Policy obtains the highest average success rate among all the base- lines (including PerAct, RVT-2 and its ablation variants), notably outperforming RVT-2 by 9.4%. Crucially, ablation results indicate that the dynamic-view stream is the primary driver of this bustness, contributing significantly larger gains than Lege under severe perturbations. Among all the 14 evaluated generalization settings, our method achieves the top performance in 9 of them. These results demonstrate that Cortical Policy possesses strong robustness against environmental perturba- tions. More details about the data and results of the COLOSSEUM benchmark are in Appendix G
4.5 REAL-WORLD EXPERIMENT
We design four real-world tasks for evaluation: a basic task ("stack 2 blocks") aligned with the RLBench "stack blocks" task, spatial reasoning task ("stack 2 blocks in between the bottles") and two challenging dynamic tasks ("stack 2 blocks with target/base displacement"). These tasks extend the simulated stacking scenario by introducing real-worle complexities including spatial constraints and unpredictable scene dynamics. Each task is evaluated through 10 trials (see Appendix [B.2 for hardware details). As shown in Fig. (b), compared ablated variants (B, E), RVT, RVT-2, and 3D-

---

Published as a conference paper at ICLR 2026
RVT
3D-MVP
RVT-2
Varizet
Variont E
Cortical Policy
Hed
90
Duander
8.9
Success Rate
60
60
60
60
Tine& te nitgpes
VOGT Aggraaee
"Stnck blocks between the
"Stack blacks with tergot displacemen"
"Stack blocks with base displacement"
Kopilas buneek
"Stack blocks"
8.5
4.06
Initial State
0.568
Dynamics
10.0
0.727
0.334
Suako
Dynamic-view
Sapariiise
Head
Final State
(a)
(b)
Initial State
piok
place
pick
&
irojectory under dynamics
plane
Ours RVT-2
Success Failure
Figure 4: (a) Training time of Cortical Policy modules, with time cost of 3D supervision gen- eration, dual streams, action head. (b) (Top) Real-world performance comparison. (Bottom) Visualization of the initial and final states for the four real-world tasks. (c) Trajectory visualiza- tion for "stack 2 blocks with base displacement" task.
MVP, Cortical Policy achieves: (1) 30% higher success than RVT and RVT-2 (and 10% 3D-MVP) in the spatial reasoning task, confirming that Lage enhances geometric understanding, (2) an 80% success rate under dynamic perturbations, whereas static-view-only approaches completely fail (0%). Fig. 4 4[c) demonstrates that our method succeeds by dynamically re-planning trajectories, while baselines fail do so, highlighting Cortical Policy's s adaptation capability through dynamic- view processing. These real-world results collectively validate the robustness and superionity of our method in physical deployment. Demos can found in supplementary material.
5 CONCLUSION
This paper presents Cortical Policy, dual-stream framework for enhancing spatial reasoning and dynamic-scene adaptability of robotic manipulation policies. Through VGGT-superviied geometric consistency optimization, inject strong 3D priors into the policy, thereby improving spatial un- derstanding. Complementing this, the dynamic-view stream learns to discover and attend to action- critical targets, demonstrating its effectiveness adjustment to task dynamics, an ability absent in prior work. Extensive experiments demonstrate the superiority of Cortical Policy in both simulated and real-world scenarios, highlighting the con- tribution of the dynamic-viee view stream to handling unpredictable scene perturbations.
tracking the end-effector. This enables adaptive
Limitations and Future Work. While Cortical Policy demonstrates strong within-task general- ization (validated on COLOSSEUM), its zero-shot transfer to novel tasks remains challenging, as reflected by the 24% success rate on the unseen "close laptop lid" task. promising direction is to enhance its compositional abstraction capability for task generalization (e.g., by recombin- ing learned perceptual and motor primitives). Building on its modular design, we plan to extend this framework with multi-resolution encoders and hierarchical attention mechanisms to handle ex- tremely fine- ree-grained manipulation. The adaptive fusion of dual-stream representations at token and viewpoint levels also warrants further exploration. Furthermore, extending Synamic-view stream to track diverse targets beyond the end-effector (e.g., specific objects, affordance points, multiple entities) will further probe the framework's generalization in open-world world settings.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility of Cortical Policy, our implementation details are provided in Ap- pendix B.3 Sections 3.2 and 3.3 describe the methodology and data processing steps for egocentric video dataset used in position-awes aware pretraining. Additionally, the anonymous source code is avail- able in supplementary material to facilitate validation and replication of our findings.
ACKNOWLEDGMENTS
We would like to thank the reviewers for their constructive comments. This work is supported by National Natural Science Foundation of China (Grant No. 62406092), National Natural Sci- ence Foundation of China (Grant No. U24B20175), Shenzhen Science and Technology Program (Grant No. KJZD202409031000 7022), Guangdong Basic and Applied Basic Research Foun- dation (Grant No. 2025A1515010169,, Shenzhen Science and Technology Program (Grant No. KQTD20240791102270000
REFERENCES
Ali Abouzeid, Malak Mansour, Zezhou Sun, and Dezhen Song. Geoaware-vla: implicit geom- etry aware vision- Assionlanguaage-ctionn model, 2025. URL http:::/aaxx....rr//bbb/////
14117
Maxi Becker, Tobias Sommer, and Roberto Cabeza. Insight predicts subsequent memory via cortical representational change and hippocampal activity. Nature Communications, 16(1):4341, 2025.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: smoothing the path towards large-scale image retrieval. In ECCV, pp. 677-694, 2020.
Nihong Chen, Hailin Ai, Jiewei Chen, and Peng Zhang. Hierarchical and parallel processing: from primate visual cortex to artificial intelligence. Science China Technological Sciences, 68(8):1-8, 2025.
Shizhe Chen, Ricardo Garcia, Cordelia Schmid, and Ivan Laptev. Polamet: 3d point clouds for language-guider robotic manipulation. In CoRL, 2023.
Shizhe Chen, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Sugar: pre-training 3d visual
representations for robotics. In CVPR, 2024.
Hengshuo Chu, Xiang Deng, Qi Lv, Xiaoyang Chen, Yinchuan Li, Jianye HAO, and Liqiang Nie. Ld-affordancel.LMM Harnessing large language models for open-vocabulary affordance detection in 3d worlds. In ICLR, 2025.
Wenbo Cui, Chengyang Zhao, Yuhui Chen, Haoran Li, Zhizheng Zhang, Dongbin Zhao, and He Wang. C13r: 3d reconstruction and contrastive leamiin for enhanced robotic manipulation representations, 2025. URL htthttps:: /arxiviivvv..///.. wwg//rg///.... 2507 08262
Guangzhao Dai, Xiangbo Shu, Wenhao Wu, Rui Yan, and Jiachao Zhang. Gpt4ego: unleashing the potential of pre-trained models for zero-shot egocentric action recognition. IEEE Transactions on Multimedia, 2024.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. In ICLR, 2022.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, pp. 6804-6815, 2021.
Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: integrating visual foundation model with a memory architecture for robotic manipulation. In ICML, 2025.
Galina Gheihman, Evan Samuel Madill, Shannon Coy, and Michael G Erkkinen. Clinical reasoning: 67 -ld woman with progressive headache, visual hallucinations, and seizures. Neurology, 104(8):e213496 2025.
11

---

Published as a conference paper at ICLR 2026
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu- Wei Chao, and Dieter Fox. Rvt: robotic view transformer for 3d object manipulation. In CoRL, 2023.
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Kei Chao, and Dieter Fox. Rvt-2: learning precise manipulation from few demonstrations, RSS, 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: around the world in 3,000 hours of egocentric video. In CVPR, pp. 18995-19012, 2022.
Lierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. In CoRL, pp. 175-187, 2022.
Michael N Hallquist, Kai Hwang, Beatriz Luna, and Alexandre Y Dombrovski. neward-basee op- tion competition in human dorsal stream and transition from stochastic exploration to exploitation in continuous space. Science advances, 10(8))aajj2219 2024.
Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, and Robin Walters. Equivariant single view pose prediction via induced and restriction representations. In NIPS, volume 36, pp. 47251- 47263, 2023.
Yifei Huang, Minjie Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato. Mutual context network for jointly estimating egocentric gaze and action. IEEE Transactions on Image Processing, 29:7795- 7806, 2020.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J. Davison. Coarse-to-fine q-attention: efficient learning for visual robotic manipulation via discretisation. In CVPR, pp. 13739-13748, 2022.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: large-scale in-the-wild robot manipulation dataset. In RSS, 2024.
Dwight J. Kravitz, Kadharbatcha S. Saleem, Chris I. Baker, and Mortimer Mishkin. A neural framework for visuospatial processing. Nature reviews. Neuroscience, 12:217-230, 2011.
Dwight J. Kravitz, Kadharbatcha S. Salcem, Chris I. Baker, Leslic G. Ungerleider, and Mortimer Mishkin. The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends in Cognitive Sciences, 17:26-49. 2013.
Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg. In the eye of transformer: global-local correlation for egocentric gaze estimation and beyond. International Journal of Computer Vision, 132(3):854-87777 871, 2024.
Seonho Lee, Jiho Choi, Inha Kang, Jiwook Kim, Junsung Park, and Hyunjung Shim. 3d-aware vision-language models fine-tuning with geometric distillation, 2025. URL httph:t/t///att
org/ htg/abs/2555 200.. 09883
Hao Li, Qi Lv, Rui Shao, Xiang Deng, Yinchuan Li, Jianye Hao, and Liqiang Nie. STAR: Learning diverse robot skill abstractions through rotation- augmene- vector quantization. In ICML, pp. 35562-35580. 2025a.
Peiyan Li. Yixiang Chen, Hongtao Wu, Xiao Ma. Xiangnan Wu. Yan Huang, Liang Wang, Tao Kong, and Tieniu Tan. Bridgevla: input-output alignment for efficient 3d manipulation learning with vasion-language models, 2025b.
Yin Li, Miao Liu, and James M. Rehg. In the eye of beholder; joint learning of gaze and actions in first person video. In ECCV, 2018.
12

---

Published as a conference paper at ICLR 2026
Yichao Liang, Kevin Ellis, and Joao Henriques. Rapid motor adaptation for robotic manipulator arms. In CVPR, pp. 16404-16413, 2024.
Tao Lin, Gen Li, Yilei Zhong, Yanwen Zou, and Bo Zhao. Evo-O: vision-languaage-action model with implicit spatial understanding, 2025.
Qi Lv, Hao Li, Xiang Deng. Rui Shao, Michael Yu Wang, and Liqiang Nie. RoboMP2 A robotic multimodal perception-planniin framework with multimodal large language models. In ICML, pp. 33558-335744 2024.
Qi Lv, Hao Li, Xiang Deng, Rui Shao, Yinchuan Li, Jianye Hao, Longxiang Gao, Michael Yu Wang. and Liqiang Nie. Spatial-temporal graph diffusion policy with kinematic modeling for bimanual robotic manipulation. In CVPR, pp. 17394-17404, 2025.
Teli Ma, Jiaming Zhou, Zifan Wang, Ronghe Qiu, and Junwei Liang. Contrastive imitation learning for language-guiin guided multi-task robotic manipulation. In CoRL, pp. 4651-4666) 2024.
David Milner and Melvyn A. Goodale. Two visual systems re-viewed. Neuropsychologia, 46: 774-785, 2008
Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Gio- vanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocen- tric vision. International Journal of Computer Vision, 132(11):4880-49))
2024.
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The COLOSSEUM: A benchmark for evaluating generalization for robotic manipulation. In RSS 2024 Workshop: Data Generation for Robotics, 2024.
Shengyi Qian, Kaichun Mo, Valts Blukis, David F Fouhey, Dieter Fox, and Ankit Goyal. 3d-mvp: 3d multiview pretraining for manipulation. In CVPR, pp. 22530-22539, 2025.
Eric Rohmer, Surya P. Singh, and Mare Freese V-rep: a versatile and scalable robot simulation framework. In IROS, pp. 1321-1326, 2013.
Stéphanie Rossit and Robert D McIntosh. Visuomotor control in the healthy and damaged brain. In Encyclopedia of Behavioral Neuroscience, pp. 570-578. Elsevier, 2021.
Timothy Schaumlöffel, Arthur Aubret, Gemma Roig. and Jochen Triesch. Human gaze boosts
object-centered representation learning, 2025. URL httpa///aaxvvvo////aaae www.xiii.org//ba...
02966
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: a multi-task transformer for robotic manipulation. In CoRI., pp. 785-799, 2023.
Haozhan Tang, Tianyi Zhang, Matthew
and Weiming Zhi Bi-manual joint cam-
era calibration and scene representation, 2025. URL
24819
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: visual geometry grounded transformer. In CVPR, 2025.
Weiyao Wang, Yutian Lei, Shiyu Jin, Gregory D Hager, and Liangjun Zhang, Vihe: virtual in-hand eye transformer for 3d robotic manipulation. In IROS, pp. 403 410, 2024.
Zizhang Wu, Fan Song, Yuanzhu Gan, Yunzhe Wu, Tianhao Xu, Xiaoquan Wang, Rui Tang, and Jian Pu. Advancing 3d object detection with depth-aware spatial knowledge distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-16, 2025.
Yang You, Yixin Li, Congyue Deng, Yue Wang. and Leonidas Guibas. Multiview equivariance improves 3d correspondence understanding with minimal feature finetuning. In ICLR, 2025.
Junjie Zhang, Chenjia Bai, Haoran He, Zhigang Wang, Bin Zhao, Xiu Li, and Xuelong Li. Sam- e: leveraging visual foundation model with sequence imitation for embodied manipulation. In ICML, pp. 58598. 2024.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d lasion-language-action generative world model. In ICML, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
This appendix provides supplementary materials supporting the main paper, organized as follows:
LLM Usage Disclosure
Role specification of large language models
Experimental Setup RLBench tasks, implementation details and baselines
Time Analysis and Capacity Analysis Additional computational efficiency analysis
Keypoints Visualization) Qualitative results of geometrically consistent keypoints
Egocentric Rendering Visualization Position-aware pretraining data samples
Failure Case Analysis In-depth investigation of RVT-2's spatial reasoning limitations
COLOSSEUM Experiments/
Comprehensive generalization and robustness evaluation
A LARGE LANGUAGE MODEL USAGE DISCLOSURE
In compliance with ICLR 2026 policy, we disclose the use of large language models (LLMs) in the preparation of this work:
DeepSeek-RI (https: hwwp//wwwwww.//////////////./.ws www.wweeeee.....mm com) was utilized exclusively for language polishing of non-technical sections (Introduction and Related Work).
All technical content (Method, Experiment and Conclusion) was written by humans with- out L.I.M assistance.
LLM-generatee text rigorously verified and modified by the authors.
No LLM was used for data analysis, algorithm design, or scientific interpretation.
The authors assume full responsibility for all content in this manuscript.
B EXPERIMENTAL SETUP
This section specifies the experimental framework covering RLBench tasks, real-robot setup, our implementation details, baseline architectures and processing pipelines.
B.1 RLBENCH TASKS
We briefly summarize the RLBench tasks Table comprising tasks with 249 variations across object color, category, placement, count, shape, and size. Each task requires executing manipulation sequences such as pick-and-place, tool use, drawer opening, and precision operations like peg inser- tion and shape sorting. During evaluation, the robot handles variations including novel object poses, randomly sampled language instructions, and scenes with unseen object appearances. This task vari- ety necessitates manipulation policies with generalizable comprehension of scenes and instructions, along with adaptable skill acquisition beyond specialized adaptation to individual scenarios.
For more detailed introduction of each task, please refer PerAct (Shridhar al., 2023). For training and evaluating Cortical Policy, we render four virtual camera views to get visual inputs. including static viewpoints and dynamic viewpoint. We visualize the rendered images in Fig.
B.2 REAL-ROROT EXPERIMENTAL SETUP
To evaluate Cortical Policy in real-world scenarios, we deploy tabletop manipulation system con- sisting of dual-arm Cobot Agilex ALOHA robot. As shown in Fig. 6 the experimental setup integrates two fixed cameras for static-view perception, complemented by two wrist-mounted cam- cras for dynamic-view perception. In our experiments, we utilize a single arm to execute four distinct manipulation tasks: one benchmark task aligned with RLBench and three new tasks designed to test spatial reasoning and dynamic scene adaptation abilities. Each task collects 45 human- telean--teepeeteee demonstrations with placement variations, and single agent is trained in multi-task setting all four tasks. For evaluation, this unified agent is tested on novel spatial configurations unseen in the training demonstrations, Four real-world tasks are detailed as follows:

---

Published as a conference paper at ICLR 2026
Table 4: Summary of the 18 RLBench tasks for multi-task experiments.
Task Name
Language Template
Hof Variations
Variation Type
close jar
"close the jar"
20
color
drag stick
"use the stick to drag the cube onto the target"
20
color
insert peg
"put the ring on the spoke"
20
color
meat off grill
"take the off the grill"
2
category
open drawer
"open the drawer"
placement
place cups
"place cups on the cup holder"
3
count
place wine
"stack the wine bottle to the of the rack"'
3
placement
push buttons
"push the button, [then the button]"
50
color
put in cupboard
"put the in the cupboard"
category
put in drawer
"put the item in the drawer"
placement
put in safe
"put the money away in the safe on the shelf"
3
placement
screw bulb
"screw in the light bulb"
20
color
slide block
"slide the block to target"
4
color
sort shape
"put the in the shape sorter" blocks" "stack the other cups on top of the cup"
shape
stack blocks
"stack
60
color, count
stack cups
20
color
sweep to dustpan tap
"sweep dirt to the dustpan"
2
size
"turn tap"
2
placement
Stack 2 blocks:
This basic task requires the robot to sequentially stack a yellow block onto an orange block, corresponding to RLBench "stack blocks" task for sim-to-real transfer evaluation.
Stack blocks in between the bottles: This task is extended version of the basic task, testing the understanding of spatial relationships by requiring the robot to: (1) Precisely place the orange block in the region between two bottles. (2) Stably stack the yellow block atop the orange block. Stack blocks with target displacement: This task introduces real-world unpredictability, eval- uating how effectively the dynamic-view stream handles trajectory adaptation. While the robot is approaching the first block, it is displaced, requiring adaptive trajectory re-planning to complete the original stacking task.
Stack 2 blocks with base displacement: This task also tests the dynamic adaptation capability by displacing the already-stacked orange block during the yellow block's approach phase. The robot must re-locate the orange block and put the yellow block on it.
For each real-world task, 10 independent trials conducted to calculate the overall success rate. A trial was considered successful only all sub-actions of the task are executed correctly.
B.3 IMPLEMENTATION DETAILS
All models are trained on 8 NVIDIA A800 GPUs. We measure the training efficiency of Cortical Policy NVIDIA A800 GPU, revealing that VGGT-based 3D supervision generation constitutes the most computationally intensive component. As shown Fig. (a), the average time costs for VGGT feature aggregation, VGGT decoding, and geometrically consistent keypoint extraction are 1.00 x 10-2, 2 8.80 x 10 and 8.92 x 10-3 minutes respectively, resulting in a total of 3.09 x 10-2 minutes for the complete 3D supervision generation. This process is 4.7x slower than the action reasoning procedure of Cortical Policy, primarily due to the computational demands of VGGT infer- ence. To mitigate this bottleneck, we implement multi-stage strategy that decouples 3D supervision generation from feature consistency optimization. Specifically, geometrically consistent keypoints are precomputed from VGGT, stored, and indexed by their corresponding demonstration IDs.
Cortical Policy is trained for 32.5K steps using the 8-bit LAMB optimizer (Dettmers et al., 2022) with a cosine learning rate decay schedule and 2K-step warm-up. We select the final converged model for evaluation. For baseline methods excluding RVT-2, we report evaluation results from their original publications, with the performance of Hiveformer reported by Chen et al. (2023), and the performance of PerAct reported by Goyal et al. (2023). Given the architectural similaami mamitariies between our framework and RVT-2, we conduct a controlled comparison by training RVT-2 from scratch under identical conditions as ours, including the same computing resources and matching
2

---

Published as a conference paper at ICLR 2026
Close Jar
Drag Stick
Insert Peg
Meat off Grill
Open Drawer
Place Cups
Place Wine
Push Buttons
Put in Cupboard
Put Drawer
Put in Safe
Serew Bulb
Slide Block
Sort Shape
Stack Blocks
Stack Cups
Sweep to Dustpan
Turn Tap
Figure 5: Rendered views for 18 RLBench tasks. Three orthographic static cameras and a dynamic camera (defined by the wrist camera's 's raw extrinsic parameters) are used to generate image inputs. For each task, the first three lines show the static views (top, front, right), and the last line shows the dynamic view; rendered results of the first (coarse) stage are presented in the left part while that of the second (fine) stage are shown right.
Flodd Camera 1
Rigls Wetsr Camera
Righo Gripper
Fiood Camere
Left Groper
Figure 6: Real-world setup.
hyperparameters (32.5K training steps with batch size 512). This implementation differs from the pretrained RVT-2 model (~80K training steps with batch size 192) released by /oyal et al. (2024), ensuring a fair assessment of our methodological contributions. To account for the randomness in RI RiBenchh's sampling-based motion planner, perform three independent test per model, each
3

---

Published as a conference paper at ICLR 2026
Table 5: Training Hyperparameters of Cortical Policy.
Hyperparameters
Value
Batch size
512
Learning rate
5.44 x 10 10-3
Optimizer
LAMB
Learning rate schedule
cosine decay
Weight decay
x 10-4
Warm-up steps
2000
Training steps
32.5K
Training epochs
104
Lege loss weight (A)
Negative distance threshold (O
0.1
Keypoints per view (M)
300
Sigmoid temperature (T)
0.01
Number of static views (N)
3
GLC training epochs
15
comprising 25 episodes per task. The resulting average success rates with standard deviations are reported in Tables [ and
We implement data augmentation protocols consistent with established view transformers (Goyal et al., 2023) 2024). For translational augmentation, point clouds are randomly perturbed within + 12.5 em along each Cartesian axis. For rotational augmentation, point clouds undergo random z-axis rotations bounded by 4 T our training configuration, including a batch size of 512 (64 x 8) and learning rate scaling with batch size as 1.0625 10 x bs.
B.4 BASELINES
This section details the baseline manipulation policies that are based on view transformers, analyzing their view processing architectures and vision-to-action mapping frameworks.
(1) Hiveformer (Guhur al. 2022 predicts actions conditioned on a natural language instruction, visual observations at t steps (RGB images, point clouds and proprioception from wrist, left shoul- der, and right shoulder cameras) and previous actions at steps (gripper translation, rotation, and open/close state). Multi-modal tokens are formed by concatenating word tokens and visual tokens from all camera views with embeddings of camera ID, step ID, modality type, and patch location. A transformer encoder then models relationships among camera views, observations and instructions, current and history information. Finally, a CNN decoder predicts rotation and gripper state, while UNet decoder predicts translation.
(2) RVT (Goyal et al. 2023) re- re-renders original visual observations (RGB-D images from front, left shoulder, right shoulder, and wrist cameras) into five static virtual viewpoints anchored at the robot base (front, top, left, right, back). This generates 7-channel images: for RGB, I for depth, and for pixel coordinates. These re-rendered images, along with language instruction and gripper state, are processed by a joint transformer that sequentially computes intra-view attention, cross-view attention and vision-language attention. The model outputs view-specific heatmaps for predicting 3D translation, and outputs global features that concatenate all viewpoints for estimating gripper rotation, state, and collision indicator.
(3) VIHE (Wang et al.) 2024) employs a multi-stage view rendering and action refinement framework comprising an initial global stage and two refinement stages. The initill stage replicates RVT's five- camera rendering, while the subsequent stages autoregressively generate five virtual in-hand views attached to the previously predicted gripper pose, enabling progressively finer workspace focus. The view transformer adopts masked self-attention to facilitate intra-stage and cross-stage interactions among language instructions, proprioception, multi-stage and multi-camera tokens. During refine- ment, relative transformations predicted to update prior stage outputs (gripper poses, collision indicators, and states). Final action predictions are derived from the last refinement stage.
(4) RVT-2 (Goyal al. 2024) extends RVT with two-stage architecture: the coarse stage predicts area of interest, while the fine stage renders close-up images for precise gripper pose estimation. Beyond this multi-stage design, RVT-2 improves computational and memory efficiency through
4

---

Published as a conference paper at ICLR 2026
replacing transposed convolutions with convex upsampling, optimizing network parameters, sub- stituting PyTorch3D with a point-renderer for virtual rendering, and utilizing both global and local features to predict gripper rotation, state and collision indicator. Additionally, it reduces five static virtual viewpoints to three (front, top, right), accelerating training while maintaining performance.
(5) E-agent (Ma et al., 2024) integrates visual and language encoders, multi-view query transformer (MVQ-Former), contrastive imitation learning module. The visual encoder processes five virtual images with intra-view self-attention. Language instructions encoded using CLIP and projection layers, generating language tokens for cross-attention computation. MVQ-Former transforms visual tokens into view-specific query tokens for two contrastive learning objectives: a state-language one aligns visual and text tokens in a joint embedding space to learn discriminative representations; (state, language)-
one concatenates current visual, query and language tokens, then processes them through 4 self-attention layers to derive current-state queries. These queries are contrasted against future-state features, which are extracted by feeding next-state images to the visual encoder and applying average pooling, Both objectives augment the standard imitation learning loss during training to enhance representation learning, but are excluded during inference.
(6) SAM-E (Zhang et al., 2024) incorporates the Segment Anything Model (SAM) as founda- tional visual perception module. Based on RVT's rendering strategy, it processes RGB channels through a LoRA-tuned SAM encoder, enabling generation of prompt-guided object-oriented image embeddings. Concurrently, spatial features are extracted from depth and pixel coordinate channels via a Conv2l) layer. These features channel-wise concatenated with SAM embeddings to form composite view tokens. Combined with language tokens, these view tokens are processed by transformer through view-wise and cross-view attention mechanisms. This generates enriched vi- sual tokens for action-sequence prediction. Unlike step-by-step paradigms, SAM-E models coherent action sequences by enforcing temporal smoothness end-effector poses. For translation predic- tion. it extends view-specific heatmaps with temporal channels. While rotation, state, and collision indicators are derived from view-fused global features following RVT.
(7) 3D-MVP (Qian et al. 2025) aims to augment visual encoder for learning generalizable repre- sentations, decomposing the view transformer into an input renderer, an encoder mapping static virtual images to latent embeddings, and action decoder. Rather than training the RVT architec- ture from scratch, 3D-MVP adopts two-stage training paradigm: first pretrains RVT encoder using masked autoencoding on large-scale 3D scene datasets, then fine-tunes it downstream manipu- lation demonstrations. The finetuning procedure is identical to RVT training, while the pretraining introduces MAE decoder reconstruct all five virtual images from masked multi-camera tokens. This multi-view pretraining scheme produces 31)-awrre features robust to occlusions and viewpoint changes, enhancing manipulation performance and robustness environmental variations.
C COMPUTATIONAL EFFICIENCY AND CAPACITY ANALYSIS
Time Efficiency. Inference latency NVIDIA A800 GPU (batch size 512, averaged 20 trials) in Fig. 4 (a) shows the dynamic-view stream consumes ~ 1.8x more time than the static- view stream (2.48 vs. 1.37 x 10-3 min). This cost stems from the MVIT backbone (Fan et al. 2021I. which is computationally heavier but essential for heatmap precision (Li et al.) 2018)) No- tably, both streams remain faster than the RVT-2 action head (4.06 10-3 min), ensuring Cortical Policy achieves superior performance (+3.5% gain over RVT-2) without compromising responsive- ness. Compute Control. Success rate tracking (Fig. D shows stability after 50 epochs, confirming that performance gains stem from architectural design rather than extended training. Parameter Capacity Control. We further rule out the gains from mere model scaling by comparing against a "Deeper" (18-layer) RVT-2 baseline. As detailed in Table 7 Cortical Policy outperforms this heavier model (+2.6%) with fewer parameters and lower FLOPs, validating that the dual-stream mechanism, rather than model capacity, is the key driver of performance.
D VISUALIZATION OF GEOMETRICALLY CONSISTENT KEYPOINTS
Fig. 10 shows additional qualitative results of 3D supervision generation, demonstrating the viewpoint-consiser keypoint distributions across eight manipulation tasks. The figure organizes multi-perspective keypoint visualizations follows:
5

---

Published as a conference paper at ICLR 2026
Rows (top to bottom): Close Jar, Place Cups, Sweep Dustpan, Insert Peg, Push Buttons, Drag Stick, Screw Bulb, and Stack Blocks
Columns (left to right): Coarse stage (top, front, right views) followed by fine stage (top, front, right views)
E VISUALIZATION OF EGOCENTRIC RENDERING
Fig. 7 compares dynamic-view options for position-aware pretraining data: raw wrist camera views versus rendered views from dynamic virtual cameras. As can be end-effector positions in raw wrist camera views are fixed at constant pixel coordinates. By contrast, rendered views exhibit posi- tional variations, with shadows indicating regions occluded from physical wrist-mounted cameras. Apart from image samples, egocentric video examples are also included in supplementary material.
raw wrist camera view
rendered images
coarse stage
25
Frome:
Frame: 158
rendered view coarse stage
rendered images fine stage
25
Frome: 75
Frome: 138
rendered view fine stage
und-effector position
Figure 7: Comparison of dynamic egocentric views and rendered examples.
F ADDITIONAL FAILURE CASE ANALYSIS
This section provides deeper analysis of the RVT-2 failure in the "stack 2 blocks in between the bottles" task (Fig- D. investigating whether it stems from spatial reasoning deficiency, mode collapse, language misunderstanding. As shown in Fig. 8 RVT-2 exhibits distinct action patterns between the two stacking tasks. This behavioral diversity in a novel configuration indicates both the absence of mode collapse and RVT-2's ability adapt the tasks with different scenes and instructions. The failure, therefore, points to deficiency in the precise spatial reasoning required for successful placement the "in between" relationship.
G DETAILED RESULTS ON COLOSSEUM
In this section, we provide comprehensive results the COLOSSEUM benchmark (Pumacay al.) 2024), extending the analysis in Section We evaluate the same models from Table and Table (all trained on the original RLBench tasks) under a suite of unseen perturbations. These pertur- bations encompass changes to object properties (MO/RO-Color MO/RO-Texthn MO/RO-Size), Light Color, Table Color/Texture, Distractor, Background Texture and Camera Pose. Evaluations also include the RLBench Variations described in Table 4
Following the official COLOSSEUM protocol for zero-shot generalization, we evaluate RVT-2, Cor- tical Policy, and its variants on four tasks shared by RLBench and COLOSSEUM: drag stick, place wine, stack cups, and insert peg. Results are averaged over three independent trials. For a com- prehensive comparison,
include results of RVT (Goyal et al. (023) and PerAct (Shridhar al.) 2023) from the original COLOSSEUM paper (Pumacay et al. (024). The detailed per-task results across all perturbation types shown in Table Key observations include:

---

Published as a conference paper at ICLR 2026
Initial State
"Stack 2 blocks"
Initial State
"Stack blocks in between the bottles"
Figure 8: RVT-2 behavior comparison. (Top) In the basic task "stack 2 blocks", RVT-2 places the first block near the robot arm. (Bottom) In the spatial task "stack 2 blocks in between the bottles", RVT-2 attempts to place the first block near the bottles but fails due to imprecision.
Cortical Policy achieves the highest average success all four tasks (drag stick: 80.3%, place wine: 89.0%, stack cups: 76.8%, insert peg: 32.6%), demonstrating superior robustness to unseen scene configurations.
In tasks that heavily rely on spatial reasoning, such as "stack cups", our method achieves the high- est success rate (36.0%) under the combined "All Perturbations" setting, highlighting its superior robustness against geometric variations.
Ablation results underscore the critical role of the dynamic-view stream. Specifically, variant consistently outperforms B, with a notable margin of +9.3% in the challenging "stack cups" task.
Collectively, these COLOSSEUM results demonstrate the dual-stream architecture's effectiveness handling realistic environmental variations, showing that the dynamic-view stream the primary contributor to the observed generalization and robustness.
61
19-
Deed Aveenge Succees Rlaee
57.
Worst
Troining Epochs
Figure Success rate variations with training epochs for compute-control evaluation.

---

Published as a conference paper at ICLR 2026
Table 6: Success rates of different methods under various perturbations of COLOSSEUM
Task Name
Avg. Success
All
MO- Color
MO- Texture
RO. Texture
Models
Perturbations
Perturbations
Color
Size
PerAct
17.6
12
RVT
59.2
84
24
52
88
88
92
90.7.1.4
drug stick
Variant
73.4
85.3x550
0.0ko0
90.7a11
89.3a1.9
94.7a1.9
90.7a19
92.0ss0.
Variant
Variant
78.3
88.0..
0.huou
92.0.000
96.0ion
97.311.
92.0.n0
Quou
Ours
80.3
88.0at.
0,0k00
92.0sn0
96.0s000
100,0:a0
92.0ano
96,0s00
PerAct
3.7
0
RVT
57.4
60
12
72
40
72
36
RVT-2
88.0+13
place wine
Variant
85.1
94.71sm
16.0un0
82.7.5.0
90.711.9
96. 96.0uno
96.0uon
Variant
96.0a00
Variant
88.2
7
0.0uod
97.3-19
93.3:18
97.3xs8
96.0ion
Ours
89.0
100.0200
0.0s00
100,0an.
98.7a1.9
80.0ano
100.0x00
PerAct
0
0
RVT
13.3
0
12
RVT-2
96-0etn
4.0eo0
76.0an0
De00
stack cups
Variant
67.0
97.3x1
5,3
84.0-no
96.0io0
Variant
16.0a80
Variant
76.3
98.741
34.7418
86, 76.7
96, Deno
Ours
76.8
96.0u0.
36.0.00
88.0.n0
100.0.a0
PerAct
5.1
0
RVT
0
16
24
RVT-2
21.4
32.0ett
12.0e00
20 20.0g3.
26.7+1.
21.3e19
56.0e0
insert
Variant
28.4
36.0.0.
20.0+3.1
21.3-58
30.741.
28 28.0.n0
64.0.00
Variant
20.0.00
22.7.19
60.0.on
Variant
32.1
38.7a1.9
0.0a00
24.0an.
32.0a000
24.0ano
64.0.00
Ours
32.6
42.7.1.9
4.0.00
32.0.n0
36,0000
28.0mn
64.0000
Name
Task
Models
RO-
Color
Light
Table
Table
Distractor
Background Texture
RLBench Variations
Camera
PerAct
12
12
8
0
20
64
20
RVT
52
88
RVT-2
29.3-11
73.341
53.345.
46.7-1.
80.0+1.3
84.0v00
78.7-19
84.0-13
drag stick
Variant
8.0uko
88.0uo
64.0150
52.0.n0
82.7.18
84.0u00
84.0an0
96.0m00
Variant
8.0as.0
90.721.9
68.0e00
88 Ozn.
94.7
84.0e00
88.0.00
92. 22.0000
Variant
30.7.19
92.010.0
72,0100
90.7.11
76.0
S0.0ino
88.0
92.0.00
Ours
84.0a0.5
PerAct
12
8
0
0
4
8
8
RVT
88
52
RVT-2
84.0=a0
89.3
88.0e5.
86.7-19
84.0e57
88. Deno
89 3..
88.0e33
place
Variant
88.0.65
90.741.
89.311.
88.0.n0
89.311
89.311.9
92.0mn
89.311
Variant
76.0.n0
90.7a1.
90.7118
90.7a19
94.7a19
88.0a00
94.7a5.0
97 97.3
Variant
92.0-00
92.040.
92.0+a0
92.0-na
96.0-13
93.3
96.0.s
98.741
80.0as0
98.711.
PerAct
0
16
4
0
8
RVT
12
RVT-2
80.0at
24.0e00
64 04.00e
92, 92,0e00
68 Deno
62.7e119
stack cups
Variant
88.0+an
16.04a0
80.0-a0
68.0400
69 09.3us
65.3413
Variant
84.0umm
16.0un0
72.0.n0
84.0u00
80.0un0
Qunn
Variant
9a00
52,0e00
72.0an0
72.0s00
S4..aa3.
74.721.
4.0io0
84.0.0
96.010.
92.0us
PerAct
4
8
4
28
RVT-2
20.0an.0
12.0at.
4.0x00
20 20.0am0
17.3el.9
24.0e90
18.7a1.
16:0a90
insert pog
Variant
36.0-n0
1..0
20.0+3.1
24.0a57
20.0:33
26.7+1.9
26.7.19
16.0a0
Variant
36.0.s
28.0.on
22.7:5.
28.0
48.0
24.0i00
36.0an
16.0ion
Variant
38.7-19
37.3a1.
24.0290
28.0ano
50.723.
29.3x1.
38.7a19
20.0a00
32.0100
Table 7: Capacity-controoledd ablation study on RLBench.
Models
Parameter Size
Parameter Num
FLOPs
Avg. Success
RVT-2
277.3MB
72.7M
14.98 G
77.5
Baseline (Deeper)
596.6MB
156.4 4M
25.59
78.4
Cortical Policy (Ours)
551.9MB
144.7M
22.37 G
81.0
8

---

Published as a conference paper at ICLR 2026
(1) lup course
(2) Tront 6f course
(3) right Of cxarse
(4) lop@l lime
(5) front di line
(6) right line
(7) top@ coarse
(8) front coarse
(9) coarse
(10) top@
(II) front@fine
(12) righr@ fine
(13) @
(14) front coanse
right coanse
top fine
front@fine
(18) right fine
(19) top& coarse
(20) coarse
(21) right coarse
(22) top@
(23)
(24) right
(25) lop@l coarse
(26) front d'coarse
(27) right course
(28) top
(29) fromt tine
(30) right @f
(31) top cearse
(32) front moont@ccoaee cocsse
(33) right course
(34) top lime
(35) front @ tine
(36) right time
(37) top toprrcease
(38) @Fourec
(39) right course
(40) top time
(41) front Wine
(42) right time
(43) lop@@
(44) front @enarse
right coarse
top@fine
front&ffine
(48) right fine
Figure 10: Additional visualization geometrically consistent keypoints.


Reviews:
[{"review_id": "vkKlZPPByy", "reviewer": "Reviewer_yYi1", "summary": "This paper proposes a dual‑stream view transformer for language‑conditioned robotic manipulation. A static‑view stream adds a cross‑view geometric consistency objective guided by VGGT to learn 3D‑aware features; a dynamic‑view stream reuses an egocentric gaze model (GLC) to produce action‑oriented feature maps and saliency heatmaps from a wrist‑like camera. The two streams are fused to predict 6‑DoF actions. It shows SOTA performance on RLBench and real robot.", "questions": "1. What if just provide dynamic view  as a separate image to RVT-2?\n\n2. Can we extend dynamic view to gaze not just end effector position but also other useful objects?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear motivation to force 3D consistency and fuse dynamic cues. \n\nUseful ablations: removing the geometric loss drops performance; end‑to‑end fine‑tuning the gaze model underperforms freezing; and heatmaps matter for the dynamic stream.", "weaknesses": "The framing on Cortical policy is unnecessarily complicated. My understanding is that it produces saliency map about end effector position to get inductive bias. Unsure if we need to fine-tune from a gaze model. We could also just exact the effector location from robot forward kinematics and register on camera images, which seems to be an easy baseline that may perform similarly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvTmDpLsA9", "reviewer": "Reviewer_ynp7", "summary": "Visual robotic manipulation in unstructured environments is challenging, in part, due to poor spatial understanding in 2D RGB image encoders. Recent works improve performance of visual imitation learning policies by using either (1) an explicit 3D representation which is compute-intensive but effective, or (2) fusing features from multiple static camera-views in an implicit manner (e.g. view transformers) which has been shown to perform comparably to explicit representations but without the added computational costs. However, this leads to two key failure modes: inadequate spatial reasoning and dynamic adaptation failure. Inspired by human ventral (static) and dorsal (dynamic) visual pathways, the paper proposes Cortical Policy, a dual-stream view transformer that first encodes static and dynamic view information using two separate streams, and then fuses their respective features before making an action prediction. Beyond this dual-stream architecture, the paper also incorporates several representation learning techniques including a feature/view auxiliary loss that aims to improve 3D keypoint (produced by a 3D foundation model) consistency across views, as well as a pretrained visual backbone for dynamic view feature extraction (initialized from GLC, a human gaze estimation model). Experiments are conducted on RLBench and a real hardware setup, and experiments indicate that the proposed method (Cortical Policy) performs better than RVT-2, a strong view transformer baseline that only uses static views, upon which the proposed method is implemented.", "questions": "I would really appreciate it if the authors can address my comments in the \"weaknesses\" section above using written arguments and potentially additional experimental results (if applicable). I believe that most of my comments (e.g. robustness evaluation) can be addressed without substantial compute.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I believe that this paper studies a relevant and timely problem (imitation learning from multiple RGB cameras, and in this case a combination of static and dynamic views), and is likely to be of interest to the community. The problem is clearly defined and I believe that the shortcomings of prior work is described in enough detail for an unfamiliar reader to appreciate the technical contributions. The paper is generally well written and easy to follow throughout, although the method section is a bit verbose.\n- Simulation experiments are conducted on a variety of tasks from RLBench, which has become a common benchmark for imitation learning algorithms for robotic manipulation, particularly works that focus on multi-view policy learning. The authors compare their proposed method against a series of strong baselines, including RVT-2 upon which the proposed method is implemented. Benchmark results indicate that the proposed method, on average, performs better than this set of baselines across RLBench tasks, and also generalizes better to unseen scene configurations in a cube stacking task on real hardware.\n- The ablations in Table 2 are helpful for understanding the relative importance of the proposed changes. This is rather important as the proposed method can be viewed as a combination of different architecture, objective, and modality changes on top of RVT-2.", "weaknesses": "My initial assessment of the paper is fairly neutral. I believe that the paper and contributions are interesting, but I also do have some concerns that I would like the authors to address:\n\n- Since this paper appears to follow the PerAct experimental setup, I was a bit surprised to not see PerAct listed as a baseline. While I understand that this work focuses on implicit view fusion rather than the explicit 3D representation of PerAct, I do believe that the comparison would be useful to readers (it currently is not clear how they compare in terms of task performance).\n- The proposed method is rather complex and contains multiple new design choices on top of RVT-2. However, the ablations in Table 2 indicate that the majority of the performance improvements stem from the auxiliary feature/view consistency loss (RVT-2: 77.5%, +consistency loss: 80.1%, +everything else: 81.0%), which (to me) seems inconsistent with the paper's main claim that a dual-stream view transformer is the key to implicit 3D spatial understanding. I would appreciate if the authors can please clarify whether my understanding of this is correct.\n- The real robot experiment only considers a single task but with several distinct scene configurations such as object displacement. I believe the real robot results would be more convincing if it contained more task variety like in the simulated tasks. Additionally, it is not clear from the paper whether a single agent is trained and evaluated on all four task variations in a multi-task setting like in simulation, or if it is four separate agents (from Appendix B.2: *\"Each task collects 45 human-teleoperated demonstrations with placement variations\"* is the only information I could find on this, and it is a bit ambiguous wrt this).\n- If generalization / robustness is a motivating reason for better spatial understanding in view transformers, I am a bit perplexed where there seemingly is no such evaluation in simulation nor the real world. For example, it seems pretty easy to evaluate the trained agent on e.g. unseen scene configurations, especially with the existence of generalization benchmarks such as Colloseum (https://arxiv.org/abs/2402.08191) which are based on RLBench.\n- Lastly, I am not fully convinced by the failure cases provided in Figure 1. In particular, the first example \"Stack 2 blocks in between the bottles\" could indeed be explained as a failure to understand the spatial relationship between objects, but it seems equally likely that the culprit may be a mode collapse or a failure to understand the language instruction, especially given the very small number of tasks/instructions.\n\nIt is possible that some of my concerns above stem from misunderstandings. If so, I would appreciate it if the authors can clarify those aspects of their paper and commit to revising the text to make it more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XZPm0x2Csz", "reviewer": "Reviewer_8WZE", "summary": "The paper proposes Cortical Policy, a dual-stream view transformer for robotic manipulation inspired by the ventral (static) and dorsal (dynamic) visual pathways. The static-view stream enforces cross-view geometric consistency using supervision from a 3D foundation model to improve 3D spatial reasoning; the dynamic-view stream uses a wrist/egocentric camera and a position-aware, gaze-estimation backbone to guide action with motion cues. The two streams are fused in an action head to predict 6-DoF gripper poses, state, and collision flags. Experiments are conducted on RLBench and several real-robot tasks. Authors report that the approach outperforms recent view-transformer baselines and that ablations suggest benefits from both the geometric-consistency loss and the dynamic stream.", "questions": "Why adopt the dual ventral/static and dorsal/dynamic streams with VGGT cross-view geometric consistency and gaze/pose priors? Can you provide capacity- and compute-controlled ablations and stress tests showing each component is necessary and yields independent gains (not just capacity or pretraining effects)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper specifies supervision generation, the SmoothAP-based cyclic consistency loss $L_{cgc}$, the dynamic rendering/pretraining pipeline, and shows ablations isolating architecture, pretraining, and heatmaps.\n2. On RLBench, authors claim higher average success than RVT-2 and improved performance on spatial-reasoning and dynamic scenarios; they also include small-scale real-robot tests.", "weaknesses": "1. Despite an elaborate pipeline, the trajectory-learning advantage may be modest. Even in the authors’ table, some tasks see limited gains or regressions, raising the question of whether the architectural complexity is justified by the overall deltas.\n2. It’s not yet conclusive that the dorsal (dynamic) stream is the key driver of improvement; ablations show mixed patterns, and the net gain over a strong static baseline can be small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTJHO4lUSQ", "reviewer": "Reviewer_qJao", "summary": "To address the issue that static-view observations in robotic 3D imitation learning cannot fully capture dynamic spatial information, and inspired by the dual-stream mechanism of spatial perception in the human cerebral cortex, this paper proposes Cortical Policy — a method based on view transformers that jointly encodes static 3D observations and dynamic world observations.\n\nThe model enforces geometric consistency alignment between the two observation streams in the feature space, thereby enhancing the understanding of the physical geometry of the world and improving the action generation capability of the policy model.\n\nThe authors first analyze the cortical principles of human visual-motor control, pointing out that in addition to the ventral stream responsible for reasoning about spatial scenes, the dorsal stream for dynamic understanding is equally important and mutually complementary.\n\nInspired by this, they argue that 3D imitation learning should also incorporate a dynamic observation stream to enhance dynamic spatial perception.\n\nFor the static stream, the authors introduce a cross-view consistency keypoint prediction mechanism, using VGGT to predict shared keypoints across multiple viewpoints and supervising the model to align representations of these keypoints across different views, thereby achieving cross-view geometric consistency.\n\nFor the dynamic stream, they propose a dynamic prediction mechanism for the GLC model, enabling it to accurately predict the end-effector pose after an action is executed. These predictions provide valuable features for subsequent policy learning.\n\nThrough feature extraction and fusion from both streams, the proposed model achieves significant performance improvements on the RVT-2 backbone and attains state-of-the-art results on both simulation and real-robot tasks.", "questions": "The tasks chosen in the RLBench simulator mostly rely on static observations to achieve good performance. Would it be possible to evaluate the proposed model on more challenging or dynamic simulation environments to better highlight the contribution of the dynamic stream?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper introduces a dual-stream observation mechanism inspired by the human brain, which substantially improves the model’s ability to perceive dynamic changes in the environment during task execution and addresses limitations of previous work.\n2.\tBoth static and dynamic streams emphasize cross-view feature alignment, and an efficient GLC-based dynamic prediction mechanism is designed to guide policy action generation.\n3.\tThe proposed dynamic perception mechanism has strong potential for mobile manipulation and complex dynamic scene tasks, suggesting promising directions for future research.", "weaknesses": "1.\tIn the training of the static-view stream, the proposed cross-view alignment mechanism aims to unify geometric features of keypoints across viewpoints. However, it remains unclear whether this alignment may interfere with the encoder’s original spatial semantic understanding — which is critical for the static stream. This issue warrants further investigation.\n2.\tThe paper does not evaluate the model’s generalization ability on new or unseen tasks (zero-shot transfer), leaving the claimed “cortical” system’s abstract reasoning capacity unsubstantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)