PEEK: Guiding and Minimal Image Representations for Zero-Shot
  Generalization of Robot Manipulation Policies
Authors: name: Jesse Zhang, name: Marius Memmel, name: Kevin Kim, name: Dieter Fox, name: Jesse Thomason, name: Fabio Ramos, name: Erdem Bıyık, name: Abhishek Gupta, name: Anqi Li

Robotic manipulation policies often fail to generalize because they must
simultaneously learn where to attend, what actions to take, and how to execute
them. We argue that high-level reasoning about where and what can be offloaded
to vision-language models (VLMs), leaving policies to specialize in how to act.
We present PEEK (Policy-agnostic Extraction of Essential Keypoints), which
fine-tunes VLMs to predict a unified point-based intermediate representation:
1. end-effector paths specifying what actions to take, and 2. task-relevant
masks indicating where to focus. These annotations are directly overlaid onto
robot observations, making the representation policy-agnostic and transferable
across architectures. To enable scalable training, we introduce an automatic
annotation pipeline, generating labeled data across 20+ robot datasets spanning
9 embodiments. In real-world evaluations, PEEK consistently boosts zero-shot
generalization, including a 41.4x real-world improvement for a 3D policy
trained only in simulation, and 2-3.5x gains for both large VLAs and small
manipulation policies. By letting VLMs absorb semantic and visual complexity,
PEEK equips manipulation policies with the minimal cues they need--where, what,
and how. Website at https://peek-robot.github.io/.