Sensorimotor adaptation and cue reweighting compensate for distorted 3D shape information, accounting for paradoxical perception-action dissociations

Reach-to-grasp movements can show surprising accuracy even when the perceived 3D shape of the target is distorted. One explanation of this paradox is that a specialized “vision-for-action” system provides accurate shape estimates by relying selectively on stereo information and ignoring less reliable depth cues like texture and shading. However, the key support for this hypothesis has come from studies that analyze average behavior across many visuomotor interactions where available sensory feedback reinforces stereo information. The present study, which carefully accounts for the effects of feedback, shows that these interactions are actually planned using the same cue-combination function as perception, and that apparent dissociations can arise due to two distinct supervised learning processes: sensorimotor adaptation and cue reweighting. In four experiments, we show that (1) when a distorted depth cue biases perception (e.g., surfaces appear flattened by a fixed amount), sensorimotor adaptation rapidly adjusts motor outputs to compensate for this constant error. However, (2) when the distorted depth cue is unreliable, leading to variable errors across a set of objects (i.e., some depths are overestimated, others underestimated), then the relative weighting of depth cues is gradually adjusted to reduce the misleading effect of the unreliable cue, consistent with previous studies of cue reweighting. In our third experiment, we show that (3) cue reweighting involves correlated changes in perceptual judgments and motor behavior, indicating a close functional link between perception and action, and that (4) cue reweighting occurs only in response to an unreliable depth cue, and not in response to a biased cue, presumably because movement errors caused by biased cues are rapidly resolved by sensorimotor adaptation. Taken together, these findings emphasize the mutual dependency of perception and action, with perception directly guiding action, and actions producing error signals that drive motor and perceptual learning. Author summary When interacting with 3D surfaces and objects, sensory feedback is available that could improve future performance via supervised learning processes. In four experiments, we confirm that natural visuomotor interactions with 3D objects lead to sensorimotor adaptation and cue reweighting, two distinct learning processes that are uniquely suited to resolve errors caused by biased and noisy processing of visual cues to depth. Whereas sensorimotor adaptation appears to shift motor outputs without altering 3D shape perception, we show that cue reweighting has correlated effects on perceptual judgments and motor behavior. The speed and flexibility of these two forms of learning provides a simple alternative explanation of why perception and action are sometimes found to be dissociated in experiments where some depth cues are consistent with sensory feedback while others are faulty. Rather than having a specialized “vision-for-action” system that relies preferentially on stereo information, as suggested in some current theories, the brain appears to combine multiple depth cues in the same way for perceptual judgments and visuomotor interactions. We propose that the sensory-prediction errors obtained during these interactions, which are widely recognized to drive sensorimotor adaptation, may also be responsible for producing the more gradual perceptual shifts associated with cue reweighting.