Title: Hierarchical Representational Transformations of Working Memory in Brains and Machines
Primary area: applications to neuroscience & cognitive science
Keywords: working memory, representational geometry, cortical hierarchy, learning objectives, recurrent neural networks

Abstract: Working memory (WM) maintains past inputs while processing new ones, yet how representations transform between encoding and retrieval remains unclear. Clarifying whether these representations are sustained through stable coding formats, dynamically updated subspaces, or their interplay is key to uncovering the mechanisms of WM. To address this, we combined high-resolution 7T fMRI from the Natural Scenes Dataset with recurrent neural networks (RNNs) trained on a naturalistic 1-back task. Using representational similarity, cross-decoding, and subspace geometry analyses, we directly compared rotational and non-rotational transformations between encoding and retrieval in brain regions and model layers. Our analyses revealed convergent evidence for a mixture mechanism of WM coding for encoding and retrieval information: early visual regions (V1–hV4) underwent large representational changes across encoding to retrieval phases, including both rotational and non-rotational transformations. Whereas higher-order regions in the prefrontal cortex (FEF, dlPFC) were more stable. Applying the same analyses to models showed partial alignment, but critically depended on the learning objective and the recurrent architecture. We examined two different encoder architectures, ResNet and Vision Transformer (ViT), each trained with supervised and self-supervised learning objectives. Supervised encoders preserved a hierarchical layer dissociation paralleling the cortical gradient in both rotational and non-rotational transformations, while self-supervised encoders diverged especially in reduced rotational alignment with the cortical hierarchy. Among recurrent architectures, gated architectures (GRU, LSTM) best reproduced the brain-like mixture of subspace rotational transformation. Taken together, these results established hierarchical shifts between flexibility and stability in WM representational transformation in both humans and machines, with supervised learning objectives combined with gated recurrent dynamics most closely resembling human WM mechanisms.

TL;DR: This work shows working memory (WM) in humans and machines follows a shift from dynamic to stable representational transformation along the hierarchy, with gated recurrent dynamics and supervised learning best capturing human-like WM mechanisms.