Title: Task representational dynamics for compositional generalization
Primary area: applications to neuroscience & cognitive science
Keywords: compositional generalization, recurrent neural networks, representational dynamics, representational dimensionality, initialization scaling, representation learning

Abstract: The ability to generalize compositionally is central to intelligent behavior.
While recent work shows that networks can generalize compositionally under certain conditions, many studies focus on simple compositional tasks, such as those that are purely linguistic (or unimodal), or those with no temporal structure.
Here we investigate how feature-specific representational dynamics shape compositional generalization in recurrent neural network (RNN) models during tasks with evolving temporal structure. 
We trained RNNs on the Concrete Permuted Rules (C-PRO) task, a context-dependent compositional task that requires integration of information across task phases, and is commonly used to study compositionality in humans.
In particular, we assessed how different learning regimes induced generalization and representational dynamics.
We systematically varied model initializations to generate RNNs that exhibited a wide range of compositional generalization performance, spanning from 38\% to 90\%.
Analysis of high-performing models revealed nontrivial temporal dynamics of task representations, highlighting the importance of selectively engaging the right features at the appropriate task phase for generalization.
Our findings reveal that successful compositional generalization requires the orchestration of structured intermediate representations that are dynamically composed, resulting in complex, feature-specific representational dynamics.

TL;DR: Compositional generalization in temporally structured tasks requires orchestrated feature-specific dynamics across context, stimuli, and response task phases.