Decoding the Enigma: Benchmarking Humans and AIs on the Many Facets of Working Memory
Authors: Ankur Sikarwar
Mengmi Zhang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: working memory
neuroscience
cognitive science
RNNs
transformers
behavioral
neural
benchmark
TL;DR: We investigate and compare various behavioral and neural working memory benchmarks between humans and AI.

Abstract: Working memory (WM), a fundamental cognitive process facilitating the temporary storage, integration, manipulation, and retrieval of information, plays a vital role in reasoning and decision-making tasks. Robust benchmark datasets that capture the multifaceted nature of WM are crucial for the effective development and evaluation of AI WM models. Here, we introduce a comprehensive Working Memory (WorM) benchmark dataset for this purpose. WorM comprises 10 tasks and a total of 1 million trials, assessing 4 functionalities, 3 domains, and 11 behavioral and neural characteristics of WM. We jointly trained and tested state-of-the-art recurrent neural networks and transformers on all these tasks. We also include human behavioral benchmarks as an upper bound for comparison. Our results suggest that AI models replicate some characteristics of WM in the brain, most notably primacy and recency effects, and neural clusters and correlates specialized for different domains and functionalities of WM. In the experiments, we also reveal some limitations in existing models to approximate human behavior. This dataset serves as a valuable resource for communities in cognitive psychology, neuroscience, and AI, offering a standardized framework to compare and enhance WM models, investigate WM's neural underpinnings, and develop WM models with human-like capabilities. Our source code and data are available at: https://github.com/ZhangLab-DeepNeuroCogLab/WorM

Decoding the Enigma: Benchmarking Humans and AIs on the Many Facets of Working Memory
Ankur Sikarwar and Mengmi Zhang
Deep NeuroCognition Lab, Institute for Infocomm Research (I2R), S Singapore
Center for Frontier Al Research (CFAR), Agency for Science, Technology, and Research (A*STAR), Singapore School of Computer Science and Engincering, Nanyang Technological University, Singapore Address correspondence mengmi@@12ra-star.cdu..
Abstract
Working memory (WM), a fundamental cognitive process facilitating the temporary storage, integration, manipulation, and retrieval of information, plays vital role in reasoning and decision-making making tasks. Robust benchmark datasets that capture the multifaceted nature of WM are crucial for the effective development and evaluation of AI WM models. Here, we introduce a comprehensive Working Memory (WorM) benchmark dataset for this purpose. WorM comprises 10 tasks and a total of I million trials, assessing 4 functionalities, 3 domains, and 11 behavioral and neural characteristics of WM. We jointly trained and tested state-of-the-art recurrent neural networks and transformers all these tasks. For comparison, we also include human behavioral benchmarks. Note that all computational models were never trained with any human data or behavioral biases; yet, these models remarkably replicate some characteristics of WM biological brains, such as primacy and recency effects. Moreover, we performed neural population analysis on these models and identified neural clusters specialized for different domains and functionalities of WM. Not all computational models exhibit a strong alignment with all human behaviors. Our experimental results also reveal several limitations in existing models to match with working memory capabilities of humans. This dataset serves as a valuable resource for communities in cognitive psychology, neuroscience, and AI, offering standardized framework to compare and enhance WM models, investigate WM's neural underpinnings. and develop WM models with human-like capabilities. Our source code and data available link
Introduction
Working memory (WM) defines core cognitive process enabling temporary storage, integration, manipulation, and recall of information. vital for many downstream tasks involving reasoning decision-makin
Understanding and modeling WM significant implications for both neuroscience and Al research. neuroscience and cognitive science, extensive investigations have been conducted to unravel the underlying mechanisms of WM. The non-exhaustive list of works involves (1) identifying brain regions such as the prefrontal
57 46
S4,
[50) 26], and language comprehension S I
cortex [1] 51 37), hippocampus [6] [55], and parietal cortex (28 32 47] as key players in WM
processes; (2) exploring the role of WM capacity in cognitive processes such as selective attention
48). and executive control [19 7 and (3) investigating how WM abilities differed in individuals
of different ages (20] 53 intelligence levels 12), and neurological disorders
23
However, despite significant progress in individual subdomains, there remains notable gap in the broader study of WM and its relevance for the AI community, To date, research efforts in neuroscience, and cognitive science have mostly focused on specific aspects of WM or individual
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

memory tasks. There is lack of systematic, integrative, and quantitative exploration that covers multiple tasks, encompasses various functionalities and domains of WM, and provides systematic benchmark for evaluating both humans and Al models in these memory tasks.
In this work, we aim to address this gap by establishing general framework study WM. To capture the multifaceted nature of WM due to its complexity, flexibility, and variability, we include 10 WM tasks and curate million trials. Each of the state-of-the-ar recurrent neural networks and transformer models is jointly trained and tested on these tasks with the following goals: (I) to assess model behaviors, such as set size effect and retention interval; (2) to investigate neural populations, such as task-speciaa sseciilei neural clusters and neural correlates of executive control; and (3) to examine and compare performance across multiple WM models when performing different memory functions: storage, integration, manipulation, and supervision. We include human performance on the WM tasks for behavioral comparison with AI models. In the experiments, we observe that AI models replicate some human-like characteristics of WM; however, we also identify several limitations in the existing WM models, such as disparities between human and model behaviors and inferior model performances in tasks that humans excel at. Through this interdisciplinary exploration, we strive to enhance our understanding of WM processes, uncover novel insights into the cognitive architecture of human WM, and propel the development of Al systems with more robust and human-like WM capabilities.
Main contributions are highlighted:
1. We introduce WorM, comprehensive large-scale WM benchmark dataset, covering 10 WM tasks and containing million trials, encompassing 4 functionalities and 3 domains of WM.
2. We introduce evaluation metrics for all our tasks and establish general methodology to jointly train, study, and benchmark WM models on all tasks.
3. We explore 8 WM behavioral benchmarks and find that recurrent neural networks are more closely aligned with humans, while transformers exhibit limitations in replicating human behavior.
4. We examine the neural population of WM models. Our analysis reveals apec-sped neural clusters and uncovers neural correlates of executive control in WM models.
2 Related Works
Working memory (WM) has been a topic of interest in cognitive psychology and neuroscience for
decades. Numerous studies 40 24 39 35 61
3 g have investigated different behavioral effects and
in developing computational models of WM that can explain and approximate human behavior on WM
their underlying mechanisms and neural correlates. There has also been an interest [14
42
36
43
tasks. Several WM tasks [16] 54 1 35 41 31 have been proposed to evaluate WM performance
of humans. For example, the N-back task is a widely used WM task that requires participants to remember sequence of stimuli and respond when stimulus matches the one presented N trials earlier. It is important to emphasize that the majority of these prior works have typically focused on isolated aspects of working memory evaluated a single working memory model with restricted range of working memory tasks. In contrast, our contribution involves the development of comprehensive and quantitative benchmark that encompasses 10 distinct working memory tasks, spanning four functionalities and three domains of working memory, for the evaluation of both human performance and AI models.
In the field of AI, WM has also attracted considcrable attention. AI researchers have aimed to develop computational models that emulate and augment human-like WM capabilities. Various memory architectures [0 27 25| 62 [6) have been proposed, such as neural networks with memory cells, recurrent neural networks, and memory-augmented neural networks. These models have demonstrated promising results in specific WM tasks, such as copying, sorting, and memory recalls, as well as real-world applications [22 B3) such as navigation in naturalistic environments, video recognition, and language processing.
Recently,
45 established a comprehensive working memory (WM) benchmark to aid in the development of WM theories by collecting empirical findings from previous human studies in wide range of WM tasks. Different from their work, in this paper, we present large-scale WM
2

---

A Spatial Free Recall (SFR)
Change Detection (CD)
Visual Item Recognition (VIR)
locations of red squzres.
change.
before.
Visual Serial Recall (VSR)
Visual Serial Recognition (VSRec)
Complex Spun (CS)
Spotial Visual
Recall serial ooder.
shown before seriall order
color blue red?
color blue reaf!
dircetion serial order.
Visual
Spatial Coordination (SC)
Spatial Integration (SI)
emergent symmetric about versical axis?
symmentic about vertical axis?
symmetric about vertical extis?
Recall the pattern serial order.
final integrtted poltem??
Spatial Memory Updating (SMU)
Spatial Task Switching (STS)
Presentation phase
Probing phuse
phase
Distractor plase
Memury updating phase
Rocall the location of marker the red square.
left right the gnd?
markee botiom of the gad!
Task switch phase
Time
Figure Schematic illustration of all 10 working memory tasks. Each panel illustrates the schematic for different task. In each panel, the arrow pointing from left to right denotes the progression of time. Overlapping frames indicate multiple time steps whereas single frames represent single time step in the trial. In F, we further divide the task into four types. Two types are shown here. See Sec. 3 for detailed description each task.
benchmark dataset to bridge the gap between cognitive psychology and the Al field, allowing modern AI models to be trained on such WM tasks for direct behavioral and neural comparison.
3 Psychophysics Experiments in WorM
Phase definitions in WM experiments. In this section, we refer our 10 WM tasks as 10 experiments. A WM experiment often consists of several but not necessarily all phases (Fig, []] described below: & Presentation phase (P present)): sequence of stimuli remember is presented sequentially. Retention phase (P retentt): A grey blank image is presented for period of time, during which the remembered patterns in Ppresent have be maintained in the memory. * Probing phase (P prohe): Responses to a series of questions asking recall certain aspects of the memory content are required. Sometimes, there will be no questions but free recalls are required. * Distractor phase (P (Pdistractor): ): A series of other distracting tasks are presented in this phase to interfere with the original WM experiments. Memory updating phase (P (Pudatte:: ): A series instructions to update the memory content given. Task switching phase (P ewiitch): A cue image is presented to signal the transition between pair of tasks.
Time steps in WM experiments. The time duration of each phase in human psychophysics experiments is often measured in milliseconds (ms). However, computational WM models do not have the notion of time. We establish mapping between the time step in WM models and ms human psychophysics experiments for all the phases of all the experiments (see Appendix A2.2 for mapping details). For consistency, we describe all the main text with time steps I. We introduce to denote the total time steps for each experiment and Tphose to denote total time steps of each phase, where phase could be from any of the phases introduced above. For example, T preseett

---

Visual
Spatial
Integration
Storage
Set size (2)
CS
Features & Conjunction (1)
VIR
STS
SI
VSRec
SFR
Rotontion Interval
CD
Primacy Recency (3)
SC
SC
VIR
VSR
Neural population (10)
VSRee
SFR
CD
Task supervision (1) List length (6)
CS
SMU
Task selectivity (10)
VSR
SMU
STS
SI
Domain conflict (1)
Temporal
Memory resolution (1)
Manipulation
Supervision
Cognitive load (1)
(a) Memory domains
(b) Memory functionalities
(c) Memory characteristics
Figure 2: Wor. covers a wide spectrum of working memory (WM) tasks encompassing
domains, 4 functionalities, and 11 characteristies. For all ten tasks, we categorize them based on their objectives in studying different (a) memory domains, and (b) memory functions. See Fig. for task acronyms. In (c), we present word clouds representing the studied memory characteristics. Larger fonts indicate that more tasks (the exact number in brackets) produce results in this category.
indicates that there are time steps for the presentation phase and lpresent 1 indicates the first time step of the presentation phase.
Nomenclatures in cognitive science and neuroscience. In human psychophysics experiments 13 299, researchers often use "length list", denoted as L, to indicate the total number of stimuli presented during each P present, and "serial position" to indicate the position of the 1th stimulus in L. For WM models, each stimulus is presented at each time step of Prresent. Thus, L = Trresent and lpresent = L denotes the 1 serial position during Ppresent. Following psychology and neuroscience literature B3), also define "set size", denoted S, to be the total number of items that need to be held and manipulated in working memory at a given time step. For all the experiments, we use these terms interchangeably, and we also use "agents" to indicate that the test subjects can be both humans and computational WM models.
Introduction to ten psychophysies experiments
We introduce 10 psychophysics experiments on Working Memory (WM) from cognitive science and 41
44 38). To facilitate comparisons with computational models, we directly copy the human behavioral data collected from these existing works our paper. Based on WM functionalities these experiments entail (Fig. 2b). we categorize them into four functional groups: (A) memory storage, (B) memory integration, (C) memory manipulation, and (D) memory supervision. We also show the overview of the three WM domains (visual, spatial, and temporal) that each experiment involves in Fig. 2a See Appendix A1 for experimental details.
neuroscience [13 35 29 60
52
Experiment A1 Spatial Free Recall (SFR) [3] (Fig. [A). At every (presents a randomly chosen square on green grid of size 10x 10 turns red. In P probe: blank grey image is shown. The task for the agent is to remember all the previous spatial locations where the squares turned red and provide a 100-way multi-label classification response.
Experiment A2- Change Detection (CD) (35] (Fig. IB). In P Prrseenss a memory array is shown, consisting of S bars with varying attributes such as color, orientation, size, and the presence/absence of a gap in each bar. Followed by sequence of blank grey images in Pretent- the agents are tested with probe array in P prrabe: which can be identical to the memory array or have single bar that differs along one feature dimension. The task for the agents is to make a binary prediction about whether there is change or not given the test array.
Experiment A3 Visual Item Recognition (VIR) 29 (Fig. 1C). In Ppreeent. a sequence of distinct 6 x 6 matrix patterns is displayed. In Pretent. series of blank grey images is shown. In P Pprobe. probe image is presented, consisting of two patterns side-by-side. One pattern matches a previously shown pattern, while the other is distractor. The agents must perform binary classification task, where 0 represents the left pattern and 1 represents the right pattern.
Experiment A4 Visual Serial Recall (VSR) 4 (Fig, ID). In P Prresent. sequence of L matrix patterns sequentially presented. These patterns consist of a 6 x 6 grid with half of the cells filled in green. In P Prrate. probe image is displayed, containing all I. patterns presented during P Present- Agents perform 9-way classification task at each lprobes recalling the matrix patterns seen at ppresent in scrial order. See Appendix A3.1 for a language-baser variant of this experiment.

---

Experiment A5 Visual Serial Recognition (VSRec) 44 (Fig. TE). P present is exactly the same as P Presenn in the VSR experiment. Differences were introduced during Pyrobe- At each fprabes target pattern from P present and distractor pattern were presented together. Distractor patterns were distinct from the initially presented matrix patterns and differed from the target pattern by n cells, where n. denotes the -attern-dittractor difference. Agents performed binary classification task, with O indicating the left pattern and indicating the right pattern.
Experiment A6 Complex Span (CS) (60) (Fig. Te). The experiment includes two types of P Present and P probe. as well two types of P Piistractor- The first type involves memorizing visual patterns and recalling them in order (visual). (II) The second type involves memorizing spatial patterns of ball movements and recalling their directions (spatial). Two types of ddietractor are introduced to interfere with memory. (III) color discrimination tasks (visual) and (IV) spatial symmetry discrimination tasks (spatial) are used as distractors. Four variations of experiment conditions are considered: visual storage visual distractor, spatial storage + visual distractor, visual storage + spatial distractor, and spatial storage + spatial distractor.
Experiment B1 Spatial Coordination (SC) [41] (Fig. [G). P Ppresent involves 10 x 10 grid where one cell is highlighted in green each prreeett In P Probees agents are presented with a blank gray image and asked to perform binary discrimination task to determine whether the pattern formed by integrating all the patterns in P Ppreeent is symmetric about the vertical axis or not.
Experiment B2 Spatial Integration (SI) [52) (Fig. TH). Ppresent involves sequentially presenting 4 4 grid, eventually completing a multi-segment figure. The number of
partial line drawings line segments in the partial drawings determines the number of integration operations required, also equivalent L. In P. probe: agents perform binary classification task to determine if the given figure matches the mentally integrated figure from P present-
(Fig. DD. In P present a stimulus with green squares is presented, each containing a red marker one of the nine possible locations on 333 grid. P Pupdate involves arrows indicating the direction for mentally updating the marker's location. In Pprote. agents perform a 9-way classification task to recall the marker's 's location in the highlighted square.
Experiment C1 Spatial Memory Updating (SMU) 44
Experiment D1 Spatial Task Switching (STS) [38 (Fig. D... In P saritch, a cue image indicates the discrimination task for subsequent P Prrobe- There are two tasks across which to switch: top-versus- -oersus and left-versus-rihht At every Pprabes a red marker appears in a x grid, and agents switch between the two tasks based on cues presented during P swwitch.
Experimental trial splits for computational WM models. For every WM psychophysics experiment introduced above, we generate 86,400 trials for training. 9,600 trials for validation, and 9,600 trials for testing by following ratio of 9:1:1. These trials are distributed uniformly among various experimental conditions for individual experiments.
4 Computational Models of Working Memory
Joint training and testing regime. There exist multiple approaches to train and evaluate computational working memory (WM) models using our dataset, such "train on many; test on one" and "train on one; test one". However, our aim here is not to exhaustively benchmark WM models across all possible training and testing regimes. Instead, we aim to lay the foundation for systematic and quantitative methodology to comprehensively study the multi-facets of WM. In this regard, we propose a joint learning paradigm where we simultancously train and test different models on all conditions of all ten WM tasks. This approach allows to explore the intricate interactions and inter-dependencies among the different WM tasks, capturing the complex nature of WM performance. For comparisons with the joint training regime, we also conduct single-task training on the VSR task. See Appendix A3.3 for more details.
Architectures for working memory models. We investigate several state-of- thethe-rrtttttt variations of recurrent neural networks, including vanilla recurrent neural networks (RNNs), Gated Recurrent Units (GRUs), and Long Short-Term Memory (LSTM) networks I0 277. Furthermore, considering the recent success of Transformers (59], we also include vanilla transformer-basee encoders (TRF). Below, we introduce details of our model architectures (Fig. 3).
5

---

Pobled Image
Loss
WM Modal
(INN GRU STM "hondleree,, shanol)
CNN
NaN
Sopuener Saima
CNN Enooder (Shared)
JVA
Liscu; Clasifie: CTack Specific, Stared over line steps)
Loss
Learrable Enthedding (shored)
Time
NaN
Loerable Pusitoral Emtheiiing (For Tnneirmmn only, dumms)
CNY
NaN
Sheel Panmmate
Baicked Iepui
Ratched and Madeed Thrryts
Figure 3: Overview of Working Memory (WM) models. All WM models take batch of stimuli in sequences from I==1 to I=- as inputs and predict the task-specific responses at every single time step These responses
compared against the batched and masked ground truths with classification losses. The gradients are only pack-propagated based on the actual target values. For time steps without responses required (denoted as "NaN"), the predicted responses from the linear classifiers not computed, and thus, no gradients are back-propagated. See legends for box notations. "shared" refers to learnable parameters shared jointly over tasks except for "lask-specife shared" in linear classifiers, which refers to learnable weight parameters shared across time steps within the same task but different over tasks. See Sec. for model details.
Feature Extraction. At each time step of trial, the visual stimulus 4 comes as 32 32 x tensor which is then fed to a 4-layer 2D-convolutional network (2D-ConvNet). See Appendix A2.1 for the 2D-ConwNet architecture. The output feature maps from 2D-ConvNet are then flattened to vector of dimension K and linearly projected to feature representation F, of dimension D using weight matrix E R RxD, Different trials can be of different lengths, therefore we pad all the trials with blank images to ensure that all trials have consistent length of 20.
Task embeddings. In addition to extracting stimuli features, also introduce learnable task-specific embeddings M E IR 11x11 informing the network about the task identity during the joint training. At every & of each task, the corresponding task embedding is concatenated with the stimulus feature representation F; from 4r, resulting in a (4m+))-dimensional representation A,. A, is further fed to various WM models introduced below for further processing. See Appendix A2.1 for details.
As we have RNN-based networks and transfommer-based networks process stimuli sequences, introduce their model designs separately, For RNNs, GRUs, and LSTMs, we use only one recurrent layer. The recurrent networks take (1i+D)ddimensional representation A, above as inputs for each t. We define the capacity of these models C' as the number of hidden units in this recurrent layer. At every i, the recurrent networks output hidden representation h2 of dimension C used for predicting task-specific responses.
Encoding Time-dependent Representations.
For TRFs, we introduce extra learnable positional embeddings P E 20 (I++ which is shared across all tasks. Each row of P indicates "time step". At tth time step, P, incorporated into stimulus feature
A, via element-wise addition. We denote this positional embeddingg-mddulae representation as ATRF; which is then fed to the TRF. We include two standard transformer encoder blocks in TRF We define the model's 'ss capacity CTRF as the dimension 14 + 1 of the stimulus feature vector A,. Additionally, to simulate a more realistic scenario where the TRF model can only rely on past context to make predictions, we apply masking in the self-attention mechanism to prevent the model from accessing future information. We take the output HR,RF of TRF corresponding to the ghh time step and use it for predicting task-specific responses
Response Generation. The output of a WM network hú or M is fed to a task-specific linear classifier with weight matrix O task (14-D)xUsusa, for generating final probability distributions over fixed set of all possible response choices in the corresponding task. Note that Otask is shared across all T time steps within the same task, but it is independent over different tasks. For example, in the VSR task, during Prooee, the WM model has to predict which pattern among the 9 shown patterns is the target pattern. this case, Utask = 9, and the linear classifier outputs a probabilistic distribution of dimension 9, and the choice with the largest probability is the final response selected by the model.

---

A
LSTM-256
18TN
C
ESTM 1604
LSTe 8 5
Lnlloogit Let 4
-0
Spatial Ruaral
Top-1 Accuroey
0.9
Top-1 Ancuracy
0.9
Top-1 Aosurecy
0.8
Wical
0.8
Lnd 7
0.8
0.6
Led Logeto
0.7
1.0
Human
0.7
0.1
0.6
D.6
0.6
0.2
M.B
0.5
2.
6 7
8.9 Sertal Position
233
6.6
0.5
no-
Serial Position
Cognitive Load
1.0
1.0
L8 TMM--99
F
Accuracy
0.4
0.5
0.8-
L6TH 2x1
GRI-256
LATH
GRC 1034
100450 26
Top-1 Accuracy
Paisaa Baaaee DEE
Paiata Blccne D4E,
Deterr 50E.
Human
Top-1 Accuracy
0.9
DNB--8 Rixe
DNN--R Cop
mNd--4d coejinction
25 Coke
0.2
1004-1024
0.8
Mlanet Cops
Planee Suc
Phanen Cuejenstion
0.0
5
Herer
2 4
0.7
6 8
10
Set S120
Serial Position
Set Size
Figure 4: Performance benchmarks and behavioral analysis for working memory models and humans. We present the behavioral accuracy as a function of (A) list lengths L for LSTM-256 in VSR task, (B) retention interval in VIR task, (C) memory domain conflicts for LSTM- 1024 in CS task, (D) set sizes S in SMU task, (E) memory resolution n in VSRec task, and (F) number of features or conjunctions of features per item in CD task. See Sec. 3 for all the task introductions. See Sec. for the analysis of these results.
Training Details. Practically, the WM models output responses for all T steps. However, training supervision with corresponding losses is only applied on the set time steps where actual responses are required. All model parameters, including the feature extractors and WM networks, are initialized using Xavier initialization and trained from scratch. See Appendix A2.1 for more training details and Tab. A1 for hyper-parameter configurations.
Comparison with humans. To quantitatively compare computational models and humans on behavioral benchmarks, we introduce two types of scores: the slope difference score (A) and the average accuracy difference score (B). We presented the experimental details and the results in Appendix A2.3 We conducted statistical tests on the results. We found that the majority of the models are more consistent with humans than the chance model; however, there still exists performance discrepancies among different models and performance gaps between these models and humans.
5 Results and Analysis
We analyze multiple memory characteristics (Fig. 2ch individually for the corresponding tasks for different WM models. We use "model-C" to refer to the WM model of memory capacity C. For example, LSTM-128 refers to the model with LSTM of memory capacity C = 128 as the
backbone. As benchmarks, we copy the exact human performances here from 13 35 29
60
41
44 38) for direct comparison with WM models. Importantly, the WM models were not trained on human data or influenced by human biases. Therefore, our focus is not on comparing absolute performance disparities between humans and WM models. Instead, we aim to examine their qualitative trends across different conditions. Due to the extensive combinations of models and tasks, we present a condensed selection of results for some models in this section. See Appendix ( [A4 for the expanded result analysis with all model variations in these experiments. For fair comparisons among different model architectures in terms of overall WM performance, we compare different architectures with either the same memory capacity sizes C or the same number of network parameters. In both comparisons, we observe that recurrent networks like LSTMs and GRUs perform better than transformers on different working memory tasks (refer Appendix A4 for more details).
52
A. Primacy and recency effect across list lengths is an emergent phenomenon. We report the 1 accuracy as a function of serial positions across various list lengths L for LSTM-256 in the VSR task (Sec. 3 Fig. [A). We make several observations. Firstly, as L increases, the memory load in WM also increases, resulting in a decrease in overall accuracy across different list lengths (indicated by the varying colors of the lines, from blue to brown). Secondly, our findings in WM models align with the well-known cognitive science and neuroscience phenomena of the primacy and recency effects. These effects demonstrate that items presented at the beginning (primacy) and end (recency)
7

---

of trial are better retained than those presented in the middle. This pattern holds true across different values, indicating that the primacy and recency effects are an emergent phenomenon in our WM model. Thirdly, we observed an asymmetry in the strength of the primacy and recency effects, with the primacy effect being more prominent than the recency effect. This is evident when comparing accuracy at positions I and 9 for I. = 9. Furthermore, the primacy and recency effects become more pronounced with longer list lengths L. suggesting that these effects are more prominent when the memory load is high. For instance, the effects are minimal for L = 3 but become significant for L 7. Fifthly, we compared the behaviors of WM models with humans and found that, though the models never trained on human data, they approximate human-like qualitative trends in the primacy and recency effects across different serial positions and L values.
Furthermore, we conduct three ablations on VSR task to validate if these findings generalize acToss modalities, longer list lengths, and training paradigms. First, we report results for L.TTM2256 in the Language Serial Recall (LSR) task, which is language-hased variant of the VSR task (Appendix A3.1 Fig. A2))) Secondly, we experiment with the LSR task containing trials with even longer list lengths (Appendix A33.2 Fig. A2C). Thirdly, instead of joint training regime on all the tasks, we conduct individual training of a model only on the VSR task (Appendix AA333 Fig. A22 In all ablations, we observe similar primacy and recency effects suggesting that these findings are general across different modalities, longer list lengths, and various training paradigms. To further explore the underpinning mechanism attributing to intriguing and emergent primacy and recency behaviors in computational models, we expanded our discussions in Appendix A4.22
B. Short retention intervals have minimal effects on working memory. We report the accuracy as a function of serial positions under the conditions of different retention intervals in the VIR task (Sec. Fig. 4B). Aligning with human results, we observe that short retention intervals have minimal effects on WM performances, as indicated by the small performance gap between Tretent of 0 and (blue versus orange lines). It is possible that the effect might be more dramatic with longer retention intervals. Moreover, same as Fig. 4 here, we noted that there also exists a small recency effect for both models and humans. In particular, the effect is slightly stronger in Tretens 5 than Tretent 0. However, this effect is less prominent than the ones observed in Fig. 4A probably due to the short Tretent = 5 or the small list length.
Moreover, we experimented on retention intervals of 2, 4, and 6 (shown in Appendix A3.5 We
observed similar results as Fig. 4B with no significant performance differences across various short retention intervals (Fig. [3)). addition to the duration of retention intervals, we also studied the effect of the stimulus complexity during P Pretent on the working memory performances. We introduced variant of the Change Detection (CD) task where we replaced gray images with random noise images (see Appendix A3.4 for more details).
C. Domain conflicts and increased cognitive loads impair working memory performances. To investigate whether domain conflicts would impair WM performance, we report accuracy in the CS task as a function of cognitive loads in different conditions involving combinations of visual and spatial domains (Sec. 3 Fig. 4C). In humans, regardless of the domain of the distractor task, we observe a monotonic decrease in accuracy with increasing cognitive load. This suggests that WM performance is impaired due to the cognitive load imposed by the distractor tasks, rather than the specific domain of the distractor task itself. However, in the WM model, we observe a decrease in accuracy only when the distractor task is of the spatial domain, implying that our spatial distractor task introduces a higher cognitive load than the visual distractor task, thereby hurting WM performance.
D. Monotonic decrease in accuracy with increasing set size. We report the agent's 's accuracy as a function of set sizes S in SMU task (Sec. 3 in Fig. 4D. For recurrent models, the recall accuracy monotonically decreases with increasing set sizes. For different recurrent backbones with the same memory capacities, GRU outperforms LSTMs and RNNs (compare GRU-96 vs RNN-96 vs LSTM-96). Moreover, for the same recurrent backbone, larger memory capacity enhances overall recall accuracy across all set sizes (RNN-96 vs RNN-256), Humans show a similar qualitative set size effect as recurrent models. In contrast, TRF-96 shows an opposite trend to humans and recurrent models, with increased accuracy as a function of increasing set size initially. Also, note that TRF-96 outperforms other RNN models in recall accuracy at set size 8 even with smaller memory capacity (e.g. RNN-256).
E. Working memory stores fine-grained details of visual patterns. We report the accuracy as function of serial positions over various pattern- distractor differences n in VSRec (Sec. 3 Fig. 4E).
8

---

The Lef Rile Tok
SC
SFR
SI
SMU
STS
VIR
VSR
VSRN
CD
Rigan Button
XBS
ass
IS
Propotion of Neurons
DAS
UNNE
SUS
sc-
MIA
SPR
2SA
SMU
Tbsk
STSC
Normalaed Task
VSRoc
FTV
Clusters
Figure 5: Visualization of neural correlates and task-specialized neural clusters (A) For LSTM-1024 in STS task, the visualization results of neural clusters based on response behaviors (left) and task identities (right) are presented. See Sec.5G for details. In (B), we present the t-SNE visualization of neural clusters based on Task Variance (TV) of all hidden units in LSTM-256 over all 10 tasks. In (C), we present the neural selectivity of all hidden units for all the tasks. The neural selectivity is defined as TV values. All hidden units from left to right are sorted based on the clusters in (B). See the colorbar for TV values. In (D), we present the matrix of histogram plots for any pairs of tasks, where each histogram indicates the number of hidden units more selective to one task over the other. The selectivity for each hidden unit over a pair of tasks is defined as the fractional task variance (FTV). The x-axis and the y-axis of the histogram denote the FTV and the frequency of hidden units respectively. See Sec. for details.
Intuitively, as it becomes easier to discern the differences between the correct pattern and the distractor, the more accurate it is to recall the correct patterns. Indeed, we see an increase in overall accuracy for bigger n. This trend is similar to human results. Moreover, as also observed in Fig. we see primacy effect on both WM models and humans, where the accuracy is highest at position I and slowly decreases over subsequent positions. Interestingly, despite the difficulty of VSRec, for both humans and the models, the accuracy of recalling 36 cells correctly from the distractors with only 2 cell differences is still way above chance. This suggests that both humans and WM models are very good at memorizing fine-grained details of complex patterns.
F. Memory capacity is independent of the features or conjunctions of features. We study the capacity for storing different features and conjunctions in the CD task based on the agent's ss accuracy over set sizes under different feature conditions (Sec. 3 Fig. 4 Humans display minimal accuracy differences across feature conditions, suggesting that memory capacity is independent of specific features or feature combinations. Similarly, WM models exhibit comparable accuracy regardless of feature conditions. However, caution is advised in interpreting model results, accuracy saturates at 100% regardless of set size, likely due to overfitting. To draw stronger conclusions, further investigation in low-data regime using our trials is needed.
G. Neural population carries active task representation in task-switching experiment. In STS task (). we present the visualization results of clusters based on neural activation of all hidden units across all trials and all conditions for L.STM-1024 in Fig. 5A. Specifically, we first take the hidden representation during P Pprobe for all the trials within the task and then perform t-SNE (88) to project the hidden representation to 2D. On the left panel in Fig. 5A, we color-code the points based on the ground truth at that particular lprobe. In this case, we did not observe distinct colored clusters whereas when we color-coded the points based on the task that was supposed to be performed at lprobe. we observe two distinct colored clusters. In other words, the neural representations in the recurrent model encode task identities within a mask-switching paradigm, thus playing a supervisory role. See Appendix A4.3 for more details.
H. Responses from neural populations reveal task selectivity during joint training. To study the task-specialized neural clusters in recurrent models, we present the visualization results in Fig. 5B-D. See Appendix A4.4 for the steps to obtain and interpret the visualization results. From Fig. 5B, C. observed 8 distinct neural clusters and each neural cluster is selective to a different set of tasks.

---

For example, cluster 4 specializes in the SFR task, while cluster is active in temporal domain tasks like VSR and VSRec. We also note that clusters of neurons mostly emerge in networks of lower capacity with memory constraints. From Fig. 5D, we show diverse neural populations where some have preferences for one task over the other, while some are equally active in both tasks. For instance, the entry in row 1, and column 5 indicates that the majority of hidden units are more selective to SC task compared with STS, as the distribution of neural population shifts to the right with the majority of the neurons having FTV 1. Additionally, we conduct experiments where we lesion specific neural clusters to investigate causal evidence. As expected, we observe the corresponding performance drops in the accuracy of specific tasks depending on the neural cluster that has been lesioned (shown in Fig. A6). See Appendix A4.5 for details.
Discussion
Despite significant research progress in studying individual aspects of working memory (WM), there remains a huge gap in the broader study of WM. We take initial steps in this direction by establishing a systematic and quantitative methodology to study the multi-facets of WM. However, is important to recognize that we are still at the preliminary stage in a broader exploration of WM. The complexity, flexibility, and variability of WM present numerous avenues for future research. These avenues may involve refining the temporal alignment between human performance and model predictions, fine-tuning task difficulty across experiments by introducing variations in experimental parameters, experimenting with different training and testing methods, investigating the models' generalization abilities across a broader spectrum of tasks and modalities, and extending benchmarking experiments to include bio-inspired working memory models.
In our work, we introduce a comprehensive benchmark dataset consisting of 10 tasks, and million trials covering 4 functionalities and 3 domains of WM. Moreover, we benchmark WM models and humans on the 10 tasks and compare their memory characteristics at the behavioral, performance, and neural levels with set of newly introduced evaluation metrics. We obtained insights about the underlying mechanism of WM from these tasks and identified the limitations in existing WM models.
It is worth noting that all the models were never trained with any human data or behavioral biases. Yet, these models still remarkably exhibit interesting alignments with human behaviors, such as in dependency on set sizes, pattern similarities, and primacy and recency effects. We report the top- 1 accuracy for each condition in all the experiments; however, caution is needed when comparing the absolute performances between WM models and humans for multiple reasons: (1) differences in the concept of time, with humans perceiving time continuously seconds while Al operates in discrete time steps; (2) varied learning mechanisms, including Hebbian learning in biological systems and back-propagee propagation in AI: (3) distinct neuronal mechanisms between artificial and biological neurons, such synaptic plasticity and structural plasticity; (4) disparities in the visual diets fed to humans and AI models, with humans exposed to video streams and AI models working with random static images; and (5) constraints memory capacity varying between humans and Al models.
Our work contributes a suite of tools with detailed documentation for stimulus synthesis and parameter configurations in psychophysics experiments, training and testing state-of-the-ar working memory models, evaluating model behaviors and neural responses, and comparing behavioral consistency between humans and models. This methodology serves as valuable resource for comprehensive investigations into working memory, fostering advancements both biological and artificial intelligence systems.
Acknowledgments and Disclosure of Funding
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-023. its NRFF award WTW-NNFF1S-2223-000 Mengmi Zhang's 's Startup Grant from Agency for Science, Technology, and Research (A*STAR), and Early Career Investigatorship from Center for Frontier AI Research (CFAR), * The authors declare that they have no competing interests.
10

---

References
Anne-Marie Adams and Susan E Gathercole. Limitations in working memory: Implications for language
development. International Journal of Language Communication Disorders. 35(1):95-116 2000.
R Matt Alderson, Lisa J Kasper, Kristen L Hudec, and Connor HG Patros. Attention-deffiiidyyyeeattttiiii disorder (adhd) and working memory in adults: a meta-analytic review. Newropsychology. 27(3):287, 2013.
[3] George A Alvarez and Patrick Cavanagh. The capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological science, 15(2):06-111, 2004.
[4] SE Avons. Serial report and item recognition novel visual patterns. British Journal of Psychology. 89(2)2255-308 1998.
[5] Alan Baddeley, Working memory and language: An overview. Journal of communication disorders, 36(3):189-200 2003.
[6] Alan Baddeley, Christopher Jarrold, and Faraneh Vargha- Khadem. Working memory and the hippocampus. Journal of cognitive neuroscience, (1112)3855-38611 2011.
[7] Alan David Baddeley and Sergio Della Sala. Working memory and executive control. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 35 (11 1996.
[8] Paul M Bays and Masud Husain. Dynamic shifts of limited working memory resources in human vision. Science, 321(5890):851-33 2008.
[9] Patricia A Carpenter and Marcel A Just. The role of working memory in language comprehension. In Complex information processing, pages 51-88. Psychology Press, 2013.
[10] Kyunghyun Cho, Bart Van Merriënboer. Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-dceoder for statistical machine translation. arXiv preprint arXiv: 14001110000000 1078, 2014.
[11] Alex Clark. Pillow (pil fork) documentation, 2015.
[12] Andrew RA Conway, Michael Kane, and Randall W Engle. Working memory capacity and its relation general intelligence. Trends in cognitive sciences, 7(12):447555 552, 2003.
[13] Cathleen Cortis, Kevin Dent, Steffan Kennett, and Geoff Ward. First things first: similar length and output order effects for verbal and nonverbal stimuli. Journal of Experimental Psychology: Learning, Memory, and Cognition, 1174) 2015.
[14] Nelson Cowan Attention and memory: An
Oxford University Press, 1998.
[15] Clayton Curtis and Mark D' E'pposioo. Persistent activity in the prefrontal cortex during working memory. Trends in cognitive sciences, 7(9):15-442 2003.
[16] Meredyth Daneman and Patricia A Carpenter. Individual differences in working memory and reading, Journal of verbul learning and verbul behavior, 19(4):450-466 1980.
[17] Jan W de Fockert, Geraint Rees, Christopher D Frith, and Nilli Lavie. The role of working memory in visual selective attention. Science, 291(5509):1803- 1801-1806.00000.. 2001.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[19] Mark D' essossio, John A Detre, David Alsop, Robert K Shin, Scott Atlas, and Murray Grossman. The neural basis the central exccutive system of working memory, Natur, 378(6554):79--88 1995.
[20] Allen R Dobbs and Brendan Gail Rule. Adult age differences working memory. Psychology and aging. 4(4):500, 1989.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint urXiv:2010. 11929, 2020.
[22] Yang Feng. Shiyue Zhang, Andi Zhang, Dong Wang, and Andrew Abel. Memory-augmented neural machine translation. arXiv preprint arXiv: 02005. 2017.

---

[23] NF Forbes, LA Carrick, AM McIntosh, and SM Lawrie. Working memory in schizophrenia: meta-analysis. Psychological medicine, 3966):88-9955, 2009.
[24] Joaquin M Fuster and Garrett E Alexander. Neuron activity related to short-term memory. Science, 173(3997):652 654, 1971.
[25] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 14055:1410.540, 2014.
[26] John M Hinson, Tina L Jameson, and Paul Whitney. Impulsive decision making and working memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(2):298, 2003.
[27] Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural computation, 98):1735-1778,, 1997.
[28] John Jonides, Eric H Schumacher, Edward E Smith, Robert A Koeppe, Edward Awh, Patricia A Reuter-Lorenz. Christy Marshuetz, and Christopher R Willis. The role of parietal cortex in verbal working memory. Journal of neuroscience, 18(13)50026 5034, 1998.
[29] June R Ker.. SE Avons, and Geoff Ward. The effect retention interval on serial position curves for item recognition visual patterns and faces. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25(6):1475, 1999.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
[31] Wayne K Kirchner. Age differences in short-term retention of rapidly changing information. Journal of experimental psychology, 55(4):352, 1958.
[32] Michael Koenigs, Aron K Barbey, Bradley Postle, and Jordan Grafman. Superior parietal cortex is critical the manipulation information in working memory. Journal of Neuroscience, 29(47); 14980- 14986. 2009.
[33] Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit lyyer, James Bradbury, Ishaan Guirajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning, pages 1378 1387, PMLR, 2016.
[34] Patrick C Kyllonen and Raymond E. Christal. Reasoning ability is (little more than) working-memory capacity?! Intelligence, 14(4):399-433, 1990.
[35] Steven Luck and Edward K Vogel. The capacity of visual working memory for features and conjunctions. Nature, 39066557) 1997.
[36] Brian McElree. Accessing recent events. Psychology of learning and motivation, 46:155-200 2006.
[37] Fiona McNab and Torkel Klingberg. Prefrontal cortex and basal ganglia control access to working memory. Nature neuroscience, 11(1):103-107, 2008.
[38] Nachshon Meiran. Reconfiguration of processing mode prior to task performance. Journal of Experimental Psychology: Learning, memory, and cognition, 22(6):1423, 1996.
[39] Earl K Miller, Cynthia A Erickson, and Robert Desimone. Neural mechanisms of visual working memory
in prefrontal cortex of the macaque. Journal of neuroscience, 16(16):5154 1550:5555111677 1996.
[40] George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63())81, 1956.
[41] Klaus Oberauer. Die koordination kognitiver operationen. eine studie zum zusammenhang von "workingmemorry" und intelligenz (the coordination cognitive operations: a study on the relationship between working-memory and intelligence). Zeitschrifi für Psychologie, 201:57-84, 1993.
[42] Klaus Oberauer. Access to information in working memory: exploring the focus of attention, Journal of Experimental Psychology: Learning, Memory, and Cognition, 28(3):411, 2002.
[43] Klaus Oberauer. Design for a working memory. Psychology of learning and motivation, 51:45-1066 100, 2009.
[44] Klaus Oberauer and Reinhold Kliegl. A formal model of capacity limits in working memory. Journal of memory and language, 55(4):601-62) 2006.
12

---

[45] Klaus Oberauer, Stephan Lewandowsky. Edward Awh, Gordon DA Brown, Andrew Conway, Nelson Cowan, Christopher Donkin, Simon Farrell, Graham Hitch, Mark J Hurlstone, al. Benchmarks for models of short-term and working memory. Psychological bulletin, 144(9):885, 2018.
[46] Klaus Oberauer. Heinz-Martin Süβ, Oliver Wilhelm, and Nicolas Sander. Individual differences in working memory capacity and reasoning ability, 2007.
[47] Pernille J Olesen, Helena Westerberg, and Torkel Klingherg. Increased prefrontal and parietal activity after training of working memory. Nature neuroscience, 7(1):75-79 79, 2004.
[48] Matthew F Panichello and Timothy Buschman. Shared mechanisms underlie the control of working memory and attention. Nature. 592(7855)6001665,, 2021.
[49] David H Raab. Backward masking. Psychological Bulletin, 60(2):118, 1963.
[50] Tim Rakow, Ben R Newell, and Konstantina Zougkou. The role of working memory in information acquisition and decision making: Lessons from the binary prediction task. Quarterly Journal of Experimental Psychology, 63(7))1335-1360, 2010.
[51] Ranulfo Romo, Carlos D Brody, Adrián Hemández, and Luis Lemus. Neuronal correlates of parametric working memory the prefrontal cortex. Nature, 399(6735):47044777 1999.
[52] Timothy A Salthouse. Adult age differences in integrative spatial ability. Psychology and Aging, 2(3):254, 1987.
[53] Timothy A Salthouse and Renee Bahcock. Decomposing adult age differences in working memory. Developmental psychology. 27(5):763, 1991.
[54] Priti Shah and Akira Miyake. The separability of working memory resources for spatial thinking and language processing: an individual differences approach. Journal of experimental psychology: General, 125(1):4, 1996.
[55] Timothy Spellman, Mattia Rigotti, Susanne Ahmari, Stefano Fusi, Joseph Gogos, and Joshua A Gordon. Hippocampal-prefronli input supports spatial encoding in working memory. Nature, 522(7556))309-314 2015.
[56] Sainbayar Sukhhaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
[57] Hertz-Martin Süß, Klaus Oberauer, Werner W Wittmann, Oliver Wilhelm, and Ralf Schulze.
Working-memory capacity explains reasoning ability-and a little bit more. Intelligence, 30(3):611228, 2002.
[58] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[60] Evie Vergauwe, Pierre Barrouillet, and Valérie Camos. Visual and spatial working memory that dissociated after all: a time-based resource-sharing account. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(4):0012, 2009.
[61] Edward Vogel, Geoffrey F Woodman, and Steven Luck. Storage of features, conjunctions, and objects in visual working memory. Journal of experimental psychology: human perception and performance, 27(1):92, 2001.
[62] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916. 2014.
[63] Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks, Nature neuroscience, 22(2))297-306, 2019.
13

---

A1 Detailed Descriptions about Psychophysics Experiments in WorM
Experiment A1 -Spatial Free Recall (SFR) 13 This experiment focuses on the spatial domain of WM and aims to assess the ability to remember spatial locations and engage in immediate free recall of that information. Experiment schematic: The experiment begins with P Present followed by Prrobe- In each trial, the stimulus image in P present contains 30 green squares randomly chosen from a 10 X 10 grid. At every (presents one randomly selected square of the 30 green ones turns red. Finally, in Pprone. grey blank image is presented signaling the WM model recall all the past spatial locations where the squares turned red. On the other hand, humans received an auditory cue to begin the recall. Humans indicate the highlighted squares by clicking on them in any order using a mouse cursor. The WM models, however, engage in a 100-way multi-label classification task with multi-hot encoded targets corresponding to the highlighted square locations. Afterward, we select the top-L L squares from the WM model's prediction based on the output probabilities, sort them in descending order, and generate the model's s recall response, where I. T present refers to the list length in the trial. Experiment conditions: We vary list length I. = present = 1 - 8, 10, 12, 15, 18. Refer Fig. ATA for example trial.
Experiment A2 Change Detection (CD) (35 This experiment involves the visual domain of WM and aims to evaluate the agent's 's memory capacity for simple features and conjunctions. Experiment schematic: The experiment consists of P Ppresent. Pretent, and P proobe in the respective order. In Ppresent where T prreeent = 1, a memory array, consisting of S bars with varying attributes in color (red or green), orientation (horizontal or vertical), size (small or big), and presencc/absence of a gap in the middle of each bar, is presented. Here, S refers to the set size of the trial. During Pretent. a series of blank grey images are presented. In P Pprobe, where T probe = 1, agents are prohed with test array that can be either identical to the memory array or has one bar that differs along a single feature dimension. The agents have to make binary prediction about whether or not there is a change given the test array. Experiment conditions: We formulate main experiment conditions by varying the change of the test array in five aspects: color, orientation, bar size, presence or absence of the gap, and conjunctions of the four features where either of the four features could change, Additionally, we vary S from 2, 4, 6, 8, 10, 12 and also vary T Tretent from O, 6, 12, and 18. See Fig. ATB for an example trial from this experiment.
Experiment A3 Visual Item Recognition (VIR) 229 Here, we examine visual WM with different length lists and investigate the effect of retention interval on recognition performance. Experiment schematic: The experiment is structured with three phases presented in the following order: P Ppresent- Pretents and Prrobe. In P presents a series of distinct matrix patterns are presented in sequence. These patterns are composed of 6 x 6 square cells, with half of the cells being filled with red. In P Pretents a series of grey blank images are presented. In P Prrabe, where Trrabe = 1, a probe image is presented, consisting of two patterns presented side-by-side. One of these patterns is an exact match to one of the previously shown patterns, while the other pattern serves as a distractor. The distractor pattern generated from the target pattern by changing 2 unfilled cells to filled ones and vice-versa. The agents have to perform a binary classification task, wherein represents the left pattern and represents the right pattern. Experiment conditions: We vary list length L T Tpresent from 4, 6, 8, and 10, and also vary T Tretent from 0, 2, 4, 5, and 6. See Fig, A1 C for an example trial.
Experiment A4- Visual Serial Recall (VSR) 4 This experiment evaluates the ability to accurately recall matrix patterns as well as the order which they were presented. Therefore, this experiment encompasses both the visual and temporal domains of WM. Experiment schematic: The experiment consists of Prresent followed by Prrobe- Ppresent involves the presentation of L matrix patterns in a sequential manner. These matrix patterns consist of square cells arranged in 6 x 6 grid, with half of the cells filled in green. In Pprobe. a probe image is displayed containing all the L patterns presented during P present. These L patterns are placed simultaneously at randomly chosen locations from set of 9 evenly spaced locations in a 3 x 3 grid. Locations that were not used remained blank. At each lprobe: the agents perform a 9-way classification task to indicate the first pattern presented during Prresent, followed by the second pattern, and so on. Experiment conditions: We vary list length L = Tpresent from 2 9. An example trial is shown in Fig, AID.
Experiment A5 Visual Serial Recognition (VSRec) 4.) In this experiment, agents are required to recall sequentially presented matrix patterns in a forward serial order within a recognition paradigm. Here, we also investigate the resolution of the recalled patterns by varying the difference between target patterns and distractor patterns. This experiment again involves both the visual and temporal
14

---

domains of WM. Experiment schematic: The experiment is similar to the VSR experiment above in terms of the setup. The experiment consists of P present and P prooee: P Prresent is exactly the same as P. present in the VSR experiment. The only differences happen at pprabe- During P prooee series of two-alternative recognition tests are presented in consecutive time steps to assess memory for the presented patterns in forward serial order. Essentially, at every (probe, the target pattern from corresponding lyresent and a distractor pattern are presented side by side. The agents perform a binary classification task to indicate the target pattern, where 0 corresponded to the left pattern and 1 corresponded to the right pattern. The distractor patterns were generated from the target pattern by changing the values of cells such that the total number of filled cells remained constant. We denote n as the pattern-diss distractor difference. We made sure that the distractor patterns were not among the initially presented matrix patterns. Experiment conditions: We vary list length L = Tpresent from 2 to 9 and the pattern-distractor difference n from the range of 2, 4, 6, 8, and 10. See A for an example trial.
Experiment A6 Complex Span (CS) 60). This experiment aims to investigate the interference between the visual and spatial domains of WM, as well as the effect of cognitive load recall performance. Moreover, in this experiment, agents are required to perform a serial recall of information. Hence, this experiment encompasses all three domains of WM: visual, spatial, and temporal. Experiment schematics: The experiment consists of two types of P Prreeent and two corresponding types of probee and two types of P Pdistroctor- (I) The first type of P present and P Pprobe corresponds to visual storage and involves memorizing a sequence of visual patterns in Ppresent and recalling the exact visual patterns in a serial order in P Prrobe. The visual patterns are represented as 4 x 4 grid, where half of the cells are in red. (II) The second type of P present and P probe corresponds to spatial storage and involves memorizing sequence of spatial patterns involving ball movements in P Prresent and recalling the exact directions of ball movements in a serial order in P proobe- To interfere with the spatial/visual memories in P prrseent two types of P Pdistractor are introduced: (III) The first type of Pdistractor corresponds to the visual domain and involves color discrimination tasks where agents have to classify the color of the given panel as blue or red. (IV) The second type of P ddistractor corresponds to the spatial domain and involves symmetry discrimination tasks where agents have classify whether the given pattern symmetric about the vertical axis or not. Experiment conditions: We consider four different variations: I(P (resene))) + III distractor) + I(P ((Pprote): II(P present) + III distractor) II(P pr(Pprohe): I(P present) IV(P
I(P I(P probe); and 11(P ((Ppreseet)) + (Vstarstra)tor) + probe: namely, visual + visual, spatial+visual visual+spatian and spatial spatial respectively. Additionally, we change the cognitive load by varying Taistractor from 0, 1, 3, and 5. See Fig, A1F for an example trial.
Experiment B1 Spatial Coordination (SC) [1]. The objective of this experiment is to assess the agents' working memory (WM) capacity in the spatial domain, specifically focusing on their ability to coordinate and integrate spatial information across various time steps. Experiment schematics: The experiment consists of P present and P probe in the respective order. At every (present, one cell in 10 x 10 grid is highlighted in green. In probee. the agents are probed using blank grey image to perform a binary discrimination task to indicate whether the pattern integrated mentally over all T prresent is symmetric about the vertical axis or not. Experimental conditions: We vary the list lengths L = T present from 10, 12, 14, 16, and 18. Refer Fig. A1 for an example trial.
Experiment B2- Spatial Integration (SI) (52). This experiment focuses on the spatial domain and aims to investigate the agents' abilities to integrate spatial information. Experiment schematics: The experiment consists of Ppresent and probe: In P Ppresent, there is sequential presentation of partial line drawings at every Spresent- In P probe. where Turobe = 1, a complete line drawing containing 12 line segments is presented. The task is to mentally integrate the partial drawings shown during P Prresent and compare them to the complete line drawing shown during P Prrobe. Essentially, the agents perform a binary classification to indicate whether the figure shown in Prrote is identical to the mentally integrated figure from P Ppresent- All line drawings were generated by connecting neighboring points within an imaginary x 4 grid. Experiment conditions: We vary the number of line segments in the partial drawings from 12, 6, 4, 3, 2. and I, resulting in list lengths L = T present of 1, 2, 3, 4, 6, and 12 respectively. As the list length increases, the number of integrations needed to solve the trial also increases. See Fig. ATH for an example trial.
Experiment C1 Spatial Memory Updating (SMU) 44), This experiment assesses the capabilities of memory manipulation in the spatial domain. Experiment schematics: The experiment consists of Ppresent P Pupdate, and P Prrobe in the respective order. In P Ppresent where T preeeet 1, a stimulus is
15

---

presented, consisting of S green squares, arranged in an imaginary circular formation. Within each square, there is a red marker located in one the nine possible locations within an invisible x grid. Agents are required to memorize the spatial locations of the markers within each square. Next, at every tupdate Pupdate, an arrow presented at the center of one of the squares, indicating the direction for mentally updating the marker's new location. The arrows can be oriented vertically, horizontally, or diagonally, with the condition that the marker never exits the square. A sequence of total of 8 memory update operations is presented in clockwise sequence through the S squares. Hence, Tupdate = Finally, at every lprobe of P probes one out of the S squares highlighted in red. This red square acts particular square through a 9-way classification task. The red marker locations for all S squares are probed in random order until lprobe = S. Experiment conditions: We vary S from to 8. Refer Fig. A for an example trial.
for the agents to recall and indicate the marker's location within that
Experiment D1 Spatial Task Switching (STS) 38 This experiment involves the spatial domain of WM and investigates the agent's capacity flexibly transition between two spatial discrimination tasks based on supervision cues. Experiment schematies: The experiment consists of two types of phases Pawitch and P probe presented in random order. In Swwiteh, cue image is presented to the agents indicate the discrimination task to be performed for subsequent probe. There are two binary discrimination tasks: top-versus-bottom bottom and left-versus-righee right. For the teft-verrus-right discrimination task, the cue image contains dash markers placed for the versus-botton discrimination task, the dash markers are positioned on the top and bottom sides of the image. At every probe in P probe: a red marker is presented in any of the four locations within a x 2 grid. The agents are required to switch between the two discrimination tasks during every P switch and respond with 0 for top/left positions and for bottom/right positions. Experiment conditions: We randomly vary the number of task switches i.e. Pswitch in different trials as well as the time step at which task switches occur See Fig. AIJ for an example trial.
the left and right sides of the image. In contrast,
Experimental trial splits. In each WM psychophysies experiment, we generate total of 86,400 trials for training, 9,600 trials for validation, and 9,600 trials for testing. Importantly, note that for the CD experiment, there are five distinct conditions, each of which consists of its own set 86,400 trials for training, 9,600 trials for validation, and 9,600 trials for testing.
16

---

B
C
D
E
F
G
HH
CD
VIR
VSR
VSReo
CS
SC
SI
SMU
STS
phase
Probing phase
Retention
Distractor pluse
Memory updating phuse
Task sw switch phase
Zeno Padding
Time
Figure A1: Detailed schematic each experiment. We expand the overviews all ten experiments introduced in Fig. I and Sec. 3 with detailed experiment schematics. Each column represents the schematic of one experiment. The time goes from the top to the bottom. Each row presents the example stimulus at that time step t. The legend and interpretations of this figure follow Fig. 1
17

---

A2
Computational Models of Working Memory
A2.1 Additional Implementation and Training Details
Feature Extractor. In all our models, we employ a 4-layer 2DD-convolutional network with 64, 128, 256, and 512 channels in the successive layers. We also experimented with ViT 21 pre-trained on ImageNet, where we freeze the model and use it for feature extraction. However, features from natural images of ImageNet [88 do not generalize synthetic images of our dataset.
Task Embeddings. We learn task-specific embeddings M E R R14x14 which informs the model of which task needs to be performed. There total of 10 WM tasks. Although, the Change Detection (CD) task comprises five distinct task conditions, namely color, orientation, size, gap, and conjunction, each of which necessitates unique task identifier. Consequently, there are total of 14 (9+5) unique task embeddings.
Training Details. During training, we sample 10 trials from each task, obtain the model's responses for all the sampled trials, and compute task-specific losses for each respective task. These individual losses are then aggregated and the joint loss is utilized to perform back-propagation. In essence, the model parameters are optimized jointly over all tasks. Note that we formulate the SFR task as multi-label classification and the other tasks as multi-class binary classification. Consequently, the task-specific losses are computed using either cross-entrope or binary cross-entropy, depending on the nature of the tasks.
Implementation Details. All our models were trained from scratch. Hyperparamater tuning was done using grid search. We show the different hyperparameters for our models in Tab. AT We employed the Adam optimizer (30) with a starting learning rate of le-4 and the first moment of 0.9. The learning scheduler reduces the learning rate by a factor of 0,8 when the validation loss doesn't improve for 3 epochs. We report the average top-1 accuracy and standard error for various model architectures and capacities, computed across test trials from different conditions. For all our experiments, we used 16 NVIDIA RTX A5000 GPUs, each equipped with 24 GB of memory.
Data Generation Pipeline. Given any condition in a task, we follow the three steps below to synthesize stimulus for every time step. For an easy illustration of the three steps, we use the Visual Serial Recognition task (Appendix(A) [A] as example. First, depending on the list length of the trial, we generate multiple unique 6 matrix patterns by sampling grid cells to be filled. These matrix patterns constitute the memory and distractor items for the trial. Second, we generate the ground truth labels for P prooe based on the sampled matrix patterns. Third, given all the patterns sampled in Step 1, we synthesize the stimulus using the Pillow library [Τ]]
Our systematic process ensures the creation of task-specific trials with multiple controlled variations and difficulty. We provide detailed documentation and made our code easy to use and customize for future studies, such as generalization tests of working memory models with longer task sequences, and larger set sizes. Our code and data are publicly available: link.
A2.2 Time mapping between WM models and human psychophysics experiments
There could be multiple ways of establishing the time mapping between WM models and human psychophysics experiments (Fig. l]. Here, for the WM models, we stick to stimulus per time step rule. In Tab. A2 we report the mapping of single time step in WM models to the duration of the corresponding stimulus presentation in human experiments. For instance, in the SFR human
Hy Hyperparameters
RNN
GRU
LSTM
TRF
Input Size
512 [96, 256, 1024)
Hidden/ / Embedding Size
[96, 256, 1024]
[96, 256, 1024)
[96, 256, 1024)
Encoder Layers
TRF Attention Heads
Batch Size
[140, 70]
[140,
[140, 70]
[140,
Learning Rate
[0.00008, 0,0001, 0.0003]
[0.00008, 0.0001, 0.0003]
|0.00008, 0,0001, 0.0003]
|0.00008, 0,0001. 0.0003]
Epochs
200
200
200
200
Table AI: Different hyperparameters considered for models trained on the WorM dataset. The best hyperparameters for each model are bold. Here, batch size refers to the total number of trials from all tasks within one batch.
18

---

TASK
P pprnsent
Pretent
P probe
P distrratorr
Pwwisch
Pupdate
SFR
750 ms
CD
100 ms
150 ms
2000 ms
VIR
1000 ms
1000 ms
VSR
1550 ms
VSRec
1550 ms
CS
500 ms
1700
SC
1000 ms
SI
SMU
STS
1500 ms
203 ms
Table A2: Time mapping between WM models and Human psychophysics experiments. We show the mapping of single time step in WM models ms in different phases of the corresponding specific phase in the experiment. Dashes (-) indicate that either humans were provided with indefinite amount of time during that phase, or the duration of stimulus presentation was variable and not standardized.
human psychophysics experiments. Asterisks (*) indicate the absence
experiment, the duration each stimulus presentation during P Preeeent was 750 ms. whereas the WM models were exposed to each stimulus for only I time step. Therefore, within the P preeent phase of the SFR experiment, a single time step of the WM models corresponds a duration of 750 in the human experiments.
A2.3 Quantitative comparison with human behaviors
To quantitatively compare computational models and humans, we introduce types of scores: the slope difference score (A) and the average accuracy difference score (B). First, we define the slope difference score (A), as (MM HA) All where MA and HA are slopes for computational models and humans respectively. We use the slope difference score to capture similarity in the trend of accuracy against different conditions like serial position, set size, and cognitive load. Second, we define the average accuracy difference score (B), as (MB HB) H where MB and H are average accuracy for computational models and humans respectively across different conditions. For example, Apotternd:ff in VSRec task (Fig. 4E) for LSTM-256 model is calculated by first computing MA i.e. the slope the best-fit linear line for the model's top accuracy as a function of serial positions in the plot. Likewise, we compute HA and hence, Aputterndif Similarly, to calculate Mn. we first compute the average accuracy of the model over all the serial positions for each condition of pattern difference and then compute the slope of the best-fit linear line for the model's s top 1 accuracy as function pattern differences in the plot. Similar calculations follow afterward to obtain H p and subsequently Bpatterndff-- Note that in the VSR task (Fig, 4A), we calculate two different scores for the primacy (Aprimacy) and recency (Areceney) effect. For this, we divide the list at the point of minimum top-1 accuracy, into two lists, i.e. primacy and recency lists.
With these evaluation metrics, we report the quantitative measure of behavioral consistency with humans in Tab. A3 Ideally, if the model performs the same as humans, the slope difference would be zero. test whether the results are significantly different from the chance model, we computed the p-value using two-tailed t-tests. From the results, we found that the majority of the models are more consistent with humans than the chance model; however, there still exists performance gaps between these models and humans.
19

---

T-1024
3.480
1.102*
0.693
1.438
0.262
0.773*
1.255
0.133*
1.196'
0.323
1.003
0.110
T-256
3.287
1.124'
0.693'
0.968
0.441
0.900
0.511
1.000'
0.161
0.923
0.358'
0.999
0.110'
T-96
3.307
1.170'
0.693
0.995
0.446
0.998
0.218
1.099
0.155*
0.977
0.356
0.999
0.110'
L- 1024
6.538
1.328
0.364
0.554
0.140
0.091
0. 0.340
0. 0.909
0. 161
0.625
0.287
0.909
0.110'
L-256
6.458
1.384
0.348
0.951
0.057
0. 0.998
0.594
0. 0.993
0. 0.160
0.097
0.220
0.999
0.1
I-96
3.303
1.134
0.696
3.021
0.265
9.999
0.708
0.985
0.159
0.531
0.080
0.999
0.110
Task
VTR Task
CS Task
Task
VSRee Task
CD Task
G-1024
VSR
6.859
1.276'
0.400
0.729
0.068
0.989
0.350'
SMU
1.000
0.161
0.910
0.311
0.999
0.110
G-256
6.748
1.327
0.387
2.244
0.126
0.996
0.741
1.001
0. 0.160
0.569
0.271
0.999
0.110
G-96
8. 108
1.449
0.467
0.801
0.164
0.995
0.750
1.001
0. 1..60
0.150
0.030
1.003
0.110
R- 1024
3.112
1.138
0.705
0.958
0.261
0.996
0.718
0.006
0.310
0.979'
0.347
0.799
0.154"
R-256
3.264
1.183"
0.697
0.729
0.272
0.997
0.763
1.059'
0.396
1.382
0.318
0.999
0.1
R-96
3.300
1.164
0.692
0.324
0.245
0. 1.995
0.754
0.347
0. 0.680
1.103"
0.302
0.999
0.110"
Bratentiooittreall
Aprimacy
Arecency
Bristienath
Acagnittcetood
Beagni
Autuise
Apatterndiff
Bpatterndis/
Asctaise
Bettsize
Table A3: Quantitative scores for comparing humans and computational models on different behavioral effects. We conduct two-tailed t-tests against chance and compute corresponding p-values. Asterisk (*) indicates p-value < 0.01. See Sec. A2.3 for introduction to slope difference score A and average accuracy difference score B.
20

---

A3 Ablation Experiments
A3.1 Experiments for language modality
We introduce new experiment in language modality, with paradigm similar to that of Visual Serial Recall (VSR) Sec. 3 Here, we investigate the behavioral characteristics of the computational models in language modality instead of the visual modality.
Language Serial Recall (LSR). This experiment assesses the ability to recall words accurately as well as the order in which they were presented. Experiment schematic: The experiment consists of Prresent followed by P prrabe. P Pprexent involves the presentation of L words in a sequential manner. We randomly sample these words from the Google News dataset. This dataset contains pre-trained vectors for around million words and phrases. each time step during P Ppresent. we first map the word to the corresponding 300-dimensional word embedding followed by zero padding this vector to produce L x 300 dimensional representation which then fed to the WM model. In Probes at each time step, all the L words are presented simultaneously. We collect the embeddings for all L words and append them in a random order to generate L x 300 dimensional representation. At each lprobes based on this representation, the agents perform 9-way classification task to indicate the first word presented during P present, followed by the second word, and so on. Experiment conditions: We vary list length L = pressnt from 2 9.
Result analysis: From the experimental results shown in Fig. A2B, we observed consistent performances as the ones from the VSR experiment. First, as list length increases, there is corresponding increase in the memory load. Consequently, we observe decrease in overall accuracy across various list lengths. We also observe pronounced primacy effects across varying list lengths, alongside a recency effect that becomes apparent in longer list lengths. These results suggest that the behavioral findings generalize across different modalities.
A3.22 Training with Longer List Lengths
To verify if our behavioral findings are valid for longer list lengths, we train our model on the LSR Task (Appendix A3.1) with trials of list lengths up 16 instead of 9. Therefore, in this case, trials consist of 32 time steps in total instead of 20 time steps. We report our results in Fig. A2|c. Here, we observe similar primacy and recency effects as we observed in the LSR task with shorter list lengths (compare with Fig, A2A). J. We notice similar trends where primacy effects are prominent in shorter list lengths and the recency effects start emerging in longer list lengths with higher memory load. This result suggests that the primacy-receme effect is a general phenomenon even for longer list lengths.
A3.3 Individual task training
To isolate the effect of the joint training paradigm, we also conduct individual training on the VSR task. Specifically, we train the LSTM-256 model only on the VSR task. We report the results in Fig.A24 We make similar observations as the joint training paradigm. First, with increasing list length, the overall accuracy of the model decreases. Next, the primacy-recency effect is more prominent in cases of longer list lengths with higher memory load. Finally, the primacy effect is more prominent in smaller list lengths while the recency effect is observed for longer list lengths. This indicates that the primacy-recency effect is more general phenomenon agnostic of the training paradigm.
A3.4 Increased difficulty in CD task
In order to introduce additional difficulty in the CD task, we also experimented with more complex variants where we presented random noise images during the retention phase, instead of simple gray images. These random noise images are generated by randomly sampling a pixel value from 0-255 for every pixel of RGB channels of the image. This also aligns with the backward masking technique in neuroscience, which is often used for disrupting recurrent connections in the biological brains We train and test the LSTM-96 model on this new variant of the CD task. We observe that while L.SMM-96 was successful completely solving the initial version of the CD task with 100% accuracy, it completely failed to leam this modified variant.
49
21

---

B
1se Sength3
Lerght
Top-1 Accuracy
0.9
1se Eegth
1stt Length
Top-1 Accuracy
Last Leteth 5
Last Leugth
1ist Length
0.8
Leet Length o
0.8
Lsst Zength
List Longth 7
Lis Zength
Last Longth
0.6
0.3
0.6
0.4
0.5
3
2
4
0.3
Serial Position
Serial Position
C
+
List Length 3
Leugth
Top-1 Accuracy
0.8
Length
List Leugh
0.6
Length 7
Length
List Length
Length
0.2
Length
0.0
8 9 10 Serial Position
6
12 14 15
Figure A2: Ablation experiments. (A) We show the top-1 accuracy of the LSTM-256 model as function of the serial position of the memory items in the VSR task. Here, the model was trained only on the VSR task. In (B), we show the behavioral accuracy as function of list lengths in LSR task for LSTM-256 model. (C), we show the plot for an L56M-256 model trained only on the LSR task with longer sequence lengths (32 time steps).
A3.5 Fine-grained retention interval testing VIR task
We evaluate the LSSTM-1024 model more retention intervals in the VIR task. Specifically, test our model on 0, 2, 4, 5, and 6 retention intervals. We show these results in Fig.[A3] We observe similar results as Fig. 4B, and do not find any statistical significance in working memory performance differences across various short retention intervals.
A4
Additional Results and Analysis
A4.1
Behavioral Performance Comparison across Architectures and Memory Capacities
We analyzed various memory characteristics in Fig.
4
Here, expand different model A10
architectures and model capacities in Fig A7 performances in top-1 accuracy vary the memory capacity, the better the behavioral performance. However, there are a few exceptions where the models with smaller capacities yield higher accuracy (e.g. Fig. A1l J versus K). We also notice that some tasks are extremely difficult for all WM models to learn regardless of the
A12 We found that the architectures and memory capacities. In general, the larger
A8
AIl
1.0
Human
Top-1 Accuracy
0.9
LSTM-1024 RI
Human
LSTM-1024 RI
0.8
LSTM 1024 RI
LSTM-1024 Ri
0.7
LSTM-1024 RI
0.6
0.5
2
Serial Position
Figure A3: Short retention interval testing in VIR task. We show the results for LSTMM11024 on
the VIR task with multiple retention intervals including 0. 2, 4, 5, and 6. The chance level is 0.5 for all the above plots.
22

---

Models
Hidden size
Number parameters
RNN-96
96
1.8M
RNN-256
256
2M
RNN-1024
1024
3.5M
GRU-96
96
2M
GRU-256
256
2.5M
GRU-1024
1024
6,5M
LSTM-96
96
LSTM-256
256
2.5M
ISTM-1024
1024
8M
TRF-96
96
2.5M
TRF-256
256
4.5M
TRF-1024
1024
20M
Table A4: Different computational models of varying memory capacity with their number of parameters.
architectures. For example, in Fig. A9g almost all the WM models fail. Even with large memory capacity of 1024, which proves to be sufficient for performing other WM tasks, it still performs badly in the CS task.
For fair comparison, sizes or with an equal number of network parameters. To compare different architectures with the same hidden size, we compare RNN-1024, GRU- LSTM-1024, and T0241024. These models are jointly trained on all the working memory tasks. We show the joint average accuracy, i.c., the average accuracy of these models over all tasks in Fig. A4 We observe that LSTMM achieves the best joint average accuracy, followed by GRU-1024, TRF-1024, and RNN-1022 respectively.
either compare different architectures with equal number of hidden
Similarly, to compare architectures with the GRU-256, LSTM-256, and TRF-96 (see Tab. A4 for the number of parameters in different models). In Fig. A4 we again observe LSTM-256 is the best performing model, followed by GRU-256. TRF-96, and RNN-256 respectively. In either comparison, we can note that recurrent networks like LSTMs and GRUs perform better than transformers. While recurrent architectures (GRUs, LSTMs) generally perform better in top-1 accuracy and exhibit the primacy and recency effect (Sec. (A)..5A). the transformer architectures fail replicate such phenomenon; and their top-1 accuracy is also lower than the recurrent architectures.
number of parameters,
compare RNN-256,
Joint. Validation Accuracy
0.9
1.SM9996
LSTM-236
LSTM-1024
CRU-96
0.8
G5U
GRU-1024
RNN-96
RNN 256
RNN-1024
TRF-96
0.6
TRF-256
0.5
50
100
150
200
Epoch
Figure A4: Training curves for different model architectures and capacities. The y-axis shows the averaged validation accuracy across all tasks and the x-axis shows epochs.
23

---

A4.2 Further probing into the mechanism attributing to primacy effects
In Fig. 4A and Fig. A2 we observe consistent primacy effects. Our hypothesis for the primacy effect was that the first stimulus stored in the working memory acts as the first principal component, steering the memory representations of the subsequent stimulus align with the representation of the first stimulus. Hence, the first item was forgotten less; thus, the network exhibits the primacy effect. To validate this hypothesis, in every trial of the Visual Serial Recall (VSR) task, we first compute the hidden states at time step 1, when the first item is presented, and then compute its cosine similarity with the hidden state representations over subsequent time steps i when subsequent memory items are hypothesis, the cosine similarity between
presented. We report the results in Fig. A5 Based the 1st time step with thhh time step should be higher than the cosine similarity between 2nd time step and nh time step. However, that is not the case in Fig. AS (see along column), suggesting that our hypothesis is incorrect. Future work should explore other plausible hypotheses for the primacy effect in recurrent models.
1
2
Cosine Similarity
Serial Position
3
4-
5-
6-
7-
8-
9-
1
2
4
5
6
7
8 9
Serial Position
Figure A5: Cosine similarity of memory representations between different serial positions. We show the cosine similarity between the memory representation of different serial positions, averaged across all trials in the VSR task. We consider the LSTM-256 model for the above plot.
A4.3 Visualizing Neural Correlates in Task Switching
In the STS experiment, we first take the hidden representations each lprobe for all the test trials within the task and then perform t-SNE [88 to project these hidden representations to 2D.
Next, we introduce two types of color schemes to label hidden representations. First, on the left panel in Fig. (5,, we color-code the points based on the ground truth at that particular lprobe. We assign a blue color to label those hidden representations where the ground truth is 0 i.e. when the marker is either the left or top of the grid and we assign an orange color label those hidden representations where the ground truth is 1 i.e. when the marker is either at the right or bottom of the grids. In essence, we color-code the points based on the ground truth response for that particular lprobe-
Second, on the right panel in Fig.5A. we color-code the hidden representations based on the task that was supposed be performed oprobe. We assign green color label those hidden representations when the task at hand is to discriminate top versus bottom and we assign a red color to label those hidden representations when the task at hand is discriminate left versus right.
By color-coding the neural representations based on the task to be performed at Iprobes we observed the emergence of two distinct clusters corresponding to the two tasks involved, namely top-versus-botte and rett-versu--rigt. This observation highlights how the model's neural representations encode and maintain task identity over time, enabling accurate performance in task-switching experiments. Importantly, it should be noted that at lyrobe. the model does not receive any explicit information regarding the task be performed at that specific time step.
A4.4 Visualizing Task- Task-Specialized Neural Clusters
Task Variance (TV) 6 is scalar that quantifies the selectivity hidden unit within a recurrent model towards a specific task, where higher values indicate greater selectivity. To obtain the results
24

---

in Fig.5,, Cn the main text. we first compute the TV for each hidden unit for each task, resulting TV matrix of size 10x256 for LSTM-256. We normalize the task variance of each hidden unit such that the maximum normalized TV across all tasks was 1. We then perform t-SNE along the TV dimensions and project the TV vector of each hidden unit to 2D. Subsequently, we identify 8 clusters among the hidden units and assign unique colors to each cluster (Fi In line with [63], we chose the optimal number clusters based on the highest silhouette score. We then group the hidden units based on their clusters. The sorted TV matrix based on the clusters is shown in Fig. 5| in the main text, where the intensity denotes the magnitude of normalized TV values.
Given a pair of tasks A and B, the fractional task variance (FTV) [63] defines the preference of hidden unit for one task over the other. For particular hidden unit in WM model, the FTV with respect to task A and task
defined as (A, B) (V)(A)+TV; (B)) where (V; and (B) are the TVs for tasks A and B, respectively. FTV(A. B) ranges from -1 1, with being more selective to task A over B, and vice versa. FTV = O, when the hidden unit is neutral about both tasks and stays equally active during both tasks. To obtain the results in Fig. in the main text, for each pair of tasks, we compute the FTV for each hidden unit and plot the histogram of the total number of hidden units based their FTV values. In the histogram, the x-axis denotes the FTV whereas the y-axis denotes the proportion hidden units. If the histogram distribution is skewed towards FTV versa if the distribution is skewed toward FTV = -1 If the distribution is unimodal in the center, implies that the majority of the hidden units do not have task preference and equally active in both tasks. In the case the bimodal distribution where the two modes are located at both ends, this implies that there are two separate populations of neurons, dedicated two individual tasks respectively.
(V)
it means that the majority of the hidden units
selective Task A and vice
A4.5 Causal Analysis of Neural Population
In A6 show the drop in performance for different tasks, when specific neural clusters from the network (shown in Fig. SC) are lesioned. To lesion a neural cluster, we select neurons within that cluster and out their projection weights task-specific classifiers. As anticipated, the corresponding causal evidence when distinct clusters are lesioned. For instance, when cluster 6 is deactivated, observe the maximum decline in performance for the VSR and VSRec tasks. Similarly, when cluster 3 is lesioned, the maximum drop in performance is for the CD task.
note
A4.6 Visualizing Task Similarity Matrix
We also visualize the task similarity matrix based on the learned task embeddings for each WM model different memory capacities Fig. A13 As expected, we observe the brightest values along
SC
SPR
Normaltzed Performance Drop
SMU
Task
STSC
VSR
VSRe
CD
CS
4
3
Cluster
Figure A6: Effect lesioning neural clusters on task performance. The y-axis shows the different
tasks and the x-axis shows the neural cluster which is lesioned. See Sec. 5 and Fig. 5 in the main text for neural clusters and analysis.
25

---

the diagonal of the task similarity matrix as the embeddings from one task compare against itself. Interestingly. we found that in recurrent architectures, task similarities between different conditions within the CD task are high Fig. -1I However, we did not observe such strong similarities in transformers Fig. A13I-I.
26

---

A
RNN-96
B
RNN-256
RNN-1024
Lok Leggth
Lot Leeeth Largth
Let Lespt)
Top-1 Accuracy
Top-1 Accuracy
0.6
Lit Lergth
0.6
Lergth
06
Langth:
Lsd Length
Lit Leegth
Lat Length
3.4
Human
04
0.8
0.8
0.6
0.0
6
8
0.0
6
Serial Position
Serial Position
Serial Position
D
GRU-96
GRU-256
F
GRU-1024
1.0
1.0-
Langra
1.0
Length
Top-1 Accuracy
Top. Acturaac
Lad Langtt
Lie Lasgaa
Larpth
0.6
Lat Lengt
0.8
Le Lengra
Lise langth
Lod Lesplii
LAK Lanyl
-1-
Lod Letpti
0.1
1.0
Human
0.7
1.0
Human
0.7
1.0
Human
0.8-
0.8-
0.8
0.6
Serial Position
Serial Position
Secial Position
LSTM-96
LSTM-256
LSTM-1024
1.0
1.0
1.0
Lae Langth
Langth
Lac Langth
Top-1 Accuracy
0.8-
Lst Leggt
Top-1 Accuracy
0.9
Lis Leepth
Top- Accuracy
0.9
Lot length
Lat Langth
Las Lerpth
Lat Length
Lecoth
Larath
Leagth
Lergth
Lecpth
Leepth
9.4
1.0-
Human
1.0
0.7
3.0
Human
0.2
0.5
0.6
0.6
844
8A
2.4
0.0
0.5
Serial Position
Serial Position
Serial Position
Lat Langth
Lie Langra
Lst Leegth
Lis Leneth
List Leeoth
Appuracc
0.8
Lst Length
Acpuraoy
0.8
Us Long
Top-1 Accuracy
3.8
Ltd Length
Largth
Laqe
Langih
0.6
0.6
0.6
0.2
0.6
0.2
0.6
0.2
0.6
0.0
0.0
2
0.0
Serial Position
Position
Figure A7: Behavioral accuracy as a function of list lengths in VSR task. In addition to Fig. in the main text, we show the results of three recurrent architectures (first 3 rows) with three memory capacities (3 columns) and the transformer architecture with two memory capacities (J-L). The chance level 0.1 for all the above plots.
27

---

A
RNN-96
B
RNN-256
RNN-1024
Homa
Top-1 Accuracy
0.n
Top-1 Angurans 3
n.
mod4250
Top Aceurare
na
2065- 1834 kd
2009-3024
0.7
0.4
0.4
0.4
0.3
2
Serial Position
Serial Position
Serial Position
GRU-256
GRU-1024
1.0
Hamaa Har
1.0-
tazes Poase it
1.0
Hoove Ri
Toge Ancuracy
Tog-1 Aemracy
Top Accuracy
0.9-
Homee
9.-
CRC 2002204300 w
oad 1094 1t
0.6
10.5
Serrial Position
Serial Position
Serfal Position
LSTM-96
LSTM-1024
Top-1 Aogurssy
Top-1 Acouracy
0.9
LSTU -
Top Accuracy
07
0.0
0.7
0.7
0.4
0.6-
0.6
0.5
2
Serial Position
Secial Position
Sorial Position
J
TRF-96
K
TRF-256
L
TRF-1024
1.0
3.0
1.0
Homen Reawe
Top-1 1 Accuracy
0.9-
Acquracy
Top-1 Aequraty
192 240
TIN0224 12
0..5
8.6
3.7
0.5
0.5
Sersal Position
Serial Position
Serial Position
Figure A8: Behavioral accuracy as a function of retention interval in VIR task. In addition to Fig. 4 in the main text, show the results of four architectures with different memory capacities. The layout interpretations follow Fig. A7 The chance level is 0.5 for all the above plots.
28

---

A
RNN-86
B
RNN-256
RNN-1024
1.0
1.0
Top-1 Accaracy
0.8
SpatasSpelii
1op-1 Accuracy
0.8
Top Accuracy
0.8
SpstsiAtinal
Spanit
Spetal
Maaal Spenal
Vical-potte
0.6
6
0.8
0.4
L0
Human
0.4
1.0
0.4
1.0
0.4
0.4
84
Cognitive Load
Cognitive Load
Cognitive Load
GRU-96
GRU-256
1.0-
1.0
1.0
Top-1 Accuracy
0.8-
Tog-1 Accuracy
a.a
Top-1 Accuracy
0.5-
traae--petai
thaakiperal
Wealtheal
LP
Human
Human
Human
0.0
0.0
Cognitive Load
Cognitiva Load
Cognitive
G
LSTM-96
LSTM-256
LSTM-1024
Top-1 Accuracy
0.8
Spata:--peted Yarall Spatal
Top-1 Accurace
Spetae-ppatill
Top-1 Accurace
Spatar Yaral Spetal
Vinal Sperial
Yaaad-Vaaak
Maaad Ntaud
Human
Human
Human
0.5
0.8
0.6
0.0
3
0.0
3
Cognitive Load
Cognltive Load
Cogaltive Load
K
1.0-
Sustal Sponal
Spoisol Spstal
Rumta Sporoal
Top-1 Accurecy
0.8
Tog-1 Accuracy
0.8
Spard Voxil
Top-1 Accuracy
0.8
Spatal Yexx
Spatal
Soxial
0.6
066
06
0.1
Human
9.4
1.0
0.2
0.2
82
11.
0.4
0.0
Cognitive Load
Cognitive Load
Cognitive Load
Figure A9: Behavioral accuracy as a function memory domain confficts in CS task. In addition FCg in the main text, we show the results of four architectures with different memory capacities. The layout interpretations follow Fig. A7 The chance level is 0.04 for all the above plots.
29

---

1.0
LSTM-96
LSTM-256
Top-1 Accuracy
0.8
LSTM-1024
GRU-96
GRU-256
0.6
I
GRU-1024
RNN-99
0.4
RNN-2256
RNN-1024
0.2
TRF-96
TRF-256
TRF-1024
0.0
3
5 Set Size
4
7
+ Human
Figure A10: Behavioral accuracy function of set size in SMU task. In addition to Fig. 40 in the main text, we show results of four architectures with different memory capacities. The layout interpretations follow Fig. A7 The chance level is 0.1 for the above plot.
30

---

1.0
RNN-96
RNN-256
RNN-1024
Petacc tesu 18ll-
Artar
0.8
0.0
Puter--arras bet
0.7
Nettr Buitre DE to
Dettera Deroue
FatienSooo:: 2et
0.8
0.
0.5
0.3
0.8
0.3
22344
0.3
Serial Position
Serial Position
Serial Position
D
GRU-86
Neterr
E
GRU-256
F
1.0
GRU-1024
0.s
demme tit-
Tog- Acguracy
Rutooe tranee but Datre:
Top-1 Aucurac
Natera Pulus B.E
Pazurs Eutre:
de txlf in
0.0
Neture frstes
9.6-
Human
Human
Human
0.
0.1
123
Serial Position
Serial Position
Serial Posttion
G
H
LSTM-1024
Top-1 Accuracy
0.0
Futien Sttrec
Top-1 Accurncy
Peture Ditre tutt
Top-1 Acruncy
Peters Distue DtS.
8.8
haser therrs
Dolne tatt
Pelton Hlubur
Ruberr-Damme 3et 12
3.
6.-
Dettars dottne
8.6
Human
Humain
Human
0.8
3.6
0.3
3
Seria] Positzon
Serial Position
Serial Position
TRF-96
Tetwrr Teterr-Battre: Dttte:
K
TRF-256
DetemnZetse
1.0
TRF-1024
Accuracy
kuowe therr:
Top-1 Anguracy
Paturr
0.4
Pattace tutraa IV3
Fubser--aatt:c Giemi 3et
Detams Strme De5
Adtin
Human
Human
0.5-
Human
0.0
5.0
0.01
0.5
0.4
0.6
Position
Sertal
Figure A11: Behavioral accuracy as function of memory resolution in VSRec task. In addition to Fig. 4E in the main text, we show the results of four architectures with different memory capacities. The layout interpretations follow Fig. A7 The chance level is 0.5 for all the above plots.
31

---

RNN-96
RNN-255
1.0
1.0
1.01
Top-1 Accoracy
Top-1 Accuracy
2AN-206
INN-CEA
Top- Arnungy P
R020-1926 L924 Doke
FOOU LISE Driestatin
INN
SNN 200
Crepection Drivesatioo
IAN Conpectiion
RNSS Doglantee
Homan
0.8-
0.8-
Dluass Sox
Juutess Soa
Haman Siin
Cap
Itamn Cap
Set Sizze
Set Size
GRU-96
E
GRU-236
F
GRU-1024
1.0
1.0
1.0
Top-1 Accarscy
Top-1 Accuraay
Top-1 Accuracy
UHG LIDE Clilr
Oicentatio
CRCC-256 250
CPCL 299 Degladdo
CRR2-100 Doegunciee
0.8-
0.8
0.2
Hanas Stn
Duuaai Sere
Haman Sps
Cap
12
Sizze
Size
G
LSTM-96
H
LSTM-260 250
LSTM-1024
1.0
1.0
1.0
Top-1 Accurssy
USTH 1o:
LSTM Chintttiis
USTM DAKK
Top-1 Accurscy
LKTN endiato
LSTH Chike
LSTINO
LSTN-256 Dap
LSTH
LSTH Doejantiie
LSTN Dejuackee
LITH
Chojunctee
0.8-
Top-1
0.81
0.8
Honui Sin
Hanoe
Honut San
Cop
Set Size
Size
TRF-46
K
TRF-256
TRF-1024
1.0
1.0
1.0
2122 Coke
TRP Dulec
TRP 1024 Cole
Top-1 Accuracy
0.9
extitios
0.9
2 Driccaates
Tal Ooiraictton
0.8
TRA-256
0.8
710 be Coquaxtien
192-296 Deglaadto
Tap-toc Chspectios
Homor
Top-1
Homon Sox
Thame Stoe
Thrass Six
Cap
Cap
12
10
Set Size
Set. Sizee
Set Size
Behavioral accuracy as function of features and conjunctions in CD task. In addition to Fig. 4 the main text, we show the results of four architectures with different memory capacities. The layout interpretations follow Fig. A7 The chance level is 0.5 for all the above plots.
Figure A12:
32

---

RNN-96
RNN-256
C
RNN-1024
198
575
ST5
5T5
YIR
co cake
CD chic
UD
CD Oclanatke
20
61
CD Toarrraa
O Sat
De
CD Dee mn.o
O
GRU-98
GRU-258
F
GRU-1024
1.0
1m
SHG
USSac
0.8
CO.
cue
CD
CD cua
00
ON
0.6
D
H
LSTM-1024
40
90
0.4
545
95te
VSH
0.2
co.
CD Cake
cn
en Cre
on Pee
co.Ou Over
GOLONS
CDLOR
C Se
0.0
J
TRF-256
TRF-1024
Sc
50
508
VI
VIN
VIC
cD St
CD Wt
cn Soe
day
the
100
Sa
Dep
ci
Figure A13: Visualization of similarity matrix based on learned task embeddings from WM models with different memory capacities. We the task embedding learned by each WM model with different memory capacities and compute the cosine similarity between a pair of task embedding vectors from two corresponding tasks. We present the similarity matrix between tasks. Each row of the similarity matrix is normalized such that the maximum similarity is and the minimum is 0. See the color bar on the right for the normalized similarity values. The brighter values indicate that the leamed task embeddings are more similar for the corresponding tasks.
33

---

1.0-
SER Task
CD. Color Task
Validation Accuracy
0.8
CD. Orientation Task
CD Size Task
CD Gap Task
0.6-
CD, Conj Task
VIR Task
VSR Task
0.4
VSRec Task
CS
SI Task
0.2-
SMU Task
STS Task
0.0
50
100
150
200
Epoch
Figure A14: Training curve of individual tasks for LSTM-1024. The y-axis shows the validation accuracy for the tasks and the x-axis shows epochs. The training curves for other models show similar trends; thus, they are omitted for simplicity. Here, we plot the model with the highest joint validation accuracy, i.c., LSTM-1024
34

---

A5 Datasheet for WorM Dataset
A5.1 Motivation
1. For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
Despite significant research progress in studying individual aspects of working memory (WM), there remains a huge gap in the broader study of WM. We take initial steps in this direction by establishing systematic and quantitative methodology to study the multi-facets of WM. This dataset serves as valuable resource for communities in cognitive psychology. neuroscience, and AI, offering a standardized framework to compare and enhance WM models, investigate WM's neural underpinnings, and develop WM models with human-like capabilities.
2. Who created the dataset (e-g,, which team, research group) and on behalf of which entity (e-g,, company, institution, organization)?
The dataset was created by Ankur Sikarwar and Mengmi Zhang from Deep NeuroCognition Lab in affiliation with Center for Frontier Al Research (CFAR), and Institute for Infocomm Research (I2R), from Agency for Science, Technology, and Research (A (A*STAR), Singapore.
3.
Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number
This creation of the dataset is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: RP-2221-2222) and its NRFF award NRF-NRFF11-2023-00
4. Any other comments?
N/A
A5.2
Composition
1. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide description.
The dataset consists of 10 distinct working memory tasks. Each data instance in the dataset represents a trial that time-based and includes stimuli and corresponding responses for various time steps within the trial.
2. How many instances are there total (of each type, if appropriate)?
The dataset consists of 10 working memory (WM) tasks. One of these tasks i.e. the CD task contains 5 separate task conditions. each of which has separate training, validation, and testing trials. For each of these task conditions within the CD task and 9 remaining WM tasks, there are 86,400 trials for training, 9,600 trials for validation, and 9,600 trials for testing.
3. Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
Since the dataset is synthetic, is possible to generate an arbitrary number of samples with the provided code. For consistency, we limit the total number of generated samples, including training, validation, and testing, to 105,600 trials for each task.
4. What data does each instance consist of? "Raw" data (e-g., unprocessed text or images) or features? In either case, please provide a description.
Every instance in the dataset includes image stimuli corresponding each time step within the trial, along with the corresponding responses for that particular trial.
5. Is there a label or target associated with each instance? If so, please provide description.
35

---

The label for each instance is the ground ground-truth truth response corresponding to the task and the specific trial.
6. Is any information missing from individual instances? If so, please provide description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g.. redacted text.
There is no missing information; the data is complete.
7. Are relationships between individual instances made explicit (e.g,, users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. N/A
8. Are there recommended data splits (e.g., training, development/valication, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
We propose our training, validation, and testing splits for each task that is available on the GitHub repository.
9. Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
We generate trials through random sampling. While it theoretically possible to generate the same trial more than once, the probability of this occurrence is extremely low due to the vast number of possible combinations of stimuli across multiple time steps.
10. Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to relies external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, well links other access points, as appropriate. The dataset self-contained.
11. Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications?? If so, please provide description.
N/A. The dataset is synthetic
12. Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? so, please describe why.
N/A. The dataset is synthetic.
A5.3 Collection Process
N/A. No data collection process was involved. The dataset has been synthetically generated and we make our dataset generation code available on GitHub. The human data was collected from the existing literature. We directly borrowed their human data and plot them side by side for comparisons with WM models.
A5.4 Prepracesin//ddannnng/alllnggg
1. Was any proproesiin//teniigghbbiiigg of the data done (e-g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remaining questions in this section.
No preprocessing was done.
2. Was the "raw" data saved in addition to the preprccesee/ckeened//bbee data (e.g., to support unanticipated future uses)? If so, please provide link other access point to the "raw" data.
N/A
36

---

3. Is the software that was used to preprocess/clean/llbeel the data available? If so, please provide a link or other access point.
N/A
4. Any other comments?
N/A
A5.5
Uses
1. Has the dataset been used for any tasks already? ? If so, please provide description. The dataset being used for the first time.
2. Is there a repository that links to any or all papers or systems that use the dataset? so, please provide link or other access point.
N/A
3. What (other) tasks could the dataset be used for?
The dataset itself contains well-defined WM tasks and is intended be used for those specific tasks.
4. Is there anything about the composition of the dataset or the way it was collected and pervvessee//ceanee//abbeee
that might impact future uses? For example, is there anything that dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (c.g., stereotyping, quality of service issues) other risks or harms (e.g., legal risks, financial harms)? If so, please provide description. Is there anything a dataset consumer could do to mitigate these risks or harms?
No. there are no risks or harms of the dataset. The dataset contains no human data.
5. Are there tasks for which the dataset should not be used? If so, please provide description.
No.
6. Any other comments? N/A
A5.6
Distribution
1. Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
The dataset publicly available.
2. How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have digital object identifier (DOI)?
The dataset and the supplementary code available on the GitHub repository.
3. When will the dataset be distributed?
The dataset available from June 2023.
4. Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)) If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
The dataset released under the MIT License.
5. Have any third parties imposed IP-based or other restrictions the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
N/A
37

---

6. Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any supporting documentation. N/A
7. Any other comments? N/A
A5.7
Maintenance
1. Who will be Ankur Sikarwar supporting and maintaining the dataset.
the dataset?
2. How can the ovner/curator//manger the dataset be contacted (e-g,, email address)?
ankursikarwarde @ gmail.com
3. Is there an erratum? If so, please provide link other access point.
The GitHub repository will reflect any changes or improvements made to the dataset.
4. Will the dataset be updated (e-g., to correct labeling errors, add instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated dataset consumers (e.g,, mailing list, GitHub)? This information will be available on GitHub.
5. If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g,, were the individuals in question told that their data would be retained for fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
N/A
6. Will older versions of the dataset continue to be supporte//ustee/maiittaeee! If so, please describe how, If not, please describe how obsolescence will be communicated to dataset consumers.
N/A
7. If others want to extend/augmeent/uiil on/contribute to the dataset, is there mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified?? If so, please describe how. If not, why not? Is there a process for communicting/distrrittnn these contributions dataset consumers? If so, please provide description.
People who want to contribute encouraged to get in touch with the authors.
8. Any other comments?
N/A
38


Reviews:
[{"review_id": "ifaaBTmHcF", "reviewer": "Reviewer_Unmh", "summary": "", "questions": "", "limitations": "Yes", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The authors' effort to bridge the gap between neuroscience, cognitive science, and AI is significant. They identify a key gap in research – the lack of integrative, systematic study and benchmarking across multiple WM tasks, and take substantial steps to address this through the introduction of the comprehensive, large-scale WM benchmark dataset.\n* The authors introduce a comprehensive set of 10 WM tasks and use them to evaluate state-of-the-art recurrent neural networks and transformer models\n* The models produce key effects from the human data and classic working memory literature, including primacy effects and recency effects\n* The paper clearly outlines its methodology, experiments, and provides detailed explanations of the tasks involved in the study. This transparency enhances the reproducibility of the research.\n* The \"neural\" analysis of the models is compelling and illustrates the utility of looking into the network to interpret its dynamics", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DuSMughJ1m", "reviewer": "Reviewer_9dCi", "summary": "", "questions": "", "limitations": "The authors discussed limitations.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The authors study DL models from the perspective of working memory, which is a critical concept of understanding human intelligence. It is a critical topic from the perspective of both DL and neuroscience that deserves more attention from the community.\n- The authors provide a large-scale and carefully curated dataset, with several DL models trained and human psychophysics experiments conducted. This provide a solid and high-quality contribution.\n- The authors provide a diverse set of experiments that systematically evaluate different perspective of WM of models.\n- The paper is nicely written.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3t2AIy1USE", "reviewer": "Reviewer_AjzL", "summary": "", "questions": "", "limitations": "Generalization of the obtained results to studies on human WM is in my opinion questionable. Both dataset and employed computational models are limited in their scope, so that it is not clear whether any statements beyond that studied models behave in certain way on the introduced dataset are eligible.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Relevant topic: exploring analogies between neural WM implementation and potential artificial counterparts is fruitful direction\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3UMfOmWmAy", "reviewer": "Reviewer_1qDZ", "summary": "", "questions": "", "limitations": "The primary limitation in the work is that there is no discussion regarding the primacy bias which was observed and its connections with gradient vanishing / exploding property in RNNs. A big problem in RNNs is that distant past is lost over multiple iterations and updates which introduces the question of why the authors are seeing a primacy bias? If you increase the task-lengths, does this bias still exist?\n\nBeyond this, the authors have addressed most limitations and potentially negative societal impacts.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The authors provide a decently large dataset with multiple different tasks having different features, difficulties and capacity to test for different properties of working memory\n- The experimental results indicate an important phenomena of alignment of recurrent neural networks to the desirable / observed properties of working memory", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "SBP0pJxFRc", "reviewer": "Reviewer_E3J7", "summary": "", "questions": "", "limitations": "Limitations of current WM models are briefly discussed. Limitations of the approach/dataset/benchmark are not discussed much - this would be very insightful to readers as to potential routes for further research. Potential negative social impacts of work are not discussed, although I cannot think of any.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Approach seems novel (although more contextualization is needed)\n- Synthetic data generation technique is lucrative\n    - Can be used for new tasks as well or to change tasks and generate more data\n- Paper results have the potential to be highly insightful and impactful\n- Interesting results\n\nEdit:\n-This work could be very insightful in testing the flaws of current AI systems when compared to humans as well as providing potential insight into phenomena observed in human working memory.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper receives mostly positive reviews except one. The positive reviews acknowledges the size and insightful prelimiary empirical studies with the proposed dataset. Although the one negative reviewer raised seemingly valid questions, the authors provide comprehensive answers to them and the reviewer did not actively respond to the authors' response. Given the value of the proposed dataset, the AC recommends to accept the submission to NeurIPS 23 DB track.
Decision: Accept (Poster)