Reconstructing Spatio-Temporal Trajectories of Visual Object 1 Memories in the Human Brain

28 Our understanding of how information unfolds when we recall events from memory 29 remains limited. In this study, we investigate whether the reconstruction of visual 30 object memories follows a backward trajectory along the ventral visual stream with 31 respect to perception, such that their neural feature representations are gradually 32 reinstated from late areas close to the hippocampus backwards to lower-level sensory 33 areas. We use multivariate analyses of fMRI activation patterns to map the constituent 34 features of the object memories onto the brain during retrieval, and EEG-fMRI fusion 35 to track the temporal evolution of the reactivated patterns. Participants studied new 36 associations between verbs and randomly paired object images in an encoding phase, 37 and subsequently recalled the objects when presented with the corresponding verb 38 cue. Decoding reactivated memory features from fMRI activity revealed that retrieval 39 patterns were dominated by conceptual features, represented in comparatively late 40 visual and parietal areas. Representational-similarity-based fusion then allowed us to 41 map the EEG patterns that emerged at each given time point of a trial onto the spatially 42 resolved fMRI patterns. This fusion suggests that memory reconstruction proceeds 43 backwards along the ventral visual stream from anterior fronto-temporal to posterior 44 occipital and parietal regions, in line with a semantic-to-perceptual gradient. A linear 45 regression on the peak time points of reactivated brain regions statistically confirms 46 that the temporal progression is reversed with respect to encoding. Together, the 47 results shed light onto the spatio-temporal trajectories along which memories are 48 reconstructed during associative retrieval, and which features of an image are 49 reconstructed when in time and where in the brain. 50 Introduction 51 In contrast to the substantial scientific literature on visual perception, relatively little is 52 known about how the human brain reconstructs content that is retrieved from memory. 53 We here investigated which features of a visual memory are preferentially reactivated 54 when objects are recalled from episodic memory, and where in the brain and when in 55 time such feature reactivation occurs. We used multivariate pattern analyses 56 combined with an experimental design where the perceptual and conceptual attributes 57 of visual object stimuli were manipulated experimentally. Multivariate similarity-based 58 EEG-fMRI fusion was then used to resolve these spatial patterns in time to recover 59 the dynamic information flow during memory reconstruction. Together, the multivariate 60 analyses of fMRI patterns and the EEG-fMRI fusion offer complementary insight into 61 the reconstruction stream of lowand high-level features of episodic memories. 62 Visual perception of a realistic object triggers an information processing cascade along 63 the ventral visual stream that follows a gradient of increasing abstraction. While early 64 processing stages are dominated by low-level perceptual representations, later stages 65 along the ventral visual stream increasingly code for conceptual features of the 66 perceived object (T. Carlson et al., 2013; Cichy et al., 2014; Kolb et al., 1995; Martin 67 et al., 2018). After approximately 250 ms, the information reaches the hippocampus, 68 where one cell type, referred to as concept cell, represents an object as a highly 69 abstract and individual concept (Clarke & Tyler, 2015; Quiroga, 2012). This final 70 processing stage within the hippocampus also binds together information from 71 different sources (including the dorsal stream) and serves the longer-term storage of 72 associative information (Danker et al., 2017; Eichenbaum, 2001; Horner et al., 2015; 73 Rolls, 2010). 74 Computational models then assume that once stored, the hippocampus can use a 75 partial reminder to recover the missing, associatively linked information, a process 76 termed pattern completion (Marr, 1971; O’Reilly & McClelland, 1994; O’Reilly & 77 Norman, 2002; Teyler & DiScenna, 1986). Pattern completion in turn dictates the 78 reactivation of neocortical brain regions that re-instantiate the various features of the 79 previously encoded episode (Marr, 1971; Teyler & Rudy, 2007; Moscovitch, 2008). 80 Such neocortical reinstatement presumably leads to the subjective re-experiencing of 81 a past event (Tulving et al., 1983). 82 These computational models have provided the dominant framework for studying 83 hippocampal-neocortical interactions during memory recall. One of their central 84 characteristics is that they describe memory reinstatement as an all-or-none process 85 and assume that during recall, neocortical information contained in a memory trace is 86 re-established the way it was represented during initial perception. Indeed, indices for 87 such copy-like encoding-retrieval-similarities (ERS) have been found by various 88 neuroimaging studies using similarityor classification-based multivariate analyses 89 (Danker & Anderson, 2010; Rissman & Wagner, 2012; Staresina et al., 2012). ERS 90 has been found in activation patterns for a variety of memory features, e.g., individual 91 episodes, semantic category, sensory modality, or task context during learning (Wing 92 et al., 2014; Ferreira et al., 2019; Griffiths et al., 2019; Jiang et al., 2020; Polyn et al., 93 2005). Further, evidence for pattern reinstatement has been found all along the 94 sensory processing pathways activated during the initial perception of an event, 95 including areas most proximal to the hippocampus (Staresina et al., 2012) as well as 96 more distant neocortical structures (Polyn et al., 2005; Wing et al., 2014). Numerous 97 studies additionally showed that successful retrieval is associated with the interplay 98 between hippocampal activity and cortical reinstatement of encoding patterns during 99 retrieval (Bosch et al., 2014; Horner et al., 2015; Staresina et al., 2012). Together, the 100 previous findings support the central premise of the aforementioned computational 101 models that hippocampal pattern completion is associated with a more or less truthful 102 reinstatement of the encoding patterns. In the present study, our first goal is to 103 decompose visual object memories into perceptual and conceptual components, and 104 to test if both types of features are equally reinstated, in the same areas representing 105 these features during memory encoding. The latter question is particularly interesting 106 in the light of recent evidence that information reactivated from memory is not 107 necessarily decodable from the same regions during encoding and retrieval (Favila et 108 al., 2020; Xue, 2022). 109 While the above-mentioned classical theories and neuroimaging studies focus on the 110 spatial localization of reactivated memory representations, less focus has been placed 111 on the temporal progression of the reactivation processes. We therefore asked next 112 how this spatial mapping of the retrieved contents unfolds over time. While the finding 113 of encoding-retrieval similarity suggests at least a degree of copy-like reinstatement, 114 it is well possible that different features of a memory are reactivated with different 115 temporal priority. Such prioritisation could naturally emerge from the hippocampus’ 116 differential connectivity with neocortical regions during retrieval. Once pattern 117 completion is initiated, most information is relayed via the subiculum to the entorhinal 118 cortex which in turn has back-projections to other cortical areas (Chrobak et al., 2000; 119 Rolls, 2013; Staresina & Wimber, 2019). A majority of direct hippocampal and 120 entorhinal efferents terminate in late areas of the visual processing hierarchy where 121 representations cluster in abstract-semantic categories, while there are few 122 connections to low-level sensory areas (Dalton et al., 2022; Insausti & Muñoz, 2001). 123 We therefore expect that memories are initially reinstated on a conceptual level, and 124 information coded in lower-level visual areas can only be reached at later stages of 125 memory reconstruction (Insausti & Muñoz, 2001; Linde-Domingo et al., 2019). Indeed 126 recent studies using EEG decoding and behavioural responses demonstrated a 127 temporal prioritization of high-level conceptual over perceptual features during 128 memory retrieval (Linde-Domingo et al., 2019; Lifanov et al., 2021). Where in the brain 129 these different memory features are reinstated, and whether the conceptual-to130 perceptual gradient truly maps onto a sensory backpropagation stream, remains an 131 open question that is central to our study. 132 In the present work, we used an associative recall paradigm with visual object stimuli 133 that allowed us to differentiate between their low-level perceptual and high-level 134 conceptual features. First, using multivariate analysis of the fMRI patterns only, we 135 investigated the spatial location of these lowand high-level feature representations, 136 that is, where in the brain the different object dimensions can be decoded during 137 retrieval compared to the initial encoding. Second, we asked whether the object 138 information reconstructed during retrieval propagates backwards along the ventral 139 visual stream, from high-level semantic to low-level visual areas, using EEG-fMRI 140 fusion to resolve spatial activation patterns in time. 141