Predicting Event Memorability from Contextual Visual Semantics

Episodic event memory is a key component of human cognition . Predicting event memorability , i.e. , to what extent an event is recalled , is a tough challenge in memory research and has profound implications for artificial intelligence . In this study , we investigate factors that affect event memorability according to a cued recall process . Specifically , we explore whether event memorability is contingent on the event context , as well as the intrinsic visual attributes of image cues . We design a novel experiment protocol and conduct a large-scale experiment with 47 elder subjects over 3 months . Subjects ’ memory of life events is tested in a cued recall process . Using advanced visual analytics methods , we build a first-ofits-kind event memorability dataset ( called R3 ) with rich information about event context and visual semantic features . Furthermore , we propose a contextual event memory network ( CEMNet ) that tackles multi-modal input to predict item-wise event memorability , which outperforms competitive benchmarks . The findings inform deeper understanding of episodic event memory , and open up a new avenue for prediction of human episodic memory . Source code is available at https : //github.com/ffzzy840304/Predicting-Event-Memorability . 1 Introduction . Episodic event memory is a key component of human cognition and intelligence . Maintaining high-level event memory is good for mental health and executive functioning [ 34 ] . Episodic memory can be enhanced by cognitive training , such as regular photo review that triggers the reactivation of certain memory traces [ 25 , 42 ] . Recent advancements in visual lifelogging , including affordable hardware and powerful visual analysis software , make it possible to implement lifelog-based memory intervention [ 19 , 31 , 43 ] . However , without clear understanding of what factors influence event memory ( e.g. , what events have higher value for training ? how to select “ good ” photo cues to boost training effect ? ) , it is difficult to design effective cognitive intervention programs . This study attempts to understand event memory , i.e. , to what extent an event can be recalled . We probe the recall process with visual cues , and explore factors that influence event memorability . We aim to extend the understanding of event memory , beyond simple image recognition [ 22 ] and memory of event categories [ 8 ] , to the prediction of item-wise event memorability using contextual visual semantics . A few major gaps exist in this field . First , there is scarcity of data for evaluating event memorability . A sensible analysis requires rich event knowledge , including spatial-temporal information , personal experience and profiles , as well as quantifiable memory performance on the respective events . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Remember Know New Encoding Context Testing Context Predictive Model CNN Real-life Experience Test Set ( Stimuli ) Memory Test Visual Saliency Human Factors R3 Dataset Semantic Extraction Information Theoretic Entropy Contextual Event Segmentation Memory Score Considering that life events are highly personalized and seldom repeatable , it is difficult , if not impossible , to collect such data in laboratories by replicating a controlled process . This makes the generation of large dataset for event memorability study a major challenge . Second , event memory involves a complicated cognitive process in the human brain , so that the mapping relationship between event information and the memory outcome is obscure and fuzzy . It is affected by a number of factors , such as personal experience , cognitive status , and new , potentially conflicting input . Therefore , the memory outcome often has immense variability , making it extremely difficult to be predicted reliably . This study fills these gaps in three aspects . First , we develop a novel experiment protocol to collect personal life event information along with the human subjects ’ memory outcome on selected events ( Figure 1 ) . We build a first-of-its-kind event memorability dataset consisting of 10 , 654 valid samples with rich information about event context and visual semantic features . Based on data collected in a large-scale user study with 47 elder subjects , each in three months , we apply a suite of visual analysis methods , such as contextual event segmentation ( CES ) and information-theoretic entropy analysis , which collectively extract visual semantic features and contextual information . Second , we provide evidence that event memorability is contingent on extrinsic , contextual information , as well as intrinsic visual attributes . Third , inspired by image memorability studies [ 45 ] , we develop a simple yet effective contextual event memory network to predict event memorability by combining multimodal visual semantics , including the image cues and multiple aspects of contextual information . The model predicts item-wise event memorability with reasonable accuracy and serves as a baseline for more sophisticated methods . The dataset and baseline predicative model establish the foundation for developing computational models to decode human episodic memory . 2 Related Work . Episodic event memory has been intensively studied in cognitive psychology , exploring the human behaviors , cognitive processes and neural mechanisms of memory [ 11 , 40 ] . Regarding factors that influence how much an event is remembered , most studies focus on the macro-level , collective characteristics of events that influence subsequent memory . For example , distinctive events are remembered better than routine ones [ 21 ] ; episodic details are gradually lost with time ; memory could be distorted by interfering cues that re-activate alternative neural processes [ 25 ] . Nevertheless , little is reported on the micro-level properties of events that contribute to item-wise event memorability . We believe this issue is important because it opens up new possibilities of computational intelligence in multiple domains such as image retrieval [ 30 , 50 ] , cognitive intervention [ 8 , 51 ] and diagnosis [ 2 ] . Recent works in neuropsychology resort to brain imaging to decode the memory of individual events [ 3 , 5 , 39 ] . However , the brain imaging data is prohibitively difficult to collect , requiring special equipment and controlled environments , which restricts its application in the daily life . Meanwhile , there is a sizeable body of research that investigates image memorability . It is generally accepted that memorability is an intrinsic property of images based on the recurring evidence that human subjects show considerable consistency in image recognition tests [ 4 , 23 , 22 ] . Accordingly , computational models have been developed to predict image memorability from the visual semantic information , which is partially made possible by high-quality image memorability datasets , such as , SUN-Mem [ 22 ] , LaMem [ 28 ] , FIGRIM [ 8 ] , Mem-Cat [ 17 ] , LNSIM [ 33 ] , etc . Leveraging on deep convolutional neural network ( CNN ) and well-crafted content analysis mechanisms , recent methods have achieved superior performance in image memorability prediction [ 15 , 45 , 26 , 33 , 29 ] . Some works explore the role of extrinsic factors in image memorability , such as , eye gaze [ 1 ] , testing context [ 8 , 37 ] , image organization [ 16 ] , emotion [ 6 ] and various external sources [ 26 ] . It is shown that these extrinsic factors are conducive to the prediction of image memorability , although their predicative power is arguably lower than the intrinsic factors [ 7 , 32 , 41 ] . If we consider image viewing and recalling as a ‘ mini-event ’ , it is tempting to apply knowledge of image memorability to that of events , especially in view that event memory experiments often resort to image cues to trigger the memory . However , the memory of a static image is not equivalent to that of an event . The latter involves much more complicated cognitive processes and is affected by many internal ( event properties ) and external factors ( e.g. , event context and personal characteristics ) . Therefore , it is much more difficult to predict event memorability at the micro-level in the real-world context . Video memorability is another stream of closely related works [ 36 , 10 ] . Continuous video clips show events with temporal order , which may resemble episodic events . Nevertheless , there are fundamental differences between third-person-view video memory and the first-person-view ( FPV ) real-life event memory , namely , real-life events are of personal relevance , strongly contextualized in a myriad of life events . As such , video memorability still lacks the rich context of personal experience . Foremost , there is no dataset of event memorability that accounts for the multivariate factors in its prediction . While cued-recall has long been adopted in autobiographical memory studies ( e.g. , using lifelog photos ) , existing works mostly focus on the behavioural and neuropsychological processes [ 25 , 3 ] , rather than computational prediction of event memorability . To fill these gaps , this study will develop a novel dataset with rich information of event memory and propose a new baseline model for its prediction . 3 Dataset . 3.1 R3 Experiment . We design an experiment called R3 ( standing for record , retrieve , replay ) to investigate event memorability in the real-world context . Forty-seven elder subjects ( 37 female , age : mean = 62.7 years , std = 6.3 years ) participated the study over three months . During the first 4 to 5 weeks , they used a wearable camera ( Narrative Clip , temporal resolution set as 2 frames per minute ) for about 7 hours per day to collect lifelog data . Twice a week , they visited the experiment site to upload the collected lifelog data and underwent scheduled treatment . In a treatment session , they were shown lifelog photos selected from the past few days ( i.e. , since last visit ) , and were guided to make an effortful recall of the related events , in the same vein as the episodic specificity induction method [ 34 ] . Half of the site visits involved treatment and the other half did not , leading to a balanced experiment control for studying the treatment effect . Interested readers may refer to supplementary material and [ 52 ] for detailed information of the experiment protocol . Subjects were tested for their autobiographical memory twice in the first month , each with two weeks ’ data collection and four visits to the experiment site ( including two treatment and two non-treatment sessions ) . In each memory test , a set of 144 distinctive photo cues were presented one at a time for the subject to recall and report the respective memory strength . Among the 144 cues , 1/3 were sampled from the treatment period , 1/3 from the non-treatment period , and 1/3 were lure photos ( i.e. , not from the subjects ’ own life ) . The lure photos served a similar purpose as the “ filler ” as in a standard image memorability test [ 23 ] . Subject were aware of the existence of lure images , through pre-test briefing and warm-up trials . For event memory testing , a photo cue was shown for 6 seconds , after which a subject responded by rating the respective event memory according to the “ Remember-Know ” paradigm [ 40 ] , which is widely adopted in psychology studies . In particular , subjects rated their memory of the cued-event in two questions : ( 1 ) type of memory , which could be one of remember ( recollection with certain episodic detail ) , know ( familiar but no episodic detail ) , or new ( not from ‘ my ’ life ) ; ( 2 ) memory level , which refers to the degree of the respective memory type , including recollected event detail 1–4 ( for remember ) , familiarity level 1–4 ( for know ) , and confidence 1-2 ( for new ) [ 52 ] . This resulted in a graded memory strength ranging from 9 ( recollection with a lot of episodic details ) , to 5 ( very familiar but without episodic details ) [ 46 ] , until 0 ( no memory at all , i.e. , the cued-event was considered absolutely not from ‘ my ’ own life ) . Readers may refer to [ 39 ] for a similar mnemonic scheme . After the second test , subjects did not record any more lifelog data . However , a third test was arranged two months later using photo cues drawn from the entire lifelog repository , but not used in the treatment , nor previous two tests . This was intended to evaluate the long-term memory outcome . A total of 288 photos ( from subjects ’ own life ) were rated with a memory strength from three tests , among which 2/3 ( from test sessions 1 and 2 ) had a short encoding-testing interval ( 2.5 days on average ) , and 1/3 ( from test 3 ) had a longer encoding-testing interval ( 10 weeks on average ) . The main outcome of the R3 experiment are ( 1 ) a lifelog repository from 47 subjects , each with an average of 25 , 000 photos captured in about 30 days , and ( 2 ) about 16 , 000 photo cues ( excluding lure photos ) with memory scores that indicate the strength of the event memory . As a mechanism of quality control , data samples were excluded if they met the exclusion criteria ( refer to A.1.4 ) . In total , 10 , 654 photos cues from 40 subjects were included in the dataset . Each photo has a memory score of 0 ⇠ 9 , serving as the ground truth . The photo cues were further processed to extract ( 1 ) visual semantic features and image memorability , ( 2 ) contextual information related to event encoding , and ( 3 ) contextual information related to memory test ( Figure 1 ) , which will be elaborated next .

Summary: This paper makes 3 contributions. First, the paper proposes and builds an event memorability dataset, wherein life-logging and subsequent event recollection data are collected from 47 elder subjects over a period of 3 months (Section 3.1). Next, the authors investigate several potential predictors of event memorability, including intrinsic image cues and extrinsic factors (Section 3.2-4.1). Lastly, the authors consider baseline models, in the form of a contextual event memory network to predict event memorability (Section 4.2 and 5).