Predicting Event Memorability from Contextual Visual Semantics

Episodic event memory is a key component of human cognition . Predicting event memorability , i.e. , to what extent an event is recalled , is a tough challenge in memory research and has profound implications for artificial intelligence . In this study , we investigate factors that affect event memorability according to a cued recall process . Specifically , we explore whether event memorability is contingent on the event context , as well as the intrinsic visual attributes of image cues . We design a novel experiment protocol and conduct a large-scale experiment with 47 elder subjects over 3 months . Subjects ’ memory of life events is tested in a cued recall process . Using advanced visual analytics methods , we build a first-ofits-kind event memorability dataset ( called R3 ) with rich information about event context and visual semantic features . Furthermore , we propose a contextual event memory network ( CEMNet ) that tackles multi-modal input to predict item-wise event memorability , which outperforms competitive benchmarks . The findings inform deeper understanding of episodic event memory , and open up a new avenue for prediction of human episodic memory . Source code is available at https : //github.com/ffzzy840304/Predicting-Event-Memorability . 1 Introduction . Episodic event memory is a key component of human cognition and intelligence . Maintaining high-level event memory is good for mental health and executive functioning [ 34 ] . Episodic memory can be enhanced by cognitive training , such as regular photo review that triggers the reactivation of certain memory traces [ 25 , 42 ] . Recent advancements in visual lifelogging , including affordable hardware and powerful visual analysis software , make it possible to implement lifelog-based memory intervention [ 19 , 31 , 43 ] . However , without clear understanding of what factors influence event memory ( e.g. , what events have higher value for training ? how to select “ good ” photo cues to boost training effect ? ) , it is difficult to design effective cognitive intervention programs . This study attempts to understand event memory , i.e. , to what extent an event can be recalled . We probe the recall process with visual cues , and explore factors that influence event memorability . We aim to extend the understanding of event memory , beyond simple image recognition [ 22 ] and memory of event categories [ 8 ] , to the prediction of item-wise event memorability using contextual visual semantics . A few major gaps exist in this field . First , there is scarcity of data for evaluating event memorability . A sensible analysis requires rich event knowledge , including spatial-temporal information , personal experience and profiles , as well as quantifiable memory performance on the respective events . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Remember Know New Encoding Context Testing Context Predictive Model CNN Real-life Experience Test Set ( Stimuli ) Memory Test Visual Saliency Human Factors R3 Dataset Semantic Extraction Information Theoretic Entropy Contextual Event Segmentation Memory Score Considering that life events are highly personalized and seldom repeatable , it is difficult , if not impossible , to collect such data in laboratories by replicating a controlled process . This makes the generation of large dataset for event memorability study a major challenge . Second , event memory involves a complicated cognitive process in the human brain , so that the mapping relationship between event information and the memory outcome is obscure and fuzzy . It is affected by a number of factors , such as personal experience , cognitive status , and new , potentially conflicting input . Therefore , the memory outcome often has immense variability , making it extremely difficult to be predicted reliably . This study fills these gaps in three aspects . First , we develop a novel experiment protocol to collect personal life event information along with the human subjects ’ memory outcome on selected events ( Figure 1 ) . We build a first-of-its-kind event memorability dataset consisting of 10 , 654 valid samples with rich information about event context and visual semantic features . Based on data collected in a large-scale user study with 47 elder subjects , each in three months , we apply a suite of visual analysis methods , such as contextual event segmentation ( CES ) and information-theoretic entropy analysis , which collectively extract visual semantic features and contextual information . Second , we provide evidence that event memorability is contingent on extrinsic , contextual information , as well as intrinsic visual attributes . Third , inspired by image memorability studies [ 45 ] , we develop a simple yet effective contextual event memory network to predict event memorability by combining multimodal visual semantics , including the image cues and multiple aspects of contextual information . The model predicts item-wise event memorability with reasonable accuracy and serves as a baseline for more sophisticated methods . The dataset and baseline predicative model establish the foundation for developing computational models to decode human episodic memory . 2 Related Work . Episodic event memory has been intensively studied in cognitive psychology , exploring the human behaviors , cognitive processes and neural mechanisms of memory [ 11 , 40 ] . Regarding factors that influence how much an event is remembered , most studies focus on the macro-level , collective characteristics of events that influence subsequent memory . For example , distinctive events are remembered better than routine ones [ 21 ] ; episodic details are gradually lost with time ; memory could be distorted by interfering cues that re-activate alternative neural processes [ 25 ] . Nevertheless , little is reported on the micro-level properties of events that contribute to item-wise event memorability . We believe this issue is important because it opens up new possibilities of computational intelligence in multiple domains such as image retrieval [ 30 , 50 ] , cognitive intervention [ 8 , 51 ] and diagnosis [ 2 ] . Recent works in neuropsychology resort to brain imaging to decode the memory of individual events [ 3 , 5 , 39 ] . However , the brain imaging data is prohibitively difficult to collect , requiring special equipment and controlled environments , which restricts its application in the daily life . Meanwhile , there is a sizeable body of research that investigates image memorability . It is generally accepted that memorability is an intrinsic property of images based on the recurring evidence that human subjects show considerable consistency in image recognition tests [ 4 , 23 , 22 ] . Accordingly , computational models have been developed to predict image memorability from the visual semantic information , which is partially made possible by high-quality image memorability datasets , such as , SUN-Mem [ 22 ] , LaMem [ 28 ] , FIGRIM [ 8 ] , Mem-Cat [ 17 ] , LNSIM [ 33 ] , etc . Leveraging on deep convolutional neural network ( CNN ) and well-crafted content analysis mechanisms , recent methods have achieved superior performance in image memorability prediction [ 15 , 45 , 26 , 33 , 29 ] . Some works explore the role of extrinsic factors in image memorability , such as , eye gaze [ 1 ] , testing context [ 8 , 37 ] , image organization [ 16 ] , emotion [ 6 ] and various external sources [ 26 ] . It is shown that these extrinsic factors are conducive to the prediction of image memorability , although their predicative power is arguably lower than the intrinsic factors [ 7 , 32 , 41 ] . If we consider image viewing and recalling as a ‘ mini-event ’ , it is tempting to apply knowledge of image memorability to that of events , especially in view that event memory experiments often resort to image cues to trigger the memory . However , the memory of a static image is not equivalent to that of an event . The latter involves much more complicated cognitive processes and is affected by many internal ( event properties ) and external factors ( e.g. , event context and personal characteristics ) . Therefore , it is much more difficult to predict event memorability at the micro-level in the real-world context . Video memorability is another stream of closely related works [ 36 , 10 ] . Continuous video clips show events with temporal order , which may resemble episodic events . Nevertheless , there are fundamental differences between third-person-view video memory and the first-person-view ( FPV ) real-life event memory , namely , real-life events are of personal relevance , strongly contextualized in a myriad of life events . As such , video memorability still lacks the rich context of personal experience . Foremost , there is no dataset of event memorability that accounts for the multivariate factors in its prediction . While cued-recall has long been adopted in autobiographical memory studies ( e.g. , using lifelog photos ) , existing works mostly focus on the behavioural and neuropsychological processes [ 25 , 3 ] , rather than computational prediction of event memorability . To fill these gaps , this study will develop a novel dataset with rich information of event memory and propose a new baseline model for its prediction . 3 Dataset . 3.1 R3 Experiment . We design an experiment called R3 ( standing for record , retrieve , replay ) to investigate event memorability in the real-world context . Forty-seven elder subjects ( 37 female , age : mean = 62.7 years , std = 6.3 years ) participated the study over three months . During the first 4 to 5 weeks , they used a wearable camera ( Narrative Clip , temporal resolution set as 2 frames per minute ) for about 7 hours per day to collect lifelog data . Twice a week , they visited the experiment site to upload the collected lifelog data and underwent scheduled treatment . In a treatment session , they were shown lifelog photos selected from the past few days ( i.e. , since last visit ) , and were guided to make an effortful recall of the related events , in the same vein as the episodic specificity induction method [ 34 ] . Half of the site visits involved treatment and the other half did not , leading to a balanced experiment control for studying the treatment effect . Interested readers may refer to supplementary material and [ 52 ] for detailed information of the experiment protocol . Subjects were tested for their autobiographical memory twice in the first month , each with two weeks ’ data collection and four visits to the experiment site ( including two treatment and two non-treatment sessions ) . In each memory test , a set of 144 distinctive photo cues were presented one at a time for the subject to recall and report the respective memory strength . Among the 144 cues , 1/3 were sampled from the treatment period , 1/3 from the non-treatment period , and 1/3 were lure photos ( i.e. , not from the subjects ’ own life ) . The lure photos served a similar purpose as the “ filler ” as in a standard image memorability test [ 23 ] . Subject were aware of the existence of lure images , through pre-test briefing and warm-up trials . For event memory testing , a photo cue was shown for 6 seconds , after which a subject responded by rating the respective event memory according to the “ Remember-Know ” paradigm [ 40 ] , which is widely adopted in psychology studies . In particular , subjects rated their memory of the cued-event in two questions : ( 1 ) type of memory , which could be one of remember ( recollection with certain episodic detail ) , know ( familiar but no episodic detail ) , or new ( not from ‘ my ’ life ) ; ( 2 ) memory level , which refers to the degree of the respective memory type , including recollected event detail 1–4 ( for remember ) , familiarity level 1–4 ( for know ) , and confidence 1-2 ( for new ) [ 52 ] . This resulted in a graded memory strength ranging from 9 ( recollection with a lot of episodic details ) , to 5 ( very familiar but without episodic details ) [ 46 ] , until 0 ( no memory at all , i.e. , the cued-event was considered absolutely not from ‘ my ’ own life ) . Readers may refer to [ 39 ] for a similar mnemonic scheme . After the second test , subjects did not record any more lifelog data . However , a third test was arranged two months later using photo cues drawn from the entire lifelog repository , but not used in the treatment , nor previous two tests . This was intended to evaluate the long-term memory outcome . A total of 288 photos ( from subjects ’ own life ) were rated with a memory strength from three tests , among which 2/3 ( from test sessions 1 and 2 ) had a short encoding-testing interval ( 2.5 days on average ) , and 1/3 ( from test 3 ) had a longer encoding-testing interval ( 10 weeks on average ) . The main outcome of the R3 experiment are ( 1 ) a lifelog repository from 47 subjects , each with an average of 25 , 000 photos captured in about 30 days , and ( 2 ) about 16 , 000 photo cues ( excluding lure photos ) with memory scores that indicate the strength of the event memory . As a mechanism of quality control , data samples were excluded if they met the exclusion criteria ( refer to A.1.4 ) . In total , 10 , 654 photos cues from 40 subjects were included in the dataset . Each photo has a memory score of 0 ⇠ 9 , serving as the ground truth . The photo cues were further processed to extract ( 1 ) visual semantic features and image memorability , ( 2 ) contextual information related to event encoding , and ( 3 ) contextual information related to memory test ( Figure 1 ) , which will be elaborated next .

Summary: In service of studying event memorability, the authors construct a corpus of 10.6K images collected by 47 elderly study participants. The collection process involves the participants wearing cameras that take photographs at the rate of 2 frames per minute, and then regularly checking-in with researchers who gather memorability ratings from the participants for each image. The authors explore a variety of intrinsic (i.e., having to do with the image itself) and extrinsic (i.e., having to do with how the image relates to other images in the participant's log) feature sets for predicting the ratings. They find that extrinsic factors are the best predictors, and that adding additional CNN image features adds only a bit of extra prediction capacity.