Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper explores episodic memory and answers questions related to the impact of context and the influence of intrinsic image properties. They gathered data through an experiment with 47 elder subjects over 3 months, which led them to build a a novel dataset for event memorability. This allows them to build CEMNet, a network that predicts event memorability at the item level.  


main_review: ### UPDATED SCORE FROM 6 TO 7 BASED ON AUTHOR REBUTTAL.

## General thoughts
The paper sheds new light on a field that is seldom explored, episodic memory and the possibility of predicting it. The data collected is valuable. In general, the experimental approach to collect the data and evaluate the results is solid. I have a few concerns related to the predictive approach and models: the losses used appear to be a bit inconsistent, the model is somewhat simplistic, and there is little comparison to other models. In general, the paper is interesting but could be improved on the modeling side.

## Pros
Solid paper tackling a problem that hasn't been thoroughly addressed before.
In-depth experiment with live participants. The data collected has significant value.
Solid description of experimental protocols and methods. There is care in the development of the experimental procedure.
Interesting predictive approach.
First paper that studies episodic memory in this way.

## Cons
l. 27: there is a mention to understanding event memory beyond simple image recognition where [20] and [8] are cited. These works focus on visual working memory and not episodic memory - I'm not sure if they are the right citations there. 

Missing citations on video memorability, arguably more connected to episodic memory than image memorability:

- Newman, A., Fosco, C., Casser, V., Lee, A., McNamara, B. and Oliva, A., 2020. Multimodal memorability: Modeling effects of semantics and decay on video memorability. ECCV 2020
- Cohendet, R., Demarty, C.H., Duong, N.Q. and Engilberge, M., 2019. VideoMem: constructing, analyzing, predicting short-term and long-term video memorability. ICCV 2019

There is one main concern: the dataset is built with 10k images from 40 participants, with the objective of creating a large database that can be used to train models. Episodic memory is, however, extremely user-dependent; this makes training a model with images coming from different users an ill-posed problem. How was this addressed?

l. 283: The ordinal nature of the memorability score is not well justified: why can't it be treated as a real number between 0-1 like in previous image/video memorability works? Why was "ordinal categorical cross-entropy" used instead of MSE? If an order-aware loss was the idea here, were there attempts to use a differentiable ranking loss like [1] or [2]?

l. 296: here, you compute performance with MAE, which speaks of a real-valued memory score: why is it referred to as ordinal above? There might be an inconsistency between training and evaluation if the order is used as supervision during training but MAE is used during evaluation.

The ablation studies are not super consistent: why are you seeing such differences between each model?

There are few comparisons to other works or models in the modeling sections. What happens if you train a MemNet or a SemanticNet on this data (specifically for the visual pipeline)?

## References:
[1] Blondel, M., Teboul, O., Berthet, Q. and Djolonga, J., 2020, November. Fast differentiable sorting and ranking. In International Conference on Machine Learning (pp. 950-959). PMLR.

[2]  Engilberge, M., Chevallier, L., Pérez, P. and Cord, M., 2019. Sodeep: a sorting deep net to learn ranking loss surrogates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10792-10801).

## Questions

- l.158: There are multiple other ways to measure image distinctiveness (e.g. KDE-based techniques). Were other methods attempted?

- l. 246 comparison with Image Memorability: were those models pretrained on image memorability datasets first? 

- l. 248 why is pearson correlation used and not spearman?

- l. 313: what is your learning rate schedule?

- Calculation of event boundary condition: can you give more context about why use 10 events per day? Why is it considered normal? how does that impact your results?
limitations_and_societal_impact: The limitations are correctly outlined in Section 6. One main limitation is related to the choice of population for the study (elder subjects), which makes the results less generalizable to broader populations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper makes 3 contributions. First, the paper proposes and builds an event memorability dataset, wherein life-logging and subsequent event recollection data are collected from 47 elder subjects over a period of 3 months (Section 3.1). Next, the authors investigate several potential predictors of event memorability, including intrinsic image cues and extrinsic factors (Section 3.2-4.1). Lastly, the authors consider baseline models, in the form of a contextual event memory network to predict event memorability (Section 4.2 and 5). 
main_review: EDIT: I am happy with the authors' response and I am raising my score. 

The presented paper is very promising -- the proposed idea of understanding and predicting event memorability is a novel topic and will be a welcome addition to the memory literature (for both computational and psychology researchers). The authors have also put a remarkable amount of effort into the paper, including collecting a large-scale longitudinal dataset, and to me, the entire effort is very commendable. Thus, my impressions of the paper are largely positive and I think this paper can be a nice addition to Neurips (or a similar high-impact venue).   

That being said, I have a number of reservations about the paper in its current form. To me, the paper needs significant polish and improvement in the structure of the paper in order for this work to meet the promise it has. Thus, my current rating is borderline (I would be open to improving my score based on the rebuttal). Below are my suggestions and concerns which I hope the authors will find helpful. 

(a) First, I would like to see more clarity and details about the procedure used by the authors to collect the dataset (Section 3.1). Some questions I would like to see addressed are below. 

(i) What was the rationale for collecting 47 subjects? Did the authors do any power analysis or any pilot experiments to determine how many subjects they are going to collect? 

(ii) In the memory test, subjects were presented with 144 images one after another. What was the rationale for deciding this number i.e. presenting 144 images? Did you take any steps to ensure subjects will not ``bored" during the task? To me, 144 seems like a large number of trials presented in one session; could you also provide information on how long the memory test takes for a subject on average? I think an alternative way to collect data would have been to recruit a larger subject pool (but having fewer images per subject) -- did the author consider this possibility? Basically, my questions here seek to understand the rationale behind the data collection procedure..

(iii) Btw, were the subjects informed that some images during the memory test will be "lure" images? I am asking this because not informing subjects about the experimental manipulation could have significant consequences on the data collection. 

(b) After reading Section 3.1, I was really craving to know a bit more about the dataset that the authors collected and I was rather surprised that the paper didn't include details about the same. For instance, how accurate were the subjects in remembering the events? Were there any interesting individual differences in the participants? Were there any participants that did particularly well or particularly poorly? (I know that the authors removed the data where subjects performed poorly but I think more analysis about subjects' performance on the dataset would provide valuable insights).

(c) Section 3.3 -- The authors here have done a commendable job in coming up with extrinsic features based on extant cognitive theories and then using these features (along with intrinsic cues) to better understand event memorability (Section 4.1). However, I think that this section can be significantly strengthened and its structure can be substantially improved as suggested below. 

(i) In Section 3.3.1, the authors considered two extant cognitive theories, event distinctiveness, and event segmentation theory but it was unclear to me why the authors considered these two theories and not any other theories -- are these the two most dominant theories in cognitive science literature? Are there any other possible theories that the authors could have considered (but perhaps didn't because of computational constraints)? Perhaps, this section could be improved if the authors provide a brief overview of existing theories of event memory and highlight why they decide to consider event distinctiveness and event segmentation. 

(ii) In Section 4.1, from the results presented in Table 1, the authors conclude that the most important features include image memorability, presence of faces, event distinctiveness at encoding, activity, treatment, and encode-test interval. I think here readers familiar with cognitive research will really appreciate a more detailed discussion of the implications of these findings -- do these findings provide insights to the event memory literature? From my understanding (please correct me if I am wrong), these results provide evidence to the event distinctiveness hypothesis but they do not support the event segmentation theory (since the correlation of the boundary condition is negative). Do the authors have any hypothesis/insights on the same or why the boundary condition did not perform as expected? My point here is that Section 3.3 and 4.1 can go beyond showing event memorability can be predicted by intrinsic and extrinsic cues. Rather, I think this section provides a great opportunity to provide valuable insights to cognitive research as the large-scale dataset and the computational methods developed here can be used to evaluate existing theories of event memory.

Minor points

(a) The caption of Table 1 says that t-value in bold font means the factor significantly correlated with memory (p > 0.05). Shouldn't it be p < 0.05? 

(b) Line 48: I would refrain from using strong words such as "convincing" evidence. I think simply stating that you provide evidence should suffice as the results speak for themselves. 
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: EDIT AFTER AUTHOR RESPONSE: I read the other reviews, and I did not see the need to make any changes to my review.

This paper develops a novel experimental paradigm and introduces a dataset obtained from a lifelogging experiment where subjects rate how well they remember events corresponding to photos taken by first-person view photos.  It answers the question of whether intrinsic image features or extrinsic contextual factors are more predictive of memorability, and it further explores which factors are the most important for predicting memorability.
main_review: Originality:  High.  It appears that the experimental paradigm is novel.  The features extracted for the analyses seem to have been developed in prior work, and the network used is a minor variation of existing architectures.  But the analysis as a whole is fairly sophisticated and constitutes a non-trivial application of existing approaches.

Quality:  High.  The experiment appears to have conducted in a careful manner, and the analyses appear to be correct.

Clarity:  The paper is clearly written.

Significance:  Moderate.  Predicting event memorability may be relevant to many disciplines interested in user experiences, and this paper contributes very interesting findings which help us gain a better understanding of what factors make an event memorable.  Furthermore, the dataset could be useful in other studies.  The finding that photo reviews helps with memory retention is potentially clinically significant, but this particular finding is not the focus of this work.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In service of studying event memorability, the authors construct a
corpus of 10.6K images collected by 47 elderly study participants. The
collection process involves the participants wearing cameras that take
photographs at the rate of 2 frames per minute, and then regularly
checking-in with researchers who gather memorability ratings from the
participants for each image. The authors explore a variety of
intrinsic (i.e., having to do with the image itself) and extrinsic
(i.e., having to do with how the image relates to other images in the
participant's log) feature sets for predicting the ratings. They find
that extrinsic factors are the best predictors, and that adding
additional CNN image features adds only a bit of extra prediction
capacity.
main_review: 
The main contribution of this work, in my view, is the corpus itself:
the authors do an excellent job of making the case for the collection
of this type of data, and clearly are involved with/invested in the
outcome of memorability research for the benefit of elderly
folks. This dataset could become an excellent resource for the ML
community, both for use-cases having to do with memorability
prediction and beyond.

Technically, the authors do a good job of of exploring the corpus, and
undertake an admirable feature engineering effort to tease out what
makes a particular frame memorable versus not: extrinsic features
explored include auto-segmentation timing, relation of a frame to all
other frames, etc. Intrinsic features explored include: previous
memorability models trained on different corpora, hand annotations for
presence of faces, etc. Overall, I have no major technical concerns:
while this paper is less focused on modeling contributions (I suspect
more modern neural architectures could push performance at least a
bit), the dataset/data exploration contributions are the centerpoint
(along with reporting reasonable baselines).

The authors are straightforward in their discussion of shortcomings;
the ablation study in Figure 3 does indeed suggest that the patterns
different models use to achieve accurate predictions are not
straightforward. While this does cast some doubt on whether or not the
correlations discussed are meaningful in a more causal sense, the
authors mention this directly. The authors also correctly point out
that the gender and age distribution of their corpus may not be
representitive for many use cases.

My concerns include:

- While the authors do a per-feature analysis to demonstrate which are
  most predictive of memorability, I would have appreciated a more
  direct connection with the data. For example, they hand-coded for
  various activity types in each of the 10K frames and found a
  significant correlation, but which events types were memorable, and
  which were not? Perhaps table 1 could be augmented with this type of
  direct connection to the dataset.

- It should perhaps be noted more explicitly/earlier that the authors
  purposefully make no attempt at personalization: their
  cross-validation splits hold out entire participants, rather than
  individual images. While this is a fine decision, I would have liked
  a note earlier on the rationale for measuring globally memorable
  events (i.e., events that are likely to be memorable to people, on
  average) vs. personally memorable events. Potentially promising
  avenues of future work would be: 1) exploring deviations in the
  predictions of an "impersonal" model vs. a personal one; and 2) how
  to possibly train personalized models.

- Given the strong performance of an MLP on extrinsic features
  reported in Table 2, I would have been curious to see the
  performance of a linear model, whose coefficients may provide more
  direct hints at the correlations between input features and the
  label. Another more interpretable alternative might be the
  Generalized Additive Models; a nice library for them is:
  https://github.com/interpretml/interpret

  @article{nori2019interpretml,
    title={InterpretML: A Unified Framework for Machine Learning Interpretability},
    author={Nori, Harsha and Jenkins, Samuel and Koch, Paul and Caruana, Rich},
    journal={arXiv preprint arXiv:1909.09223},
    year={2019}
  }
limitations_and_societal_impact: See above for some discussion --- I thought the authors did a good job, overall. All my checkboxes are checked: IRB approval for human subjects research, qualification of model limitations, and discussion of potential harms in applying models trained on the dataset.
review_title: null

Meta-review: The paper studies episodic memory in humans and presents a novel event memorability dataset, along with a memory network (CEMNet) to enable computational studies. All the reviewers are positive about the paper and the authors’ responses addressed most of their major concerns. I strongly encourage the authors to make the suggested changes  by reviewers — better description and analysis of the data collected and the process used (ZmrY, 9z5U), writing clarifications to sections 3 and 4 (9z5U, ZmrY), adding ablation studies (wyti, ZmrY) and updating the references (wyti, ZmrY). 
Decision: Accept (Poster)