Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: Humans can rapidly learn from current experience as well as utilize information accumulated over longer periods. This paper tried to suggest 2 benchmarks describing each case and analyze the trade-off between them by evaluating several unsupervised learning algorithms on the benchmarks. Main contribution of this work is to find out that the negative sampling is key to building human-like learning models.

After the revision of the paper, the reviewer raised the score to 5.
strengths:  - Explained the trade-off relationship between real-time and life-long learning. In continual learning or lifelong learning we talk about plasticity-stability dilemma in the scope of a model, but this paper re-interpreted in a more macroscopic perspective by separating it into two benchmarks
 - Showed that the reason current unsupervised models fail on both benchmarks is that they don't utilize contrasting examples
weaknesses: * The way each benchmark was built is not new. Real-time benchmark used the same method presented in the previous study, but applied online streaming of data instead of offline learning. Life-long learning benchmark is meaningful in that it used children's egocentric video dataset which is differentiated from widely used benchmarks in life-long learning task, but the strategy(choose one child's data, group it into 100 segments, and evaluate every 10 segments) to use it cannot be seen as novel or unique approach.

* Showed that contrastive learning(negative sampling) is important, but few insight or analysis on why contrastive learning works so well when modeling human learning
correctness: * line 51 : it seems a leap to say that 'unsupervised algorithms can describe the learning dynamics of human behaviors under all time-scales', from the claim that 'they leverage the unlabelled stimuli'. This sentence is at a very important place bridging unsupervised learning to human learning, which is the key hypothesis of this paper, so it should be explained more logically.

* line 162: The three conditions (Swapped, Non-Swapped, and Switch) model different learning environments and the amount of performance increase and decrease is different for each condition, so it would be meaningful to look at them separately, not averaging them. Is there a reason for averaging the results of all conditions?

* line 177: If we want to see whether we can model human learning in temporal fashion through a life-long benchmark, shouldn't we see how the 10 evaluations of model performance change variates from human performance change? Here the authors averaged the results of 10 evaluations even though they contained temporally different evaluation results.
clarity: * The authors passed too many details for the real-time benchmark to the prior work(Jia et al). It is hard to fully understand the setting without reading it. In order for this paper itself to be a good benchmark, the necessary content must be described in this paper. For example, in a real-time benchmark, authors computed the categorization performance measured using d’(line 150), but they didn’t explain the definition of d’ and how to calculate it. Accordingly the next part, “to compute the learning effects through subtracting the changes of d’ on the exposed objects by the changes of d’ on the non-exposed objects” cannot be fully understood. Besides that, it would have been better if the authors explained that the exposure phase is the phase simulating unsupervised visual experience, and the reason for setting 3 experiment conditions (Swapped, Non-Swapped, and Switch).

* This paper focuses on using unsupervised algorithms to model human learning. I think it can be tested only with the real-time benchmark, so it seems unclear why the authors suggested the life-long learning benchmark. I agree with the trade-off between life-long and real-time benchmark, but real-time benchmark already is in online learning and operated with continual learning setting, so it seems life-long benchmark is intervening in the main content that the paper is trying to say. Is benchmarking life-long learning separately a prerequisite for modeling the human brain using an unsupervised algorithm?

* Imitating the way humans learn doesn't necessarily increase real-time and life-long learning performance of deep neural networks, then why is it important for DNN to achieve neural predictivity?

* What is the reason for choosing Sam among 3 children in SAYCam?

* line 161: Why using 4 test phases out of 5?

relation to prior work: Stated above.
additional feedback: * If SAYCam shoots only for 2 hours every 2 weeks, we might miss many important events that happened during the weeks. I wonder if the two-hour long video can properly describe or support evidence for a child's development of visual recognition ability during the couple of weeks. I assume the authors understand this better, so I would like to hear their opinions.
* Authors might have been working on this topic for a long time, but it would be better to give a more basic explanation that would help those who are entering into this topic or those with a moderate level of knowledge.
* The overall structure can be better organized especially for the methods part. It would be nice if the concepts are explained when they appear for the first time and formulated to clarify the meaning. 
* It would be better if the results(numbers) are organized in a table.
review_title: 2 benchmarks: Real-time and Life-long learning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper explores the gap between computer vision models and human vision and learning systems on real-time learning and life-long learning. Several self-supervised methods are evaluated and the results are presented. The reasons behind the difference of different methods are discussed. Following is a discussion on the tradeoff of real-time learning and life-long learning.
strengths: 1. The real-time learning and life-long learning tasks are complementary and most related to the gap between AI and human learning attributes.
2. The usage of children's ego video as learning curriculum is technically sound.
3. Several self-supervised learning methods across different styles are evaluated and good insights are provided
4. Discussion on the tradeoff is reasonable and inspiring.
weaknesses: 1. The section of methods of two benchmarks are kind of confusing and cannot be understood easily.
2. It can be more intuitive to present some failed cases of computer vision models and corresponding human results.
correctness:  Yes, it is constructed in a sound way.
clarity: Yes, it is well written.
relation to prior work: Yes.
additional feedback: Line 145 "each trail" may be "each trial".
review_title: Good contributions towards exploring the difference and gap between computer vision models and human vision and learning systems

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper draws a comparison between how humans and unsupervised learning algorithms learn. Two benchmarks were proposed: one is based on real-time learning, while the other is on life-long learning. Based on these metrics and other analyses, the paper arrived at the following findings: 
- newer generations of self-supervised methods show poorer performance based on the proposed metrics, while some older algorithms perform better. 
- best performing models have a commonality -- negative sampling
- a major mechanism underlying poor performance based on the proposed real-time is an algorithm's inability to capture the sparse learning signals in low-diversity environments.
strengths: - Rigorous evaluation of models can be messy and non-straightforward. Additional perspectives to evaluate models are good contributions since our DNN models could have other characteristics that conventional evaluation benchmarks cannot adequately capture. The paper presented two benchmarks for evaluating models and for comparing them with how humans learn.
- In addition to the benchmarks proposal, the insights obtained from using these benchmarks were also presented.

weaknesses: - I am unsure if the paper felt a little challenging to read because of the multiple research areas covered in the paper, some grammatical/writing style choices, or due to the general structure. I cannot spot anything glaring to suggest to make the paper clearer. Hopefully, other reviewers can help in that direction.
correctness: The claims are backed by experimental results. Likewise, some of the findings are similar to those I arrived at on a paper currently under elsewhere. One of such is that models such as the MoCo V2/V3 show even more superior performance when subjected to a more rigorous evaluation protocol, which a simple evaluation protocol might not adequately capture.
clarity: The paper felt a bit difficult to read. I cannot exactly say why though. 
relation to prior work: I believe the work clearly mentioned the other related works in literature and also introduced some literature that they based their work on.
additional feedback: None
review_title: Benchmarks that lead to new insights can benefit the research community 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents two benchmarks for human-like real-time and lifelong learning. The real-time learning benchmarks focus on comparing the visual classification behavior of unsupervised learning algorithms with that of humans captured in different sessions. Lifelong learning compares the continuous learning behavior of various unsupervised learning methods by training them on first-person videos of young children using a "child-like" learning curriculum. Different unsupervised methods (BYOL, SimSiam, SwAV, MAE, SimCLR, MoCov2) are evaluated using this benchmark. It is shown that negative sampling is beneficial in these low diversity benchmarks. In addition, a tradeoff that unsupervised methods must make in order to perform well in both benchmarks is discussed.
strengths: **Originality**: Training recent unsupervised methods on approximations of a human curriculum and comparing them is interesting and valuable. The paper draws on evidence from real human learning experiments.

**Quality**: The authors compare a number of current unsupervised learning algorithms and give error bars for them. The claims about negative sampling are well supported. The authors appear to be careful and honest in evaluating the strengths and weaknesses of their work.

**Significance**: The results of negative sampling are "nice to know." The trade-off is interesting. The benchmark itself is important.


**Accessibility and accountability** The authors provide their implementation, which benefits accessibility.

**Relevance to the broader research community**: The benchmarks appear to be relevant to communities engaged in human-like learning. The authors' results are also interesting for the field of unsupervised and continuous learning.

weaknesses: **Originality**: A comparison with the literature on human-like learning curricula in deep learning seems too sparse, so it is not clear how exactly this work curricula differs from previous contributions in this area.

**Quality**: The claim that a human-like algorithm has to struggle to achieve the discussed trade-off between the two benchmarks (line 23-27) seems far-fetched. This tradeoff is discussed in terms of batch diversity, but this would require that human-like algorithms learn batch-style. This claim may be true for current unsupervised methods, but does it hold for true human-like learning?

**Relevance to the broader research community**: As explained in more detail in the "Clarity" section, the benchmark (especially the real-time learning benchmark) is not described clearly enough. In addition, the results are only presented in graphs and not in simple numbers (in a table), which makes it difficult to compare them with the given baselines for further work. At least the supplementary material should contain plain numbers. Also, the baseline weights could be shared with the research community. 

**Ethical and social implications**: Ethical and social implications are not discussed. The authors believe that there are no potential negative social implications. This statement seems too harsh.

correctness: To the best of the reviewer's knowledge, most but not all of the assertions in the paper are correct. As mentioned earlier, the trade-off assertion may not apply to a real human-like algorithm. However, the reviewer did not attempt to replicate the results of the experiment part. The evaluation methods seem to be correctly designed, but the lack of plain numbers is a problem and affects the comparison with the paper. 
clarity: The reviewer found the methods section of the paper difficult to read. A different outline could alleviate this problem. For example, Figure 1 could be divided into its subfigures. For each benchmark (Real-time Learning Benchmark, ...), one of these sub-figures could be used to describe the benchmarks, after which the experimental results should be presented. Also, the description of each benchmark should be more consistent with the figures, and the figures should be more consistent with the training process (e.g., a literal mouse click is required in the figure. E.g., the ms in the figure and text do not always match. Figure 1A, for example, is basically the same as in the paper from which it was taken).  In addition, the description of each benchmark could be supplemented with a more detailed algorithmic description (e.g., pseudocode) to facilitate understanding of the benchmarks.  
relation to prior work: The relationship to previous work in unsupervised learning, real-time and continuous visual learning in real organisms, unsupervised deep neural network models for the visual system, curricula, and lifelong learning for neural networks is clear. Comparison with the literature on human-like learning curricula in deep learning appears sparse.
additional feedback: Q1: “The gray image serve as a proxy for the visual inputs of human subjects during inter-trial intervals.” Why are the images grayscale? 

Q2: You state that “Models learn from the whole data stream including both test and exposure phases, each of which takes 10 minutes” Why does the exposure face exactly take 10 minutes? Does the number of images the model sees then depend on the model's prediction speed?


**Needs (spell) checking (reviewer is not a native speaker)**:
- Appendix: More MLPs &#8594; Larger MLPs
- Did you discuss whether the data you are using / curating contains personally identifiable information or offensive content? We do not think the data we are using contain this. &#8594; the datasets used in the real-time learning benchmark contain human faces

**Limitations**: The authors addressed the limitations of their work, which clarified questions that arose while reading the paper. 

**Reviewer Statement**: The reviewer is willing to increase the score if the concerns are addressed by the authors

**Based on the revision 1 and 2, the reviewer increased the score to 7 (see the Reviewer Response to the Revision comment and the Reviewer Response to Revision 2 comment for more details).**
review_title: Interesting paper that needs some clarifications and a better written methods section

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces two sets of benchmarks, one on the real-time learning towards assessment of rapid changes, one on the life-long side in an online long-term curriculum. Here, the inspiration and data foundation is taken from a child’s stream of visual experiences over multiple years. In this context, various self-supervised learning algorithms are investigated, contrasted and analyses with respect to their ability to perform adequately on both benchmarks and mimic the human learner. Most importantly, choices and trade-offs are investigated when the two benchmarks are regarded from a complementary perspective.  

Post rebuttal update: see response below, rating raised from 5->6 

strengths: The main strength of the paper is that it tackles an important challenge of human-like lifelong learning from a complementary perspective of short and long-term knowledge acquisition and retention.  It’s ambitious, but there a great promise and many prospects from the type of investigations conducted on the two suggested benchmarks. In addition:

* There introduction and literature review sections are well written and intuitive to read.
* The visualizations, albeit partially a bit too small, are useful and well done.  
* The empirical evaluation seems quite comprehensive, with many different types of self-supervised approaches considered, different architectures e.g. a visual transformer, and analysis of trade-offs such as the mask ratio, replay window sizes, and the “memory mix” ratio. 

weaknesses: In principle, I believe the paper to be a great asset for the community. However, it is also a bit tough to vote for acceptance of the paper in its present form, despite its many strengths. 

Primarily, this is due to the fact that sections 3 and 4 are very hard to follow. The content is extremely dense, crucial details on the exact experimental set-up, considered methods and evaluation are missing, particularly when inspired form prior works that not everyone will be aware of. 
Examples are lines 142, 145, 165, 172, 250, 304, caption of figure 2, the role of pre-training on ImageNet/faces etc. Overall, the content suffers from the style of these two sections, where an attempt has certainly been made to exhaustively mention all aspects, at the cost of them being comprehensive enough. In essence, a reader is required to know the nits and bits of prior works, the way the datasets have been acquired (to understand the construction of the benchmarks), all self-supervised methods and their differences etc. To some degree this is reasonable, but even after rereading the sections multiple times it was hard to follow the train of thought and perceive the overall story. When lookin at the figures I felt lost to search for what precisely the captions mean, e.g. “life-long benchmark performance measured by the trajectory averaged Mini-ImageNet performance”, how precisely window means W= 20m etc are defined mathematically. I acknowledge that it may be there somewhere, but even after looking through the supplementary it was tough to find the necessary details. 

In similar spirit, the supplementary material was unfortunately less helpful than it could have been. I was hoping to find more details on the above mentioned aspects, more thorough descriptions of the dataset composition, the descriptions of the investigated methods, and additional results.  Overall the material was rather short however, following the same bullet-point list style structure of the main body that only attributes a few lines per topic. The multiple figures that are simple listed at the end are left unreferenced in text, leaving a lot of the heavy lifting to the reader.  As easy as this seems to fix, it presently imposes a large barrier for me to truly assess the paper’s main contributions. 
correctness: I believe the benchmark construction and investigation to be correct, but simultaneously will emphasize that it is hard to truly make a definite statement here, given the lack of clarity stemming from the very compressed writing. 
clarity: As mentioned as the primary point in the weakness section, sections 3 and 4 suffer from a lack of clarity, primarily due to the compressed very dense nature of the writing and a structure that quickly “moves on”. 
relation to prior work: The literature review nicely relates to various works from different fields, providing a good overview of the streams of perspectives across communities. 
additional feedback: I don’t think this is a concern necessarily, perhaps it is indeed more of a question, so I will list it under additional feedback. 

While reading the paper, I was a bit surprised about the choice of mechanism and why it has fallen on a raw experience replay perspective. Surely it is the easiest to implement, but following the many arguments on machine learning models that learn like humans and trying to investigate whether they do etc. I am wondering whether experience replay is at all realistic. Already in the old mid-90s connectionist pseudo-rehearsal papers by Robins or French, the argument has been made that from a neuroscientific perspective replay of original data instances seems rather unlikely, which is why many modern biologically inspired lifelong learning perspectives go with a mix of regulating the stability-plasticity trade-off and employing pseudo-rehearsal mechanisms, see e.g. “biological underpinnings for lifelong learning machines”, nature machine intelligence 4, 2022.  For most of the experimental take-aways it likely does not matter here, but much of the benchmark framework also seems to be built around the concrete interleaving of stored data. I would be curious to have a more detailed statement on the made choice and trade-offs here.
review_title: Interesting ideas and benchmark, held back by a lack of clarity in writing and too dense structure 

Meta-review: metareview: The two presented benchmarks for human-like real-time and life-long learning by incorporating perspectives from short-term and long-term knowledge acquisition and retention are highly interesting and useful for further research in the area of (unsupervised) human-like learning. The work is very well written and presented.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept