Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper evaluates three adaptive dynamic neural networks for their correlation with the speed-accuracy tradeoff of human object recognition in 2D images.  For this, it first collects a benchmark dataset elicited from crowdworkers under various transformation conditions using the CIFAR-10 dataset.  Then it compares the ability of the rCNN (ConvRNN), MSDNet, and SCAN models to capture the speed-accuracy trade-off exhibited in the crowdsourced perceptual data and shows the MSDNet model's predictions exhibit the most human-like speed-accuracy tradeoff. 

strengths: There are several strengths of this paper. 

S1. It is motivated by an important problem.  Using neural networks as computational models of perception or cognition is useful with potential benefits to our understanding and practice of neural networks in engineering and sciences. 

S2. The paper provides a relatively thorough evaluation of the models considered for the human-like speed-accuracy performance trade-off.

S3. It is written well, overall, and easy to follow. 

weaknesses: There are also a few weaknesses of the paper. 

W1.  The contributions to earlier work, particularly over [6], are not substantial. Also, it is unclear how statistically significant the differences in correlations across models found in the paper are.

W2.   Given W1, one would expect the paper to focus on data collection and analysis. However, the collected data is small scale and the paper focuses on evaluating different dynamic neural networks, instead of data curation and analysis. For example, one hour is at the long end of crowdsourced elicitation sessions and presents potential issues on the benchmark data quality. 

W3. The baseline model ConvRNN was originally trained on the ImageNet dataset and,  given that images are upsampled here, it is unclear why the  ImageNet dataset was not used. CIFAR-10 might be too easy for the models considered to show a performance range.

correctness: The paper misses an analysis of data quality, which can be an issue given that each MTurk session took about an hour to finish. At a minimum, the variance among participants across task conditions should be reported.  
Another issue is that the paper doesn't also report the statistical significance of correlations reported as well as the statistical significance of the differences between the models compared using these correlations.  It is unclear, for example, whether the difference between the ConvCNN and MSDNet models in predicting human behavior is statistically significant or not. 

clarity: The paper is written well and straightforward to follow. 

Minor. 
There seems to be a mismatch between Table 1 (20+7+8=35) and Sec 3.2 (33) on the number of MTurk participants.
relation to prior work: The paper can give an improved discussion on how the current work differs from [6], which introduces the baseline ConvRNN model and most of the methodology used here, and, in this context, why the ImageNet dataset was not used for the current study. 
additional feedback: I find the paper adding to our understanding of how neural networks can be used as computational models to study the accuracy-speed trade-off in human object detection. However, its contributions, both in modeling and data collection, are limited.  It'd be a better fit to be published at a cogsci venue as an update to [6]. 

**Post Rebuttal**  I appreciate the authors’ detailed response and the revision. I also enjoyed the style of their response. Nonetheless, given its contributions, I still think this work will be better assessed and appreciated at a cogsci venue.
review_title: More suitable for a cogsci venue

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This study focuses on studying human recognition performance of CIFAR-10 images, including modified versions of those images, as a function of the stimulus presentation time. The study further compares human performance against recognition by different types of neural networks. I am quite perplexed by the results, which do not match my intuitions and expectations based on previous studies. Extensive studies show that humans can recognize objects quite well in about 100-200 ms presentation time (see references below) but here the authors report chance performance up to 400 ms and even rather poor performance at 1000 ms presentation times. Given that the authors are top-notch scholars in the field, there must be something quite basic and deep that I am misunderstanding about the paper but I am afraid that I fail to see what it is. 
strengths: -- It is very useful and important for the field to compare neural networks with human performance
-- Especially relevant toward understanding visual computations is the processing time of visual information, which the authors aim to study here

weaknesses: -- It is hard to compare studies with different datasets, different chance levels, etc. Therefore, the following comparisons should be interpreted with caution. Here are a couple of examples: 
Majaj et al J. Neuroscience 2015 (first author is one of the authors in the current paper). The authors report d’ and 8-way categorization instead of classification accuracy with 10 classes (and completely different images), but the authors write: “The image presentation time was chosen based on results showing that core object recognition performance improves quickly over time such that accuracy for a 100 ms presentation time is within 92% of performance at 2 s (see Fig. S2 in Cadieu et al., 2014). Results were very similar, with slightly shorter (50 ms) or longer (200 ms) viewing duration.” Similar observations from the same lab in monkeys are reported in Rajalingham et al J Neuroscience 2018. These conclusions are completely different from the ones in the current study. 
Tang et al PNAS 2018. This is 5-way categorization. Stimuli presented for 25-150 ms. Subjects are almost at 100%, even with 25 ms presentation time. Even with heavy occlusion, the subjects do much better than in the current study.
Kirchner et al Vision Research 2006. Binary classification of animal or not. The subject with the worst accuracy had 77.1% (best accuracy was 98.7%) with 120 ms presentation time. 
Geirhos et al arXiv 2018 (see also recent Geirhos et al arXiv 2021). 16 classes, 200 ms presentation time. Accuracy for both color and grayscale images was about 90%. Even after addition of large amounts of noise (similar types of noise but much more noise than in the current study), performance was always much better than in the current study. 
There are many more studies, but I include just a few above to keep the discussion concrete. My understanding from Figure 1 is that chance is 10%. Thus, subjects seem to be at chance up to approximately 400 ms. Some subjects are even at chance at 1000 ms!!! This seems to be completely incompatible with the extensive literature showing basically ceiling performance with 100-200 ms stimulus presentation time.
-- The authors are experts in the field and the last authors is one of the key creators of the famous psychophysics toolbox, arguably one of the best ways to control timing in stimulus presentation. We are all indebted to his contributions. Thus, I am quite convinced that there is something about the current study that I am failing to grasp. But I cannot figure out what it is. 
-- Perhaps the CIFAR-10 images are heavily pixelated and therefore it is extremely hard to recognize them and this is even worse than the noise levels in Geirhos et al 2018, 2021 or the occlusions in Tang et al 2018? 
The authors do not report the size in visual angle of the stimuli (perhaps because it is very hard to control in online experiments). Perhaps subjects were doing the tests on their phone and the stimuli were incredibly small? 
How do the authors control stimulus timing in online experiments? Perhaps the x-axis does not accurately reflect the actual stimulus presentation time? But it would have to be off by almost an order of magnitude to make the results consistent with the literature. 
It would be nice to show actual psychophysics results in the lab to compare with the online measurements. Perhaps the online subjects are watching Netflix, and Snapchat while performing the tasks? 

correctness: As alluded to above, the results make no sense to me. I apologize to the authors for not being able to figure out what is going on. I hope that their rebuttal will clarify
clarity: The paper is well written.
relation to prior work: My main concern is the relationship between the results and previous work as outlined above. I fail to understand how the current results can be consistent with an extensive body of literature documenting human (and monkey) visual recognition behavior. 
additional feedback: I gave this study a very low ranking but I am willing to radically change my ranking if the authors can explain why their results are so different from everything else published in the field.
review_title: Intriguing observations about human speed-accuracy tradeoff 

final_score: rating: 3: Clear rejection
review_content: summary and contributions: This paper proposes a object classification benchmark dataset for modeling the speed-accuracy trade-off in human cognition. The authors use the benchmark to compare human cognition to neural network behavior, providing both a model for the human speed-accuracy trade-off and recommendations for computer vision research.

By way of framing this review: I am familiar with computer vision research in general, but I have very little knowledge of human cognition research of this kind.
strengths: 1. **Robust, well-documented data collection.** The data collection procedure appears to be very carefully designed based on prior work and has no obvious flaws in internal validity. (I am not sure about the standards for external validity in human cognition studies re: sampling bias - the sample seems to be primarily male.)
2. **Clever, interdisciplinary approach.** Controlling the recognition task difficulty with image perturbations was a good idea! Prima facie this seems to be a good solution for mimicking the human trade-off, and I like the idea to use neural networks to model this problem.
weaknesses: 1. **[Major] Potentially incomplete analysis.** I am concerned that Pearson's $r$ might not be the right choice for measuring correlations, especially since the results are not reported with a clearly explained standard error and the interpretation is not clear. Do you obtain any confidence bounds? What distributional assumptions are made on the data? For example, Figure 5 shows boxplots of the correlation coefficients, but doesn't provide any standard errors as far as I can tell. I would encourage the authors to clarify the statistical significance of their results, noting that Pearson's r is not robust and could be misleading if there are outliers.

EDIT (07/16/2021): The authors added significance tests and std bounds (but still no confidence intervals) - but the paper still doesn't mention distributional assumptions and significance results are not reported with each experiment (as far as I can tell the tests were performed only for Figure 8).

2. **[Major] Motivation and relevance.** The authors state several times that the speed accuracy trade-off is "an essential property of human object recognition." Without a background in this area, it's difficult to tell exactly why it's important, except for a vague alluson to dyslexia and other deficits. 

    If the main contribution is a benchmark to open up research in this area, it's important to clearly explain why this research and these results matter to biology/physiognomy/psychology. I only see one minor contribution (L281), which claims that MDSNet better models the speed accuracy trade-off than a previous attempt, but the limitations in the statistical analysis undermine this claim.

    It's especially important to demonstrate a biological contribution since there isn't a clear contribution to machine learning. For example, in L290-291, the authors state that future work in object classification should try to match the flexibility of human object recognition based on the greater performance range of human participants. But it's not clear why performance range is desirable - if machines are performing consistently well, isn't that a desirable outcome? Is there some evidence that performance range corresponds to generalizability or robustness? There is already lots of research on robustness to data augmentation in computer vision.

EDIT (07/16/2021): Conversations with the authors have helped clarify this point. I am still concerned that the authors are split between two contributions: explaining human cognition with computational models (probably more appropriate for another track/venue), and benchmarking NNs for graceful failure under computational constraints. The original submission has very little focus on the ML contribution. The authors make references to proposed changes emphasizing the latter contribution in the Intro, Related Work, and Results, but I don't see those in the manuscript yet - only a paragraph in the discussion without the new references mentioned in our discussions.
correctness: The methods and models are thoroughly described and appear to be correctly chosen and parameterized. Without further knowledge of similar cognition studies it is difficult to find flaws in the experimental design other than the ones detailed in the **Weaknesses** section.
clarity: The paper is mostly clear, but in the introduction it is difficult to follow and relies heavily on jargon. For example, in lines 42-44, the authors make reference to early exits and adaptive composition. Why are these important for explaining human speed-accuracy trade-offs? What do these terms mean? (The explanation doesn't come until later.)

The description of methods is especially clear - well done.

The figure captions are fairly sparse - for example, Figure 6 shows a blue bounding area but doesn't describe what it represents (a standard error?).

EDIT (07/16/2021): These concerns have been resolved by additional clarification and more detailed captions.
relation to prior work: The authors describe several relevant studies and position their work as an application of the field of dynamic automated object recogition to the problem of modeling human object recognition.
additional feedback: I think this is an interesting avenue of research, but I'm having a difficult time understanding why the results are relevant to machine learning or the study of human cognition. Where can we go from here?

EDIT (07/16/2021): The authors have clarified both points. I am unqualified to judge the contribution to CogSci but the authors' comments are compelling. I am less convinced about the contribution to ML research, given that the manuscripts I read focus mostly (in their motivations, methods, and results) on modeling/explaining human behavior, not benchmarking ML models.
review_title: Neat idea, weak/unclear contribution

Meta-review: The paper presents a very interesting problem, a moderate size dataset, and an associated comprehensive evaluation. Reviewers share concerns regarding novelty, and in particular regarding the size/impact of the dataset. It is appreciated the effort authors did by providing all justifications and paper updates as requested by the reviewers. However, still there are concerns regarding the size/impact of the dataset..
Decision: Reject