Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The presented work focuses on evaluation of the performance of machines and humans in perceptual and cognitive components of dynamic visual ability and understanding the gap between humans and machines capabilities. In addition, an approach for machine and human collaboration is proposed and tested. 
The experiments were performed on a wide set of models and input data showing that perceptual ability of both humans and machines are closer than the cognitive ability, in which humans outperform machines. It has been also proved that human machine cooperation is possible and can lead to improved performance. 
strength_and_weaknesses: The claims of the paper are supported with experiments performed for multiple models. The topic is introduced well and compared with relevant studies in the same area. The DVA capabilities are examined from static and dynamic perspective. The work flows logically.

Overall the work is interesting, the only concern is comparison of centers of objects. It is not clear how you ensure humans correctly mark centers of objects. It might be difficult for humans to visually find the object center. Have you encountered any issues there? If yes, then comparison of machine's performance might not be fair. Another minor issue is complexity of figures. It's difficult to quickly understand them, and careful analysis is needed. It would be nice to simplify them if possible. It might be also beneficial to compare the proposed human-machine cooperation mechanism with other existing human in the loop approaches. 
clarity,_quality,_novelty_and_reproducibility: The work is well structured, clear, and supported by extensive experiments. The code will be made publicly available, so it will be possible to reproduce results. 
summary_of_the_review: The work is interesting and claims are supported by extensive experiments. The addressed problem and approach seems novel, even though it's using existing models/metrics.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: With the observation that no other work compared humans and machines in dynamic vision ability, the authors designed an overall evaluation system followed by previous work. Specifically, they evaluated humans and machines on the single object tracking task and compared their performance in multiple aspects. The analysis found that humans have stronger DVA, but the gap is not big, especially in the SOTA models. Finally, they show when human-machine cooperate, the performance can be even better than humans.
strength_and_weaknesses: **Strengths:**

* The paper is well written, and the reviewer enjoyed reading it. 

* The authors provide details about the experimental settings so that the readers can understand the experiments well. 

* The authors conducted deep analysis for comparing human and machine DVA. 

**Weaknesses:**

* The paper lacks contributions. The authors collect additional human subjects' responses for the SOT task and run 20 models to get machine responses, which is followed by deep analysis to compare the human and machine DVA. However, this reviewer is unclear what are the contributions (especially technical contributions) of this paper. 

* Section 2.2 does not contain the related work on the computer vision tasks, but the section rather looks like it introduces the term. Indeed, the tasks were already presented in Section 1.

* In Figure 11: "the blue sequences" looked like they indicated the letters with a blue background, i.e. all the sequences. It is recommended either change the text background color or make it transparent. Further, it would be better to call the sequences A, B, C, ... instead of B2, D2, G4, ...  for better readability unless B2, D2, G4, ... has meanings. 

* Typos: 

"Aritciture" in Table 1
clarity,_quality,_novelty_and_reproducibility: **Clarity:**

* Figure 1(b). What is the table for? Where is the table referred to?

* In human-machine cooperation (Section 4.2), when machines fail to track, did the authors ask humans to provide the current target position? In other words, did the human subjects wait until the machine failed and then pick the position whenever the machine failed? Also, how many subjects participated in the cooperation experiments?

**Novelty:**

As written in the above section, it is not clear about the contributions of this paper. 

**Reproducibility:**

The authors said in the introduction that they would open-source the toolbox, code, and metrics. 


summary_of_the_review: The paper is well written with deep analysis to compare humans and machines in dynamic vision ability. However, the paper lacks novel contributions on the topics of learning representation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors proposed an overall evaluation system to investigate the gap between humans and machines in dynamic vision tasks. In particular, they chose single object tracking as the representative task, and design an overall evaluation system from three aspects (Experimental environment construction, Experimental executor selection, and Evaluation method design). The results provide insights for fellow researchers in the related fields. 
strength_and_weaknesses: Strength: 

The paper works on an interesting problem. The presentation of this paper is clear. 

The authors have provided detailed supplementary materials to support the main paper. 


Weakness:

The authors used SOT as a single representative task for the evaluation system, but the concept of dynamic vision ability is much broader and includes numerous other tasks. Perhaps the authors can tune down a bit their claims throughout the paper. 

Correspondingly, it is unclear to me how the results could contribute to the real applications, since the evaluation is only on SOT. 

As we all know, human subjects have their biases due to their diverse cognitive background. I wonder if the authors should recruit more diversified human subjects, instead of 15 subjects while all of them are computer vision researchers aged 20-30.  Furthermore, how the machine models are trained also matters a lot. The authors can discuss the above potential issues in order to give a more convincing evaluation. 

clarity,_quality,_novelty_and_reproducibility: The paper’s presentation is clear. It is of moderately good quality and originality. 
summary_of_the_review: The paper is interesting and has its merits. However, as mentioned above in the weakness part, I feel the title is a bit overclaimed and the actual contribution is not very clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper received 2 rejects and 1 borderline accept and while during discussion one reviewer remained somewhat positive about the work the paper lacks support. The paper describes a new set of synthetic visual object tracking tasks together with human benchmarks. 

In general, the 2 negative reviewers challenged the contributions of this work and the AC agrees with them.  It makes sense to build computational models of biological vision and to compare their performance against human observers but this is not what this work is about. The models are just AI models and they are not biologically plausible in any way. 

Conversely, since the goal for these systems is to achieve the best possible accuracy and humans are far from perfect on these tasks why should AI researchers care about these human baselines? A clear demonstration of the benefits of this artificial challenge for real-world tracking in videos is needed.
justification_for_why_not_higher_score: Overall, the proposed benchmark and associated human baselines failed to convince the reviewers and the AC thus recommends the paper to be rejected.
Decision: Reject