Kati8/ICLR_2023_2025 huggingface dataset:
authors: Aidas Aglinskas
authors: Stefano Anzellotti
reviews: 
Comments: The authors investigate how neural networks label action-recognition video frames and compare the neural network against human behavioral performance. They manipulate the stimuli to separate bodies and background and show that both neural networks and humans perform very well with full stimuli, body only or background only. Humans perform better with the body only compared to background only conditions. The authors propose an architecture that is loosely based on notions of modularity in the brain and this new architecture improves performance and matching to human behavioral data.
Strengths: Building networks to recognize actions is of high importance to practical applications 

Comparing how well networks perform to human performance is also of interest in terms of aligning machine and human visual capabilities.
Weaknesses: The bottom line is that highly uncontrolled and bad datasets lead to spurious and uninterpretable results. This is the main challenge throughout. 

In Fig. 1 bottom left, the authors claim that baseline models perform similarly well when tested with ORIG, body or BG. This is NOT what the results show. The results do not have error bars, let alone any minimally rigorous statistical analysis. From eyeballing the figure, it seems that ORIG>>BG>>Body. 

The fact that humans can identify actions purely from the background frames with over 0.7 accuracy shows that
(1) The dataset is way too easy
(2) Background is a major confounding factor 
(3) Time is not needed in such an easy task 

As far as I understand, the proposed architecture is trained very differently from the baseline architectures. The proposed architecture is trained with Body-only stimuli and does better on Body-only stimuli, and it is trained on BG-only stimuli and does not perform better on BG-only stimuli (again, no error bars, no statistics, this is all from eyeballing). Training yields better performance in general. 

It would be great to present actual results on how well Yolo v8 separates body and background.

The manuscript only has one main figure and minimal additional information that does not satisfy basic standards in the field. There are no error bars, there are no comparisons across multiple different models, no comparisons with different datasets, no ablations, no description of the effects of key variables like size, etc.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This work begins by examining the similarities and differences between humans and deep neural networks in terms of action recognition. It demonstrates that a deep neural network trained with cross entropy on the entire video cannot perform action recognition when background information is omitted from the training data. In contrast to this, human subjects are capable of identifying activities solely from the body information. This highlights that DNN trained for action recognition incorrectly balances the body and background information present in the video data. In order for the deep neural network to exclusively distinguish actions coming from the body, the authors suggested using two different backbones, one for the body and one for the background. In addition to this, they implemented a loss function that was more complex and yet nevertheless compatible with their category-selective design. As a consequence of this, they demonstrated that a body-background separated backbone may produce an action recognition pattern that is comparable to the pattern seen in human participants, albeit with a significantly lower level of accuracy.
Strengths: The authors try to optimize deep neural works towards reproducing action recognition patterns observed in human subjects.
Weaknesses: As the authors already included in the related works, having separate streams for different information in video is not new. For example, an early work on dynamic texture processing used two separate backbones for “appearance” (the scene) and “dynamic” (the optic flow). Their loss function also combines the matching of both appearance and dynamic features. It is possible that the L_{combined} here is new. However, the authors do not include any details on how they define L_{body}, L_{background} or L_{combined}. If the only difference this work has with other work is its usage of L_{combined} (I guess this is the cross entropy between predicted frames vs. the true frames), the novelty is very limited. 



Tesfaldet et al 2017 Two-Stream Convolutional Networks for Dynamic Texture Synthesis



This work needs a much more developed result section to fit as an ICLR manuscript. Its current format has one result. This result is not surprising given previous literature. I would encourage the authors to include a more detailed investigation of the proper loss functions, what predictive features are being used in humans to perform action recognition, etc. These extensions may strengthen this paper.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 1 poor
reviews: 
Comments: This paper takes insights from neuroscience and builds a model that processes body and background in images separately, aiming to improve the performance of action recognition.
Strengths: This paper propose novel ideas of incorporating inductive bias from neuroscience in building artificial neural networks and shows that it does improve the performance of action recognition when compared with a baseline network. The paper is well-written and very clear. The human dataset collected in this paper is also valuable and should be perhaps incorporated into action recognition benchmarks.
Weaknesses: This paper is a rudimentary effort in showing incorporating certain inductive bias from neuroscience could potentially help with artificial networks in certain tasks. However for the scope of the conference, I think the lack of comparison to state-of-art models as well as insights on how to even combine this inductive bias with state-of-art models makes this paper not suitable for application and making real impact on the task of action recognition. It is also not entirely true to assume that state-of-art model, which is much more complicated than a ResNet50 network does not implicitly extract information from the background and body when recognizing action.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: This work examines how neural networks classify actions in video frames and provides a comparison with human behavioral performance. The authors manipulate visual stimuli by isolating bodies from backgrounds and demonstrate that both neural networks and humans achieve high performance across full, body-only, and background-only conditions. Performance differences emerge in the human data, with body-only stimuli outperforming background-only stimuli. A proposed brain-inspired modular architecture is introduced, which enhances overall performance and aligns more closely with human behavioral patterns.
Strengths: Developing networks for action recognition has substantial practical relevance. Additionally, analyzing how well neural networks align with human visual perception represents an important research direction for advancing machine-human visual correspondence.
Weaknesses: The core issue is that poorly curated and insufficiently controlled datasets produce misleading and uninterpretable findings, which represents a persistent problem in this work. Figure 1 (bottom left) claims baseline models show comparable performance across ORIG, body, and background conditions, but the data do not support this claim. Results lack error bars and lack any reasonable statistical rigor. Visual inspection suggests the ranking is ORIG>>BG>>Body. The observation that humans achieve over 0.7 accuracy on background-only frames reveals critical problems: the dataset is overly simplistic, background information serves as a strong confound, and temporal information appears unnecessary for this task. Regarding the proposed architecture, it appears to be trained using substantially different procedures than baseline models. The architecture trained on body-only data outperforms on body-only tests, while the architecture trained on background-only data does not show improvement on background-only tests (and again, without error bars or statistical testing, all observations are approximate). Better training protocols improve outcomes generally. A thorough validation of how effectively YOLO v8 isolates bodies from backgrounds would strengthen the work. The manuscript presents only a single primary figure with minimal supporting material, falling well short of publication standards. The work lacks error bars, comparisons across multiple architectures, analysis across different datasets, ablation studies, and investigation of key variable effects such as stimulus size.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This work examines how artificial neural networks assign labels to action-recognition video frames and contrasts this neural network behavior with human visual performance. The authors modify stimuli to isolate bodies from backgrounds, demonstrating that both neural networks and humans exhibit strong performance across full stimuli, body-only, and background-only conditions. Human performance is notably superior in body-only versus background-only scenarios. A proposed architecture inspired by principles of modularity observed in biological brains is introduced, yielding improved performance and better alignment with human behavioral patterns.
Strengths: Developing networks capable of recognizing actions holds significant practical value in applied domains. Additionally, evaluating network performance relative to human performance is valuable for understanding how to better integrate machine vision systems with human visual understanding.
Weaknesses: The central issue is that poorly curated and insufficiently controlled datasets produce confounded and difficult-to-interpret findings throughout the work. In Fig. 1 (bottom left), the authors assert that baseline models perform equivalently across ORIG, body, and BG conditions, which the data plainly contradicts. Specifically, the visual pattern appears to be ORIG>>BG>>Body. The absence of error bars and statistical testing prevents rigorous evaluation. The observation that humans achieve accuracy exceeding 0.7 on background-only frames indicates three critical problems: (1) the dataset lacks sufficient difficulty, (2) background information serves as a substantial confound, and (3) temporal information is unnecessary for such a trivial classification task. The proposed architecture is trained under substantially different conditions than baseline models. When trained exclusively on Body stimuli, it outperforms on Body-only tests, yet when trained on BG stimuli, no improvement is observed on BG-only tests (again, lacking error bars and statistical rigor). Superior training protocols generally yield better results. Quantitative validation of how effectively Yolo v8 segments body from background would strengthen the work. The paper presents only one primary figure with insufficient supplementary material failing to meet basic disciplinary standards. Critical components are missing: error bars, evaluations across multiple models, comparisons using different datasets, ablation studies, and systematic analysis of key variables including spatial scale.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor