Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception
Authors: Shuangpeng Han
Ziyu Wang
Mengmi Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: biological motion perception
generalization
video action recognition
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose an AI model for video action recognition, which can generalize to biological motion perception tasks.

Abstract: Biological motion perception (BMP) refers to humans' ability to perceive and recognize the actions of living beings solely from their motion patterns, sometimes as minimal as those depicted on point-light displays. While humans excel at these tasks \textit{without any prior training}, current AI models struggle with poor generalization performance. To close this research gap, we propose the Motion Perceiver (MP). MP solely relies on patch-level optical flows from video clips as inputs. During training, it learns prototypical flow snapshots through a competitive binding mechanism and integrates invariant motion representations to predict action labels for the given video. During inference, we evaluate the generalization ability of all AI models and humans on 62,656 video stimuli spanning 24 BMP conditions using point-light displays in neuroscience. Remarkably, MP outperforms all existing AI models with a maximum improvement of 29\% in top-1 action recognition accuracy on these conditions. Moreover, we benchmark all AI models in point-light displays of two standard video datasets in computer vision. MP also demonstrates superior performance in these cases. 
More interestingly, via psychophysics experiments, we found that MP recognizes biological movements in a way that aligns with human behaviors. Our data and code are available at https://github.com/ZhangLab-DeepNeuroCogLab/MotionPerceiver.

Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception
Shuangpeng Han 1.2, Ziyu Wangl.24 12.33 Mengmi Zhang
Cooleee of Computing and Data Science, Nanyang Technological University, Singapore 2 Deep NeuroCognition Lab, Agency for Science, Technology and Research (A (A*STAR) Show Lab, National University of Singapore, Singapore Address correspondence to mengmi.zhang @ ntu.edu.sg
Abstract
Biological motion perception (BMP) refers to humans' ability to perceive and recognize the actions of living beings solely from their motion patterns, sometimes as minimal as those depicted on point-light displays. While humans excel these tasks without any prior training, current AI models struggle with poor generalization performance. To close this research gap, we propose the Motion Perceiver (MP). MP solely relies on patch-level optical flows from video clips inputs. During training, learns prototypical flow snapshots through competitive binding mechanism and integrates invariant motion representations to predict action labels for the given video. During inference, we evaluate the generalization ability of all Al models and humans 62,656 video stimuli spanning 24 BMP conditions using point-light displays in neuroscience. Remarkably. MP outperforms all existing AT models with maximum improvement of 29% in top-1 action recognition accuracy on these conditions. Moreover, we benchmark all AI models in point-light displays of standard video datasets computer vision. MP also demonstrates superior performance in these cases. More interestingly, via psychophysies experiments, we found that MP recognizes biological movements a way that aligns with human behaviors. Our data and code are available at link
Introduction
Training
Testing
Temporat orcer
Temporatrrsot
180
Amount/lffttim of vivicual Info.
Invoriance to camera view
RGB videos
Joint videos
SP videos
Figure 1: Humans excel at biological motion perception (BMP) tasks with zero training, while current AI models struggle with poor generalization performance. recognize actions from natural RGB videos and tested using BMP stimuli on point-light displays, which come two forms: Joint videos, which display only the detected joints of actors in white dots, and Sequential position actor videos (SP), where light points in white are randomly positioned between joints and reallocated to other random positions on the limb subsequent frames (See. (.1). Note that skeletons, shown in gray in the example video, are not visible to humans AI models during testing. The generalization performance of both humans and models assessed after varying five properties in temporal and visual dimensions. See Appendix, Sec. A.1 for example videos.
models are trained to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Biological Motion Perception (BMP) refers to the remarkable ability to recognize and understand
the actions and intentions of other living beings based solely on their motion patterns B3 BMP is
crucial for tasks such as predator detection [24], prey selection [23 courtship behavior [69] and
social communications [88 2 among primates and humans. In classical psychophysical and
neurophysiological experiments [4], motion patterns can sometimes be depicted minimally, such as in voint-light displays where only major joints of human or animal actors are illuminated. Yet, without any prior training, humans can robustly and accurately recognize actions [6
71 67 and
characteristics of these actors like gender [53 78). identity
[6 731, personalities 77 emotions 200
social interactions [93], and casual intention
To make sense of these psychophysical and neurophysiioogical data, numerous studies [3 34 55 89 9 [52) have proposed computational frameworks and models in BMP tasks. Unlike humans, who excel at BMP tasks without prior training, these computational models are usually trained under specific BMP conditions and subsequently evaluated under different BMP conditions. However, the extent to which these models can learn robust motion representations from natural RGB videos and generalize them to recognize actions on BMP stimuli remains largely unexplored.
In parallel to the studies of BMP in psychology and neuroscience, action recognition on images and videos in computer vision has evolved significantly over the past few decades due to its wide The field has progressed from relying 54 to employing deep-learning-beee based approaches [90, 995
range of real-worls applications [1]
41
81
82
57 74
heavily on hand-crafte crafted features [99 75
These modern approaches can capture the temporal dynamics and spatial configurations of complex activities within dynamic and unstructured environments
101 30
29 25
[80] 92
Despite these advancements, existing Al models still struggle with generalization issues related to noisy environments [109]. viewpoint variability (27 61 100). subtle human
occlusion [03 3
58
movements 83 44 and appearance-free motion information [42]. While various solutions have been
proposed to enhance AI generalization in action recognition 122 72 14, 70, 104 48 59
13 79 50
they do not specifically tackle the generalization challenges in BMP tasks.
Here, our objective is to systematically and quantitatively examine the generalization ability of AI models trained on natural RGB videos and tested in BMP tasks. To date, research efforts in neuroscience and psychology 44 34 37 have mostly focused on specific stimuli for individual BMP tasks. There is lack of systematic, integrative, and quantitative exploration that covers multiple BMP properties and provides a systematic benchmark for evaluating both human and AI models in these BMP tasks. To bridge this gap, we establish benchmark BMP dataset, containing 62,656 video stimuli in 24 BMP conditions, covering 5 fundamental BMP properties. Our result indicates that current AI models exhibit limited generalization performance, slightly surpassing chance levels.
Subsequently, to enhance the generalization capability of AI models, we draw inspiration from (33 in neuroscience and introduce Motion Perceiver (MP). MP only takes dense optical flows between any pairs of video frames as inputs and predicts action labels for the given video. In contrast to many existing pixel-level optical flow models [90 91
86 99), MP calculates dense optical flows at the granularity of patches from the feature maps. In MP, we introduce a set of flow snapshot neurons that learn to recognize and store prototypical motion patterns by competing with one another and binding with dense flows. This process ensures that similar movements activate corresponding snapshot neurons, promoting consistency in motion pattern recognition across patches of video frames. The temporal dynamics within dense flows can vary significantly depending on factors such as the speed, timing, and duration of the actions depicted in video clips. Thus, we also introduce motion invariant neurons. These neurons decompose motions along four motion directions and integrate their magnitudes over time. This process ensures that features extracted from dense flows remain invariant to small changes and distortions in temporal sequences.
We conducted a comparative analysis of the generalization performance of MP against existing AI models in BMP tasks. Impressively, MP surpasses these models by 29% and exhibits superior performance in point-light displays of standard video datasets in computer vision. Additionally, we examined the behaviors of MP alongside human behaviors across BMP conditions. Interestingly, the behaviors exhibited by MP in various BMP conditions closely align with those of humans. Our main contributions are highlighted:
1. We introduce a comprehensive large-scale BMP benchmark dataset, covering 24 BMP conditions and containing 62,656 video stimuli. As an upper bound, we collected human recognition accuracy in BMP tasks via a series of psychophysies experiments.
2

---

Patch-Level Optical Flow
Slots
Attention
Lsstot T Emb.
flow
Fusion
Flow Snapshot Neurons
ONIO
Patch-level Optical Flow
Fuslon
Feature
L
Motion Invariant Neurons
Fusion
Feature
Linvar
Figure 2: Architecture of our proposed Motion Perceiver (MP) model. Given a reference patch (yellow or green example patches), MP computes its patch-level optical flow (red arrows, Sec. on the feature maps extracted from DINO [10] Subsequently, these flows are processed through flow snapshot neurons (Sec. 2.2) and motion invariant neurons (Sec. 2.3 two pathways. Activations
from both groups of neurons then integrated for action classification (Sec. 2.4) Time embeddings (T Emb.) are used in the feature fusion process.
2. We propose the Motion Perceiver (MP). The model takes only patch-level optical flows of videos as inputs. Key components within MP, including flow snapshot neurons and motion invariant neurons, significantly enhance its generalization capability in BMP tasks.
Our MP model outperforms all existing Al models in BMP tasks, achieving up to 29% increase in top-1 action recognition accuracy, and demonstrating superior performance in point-light displays of two standard video datasets in computer vision.
The behaviors exhibited by the MP model across various BMP tasks demonstrate a high degree of consistency with human behaviors in the same tasks. Network analysis within MP unveils crucial insights into the underlying mechanisms of BMP, offering valuable guidance for the development of generalizable motion perception capabilities in Al models.
2
Our Proposed Motion Perceiver (MP)
Our proposed model, Motion Perceiver (MP), aims to learn robust action-discriminative motion features from natural RGB videos and generalize these capabilities to recognize actions from BMP stimuli on point-light displays. MP takes the inputs of only patch-level optical flows from video V. which comprises T frames denoted {21, It, (e.....ii Ir). While visual features from videos are typically useful for action recognition, our research focuses on extracting and learning motion information alone from natural RGB videos. Finally, MP outputs the predicted actions from predefined set of labels y. See Fig. 2 for the model architecture.
2.1
Patch-level Optical Flow
Pixel-level optical flow has been a common approach for modelling temporal dynamics in videos 99]. Different from these works, we use the frozen VIT (21), pre-trained on ImageNet 19 with DINO 10,, to extract feature map F, RCC from 1, and compute its optical flow relative
90
91
86
to other frames. Consider F, 2D grid of patches in N H x W, where H and W are the height and width of F, N represents the number of patches in the feature map and is the feature dimension per patch. A unique spatial location for each patch can be defined by its 2D coordinates. We concatenate the X and Y coordinates all patches in F; to form the patch locations G E IR N x2
Two reasons motivate us to design patch-level optical flows. First, the empirical evidence [36 suggests that current unsupervised feature learning frameworks produce feature patches with semantically consistent correlations among neighboring patches. Therefore, patch-level optical flows convey meaningful motion information about semantic objects video. Second, pixel-level optical flows can be noisy due to motion blur, occlusions, specularities, and sensor noises. Feature maps obtained from feed-forward neural networks often mitigate these low-level perturbations and provide more accurate estimations of optical flows.

---

Next, we introduce how the patch-level optical flow for video V is computed. Without loss of generality, given any pair of feature sets a E Rmxd and E Ruxd where m and are the numbers of patches in the feature sets and d is the feature dimension, the adjacency matrix Qa. between a and b ( /r can be calculated their normalized pairwise feature similarities: Q(a.0) Ee RRRx.
where the superscript T is the transpose function, f(-) is the ly-normalization, and T is the temperature controlling the sharpness of distribution with its smaller values indicating sharper distribution. We set temperature 7 = 0.001 and the influence of r is analyzed in Appendix, Tab, S4
Using the patch features of I as the reference, the optical flow between any consecutive frames Im and m++ is defined 0 m-+m+1 ERNx2
O m1 = -lm+ - Gl,-tmm where GI,-11, = Q(F,FFF)G.
(1)
Cr,-r represents the positions of all patches from F; of I, after transitioning to F3 of 15. Essentially, the positions of patches with shared semantic features on F; tend to aggregate towards the centroids of corresponding semantic regions on Pj. Consequently, the optical flow - indicates the movement from Lm to 1 for patches that exhibit similar semantic features as those in 4.-
For all m E f1,2, .... T 1} and E f1.2...., T}, we compute Oh and concatenate them to obtain the patch-level optical flow Ô E (x*)Xxx((-)) for video V as Ô = 10' .... OF (1-1)-
Since small optical flows might be susceptible to noise or errors, the optical flows in Ô with magnitudes less than 7 0.2 are set 10-6 to maintain numerical stability. Unlike [90] where optical flows are computed only between two adjacent frames, our method introduces Ô, where we compute a sequence of optical flows across all 7 frames for any reference patch of video frame. This dense optical flow estimation approach at the patch level captures a richer set of motion dynamics, providing more comprehensive analysis of movements throughout the video.
2.2 Flow Snapshot Neurons
Drawing on findings from neuroscience that highlight biological neurons selective for complex optic flow patterns associated with specific movement sequences in motion pathways [3 introduce "flow snapshot neurons". These neurons are designed to capture and represent prototypical moments or "slots" in movement sequences. Mathematically, we define K 6 flow snapshot neurons or slots Z E IR KxD where 1) 2 x (T' 1) is the feature dimension per slot. ž contains learnable
parameters randomly initialized with the Xavier uniform distribution 35 The impact of K is
discussed in Appendix, Tab. S4
63 separates and organizes different elements of an input into a fixed number of learned prototypical representations or "slots". 'he learned slots have been useful 51 22]. Here, we follow the training paradigm and implementations of the slot attention mechanism in 63 and apply it to Ó. Specifically, each slot in ğ attends to unique optical flow sequence patterns in Ô through competitive attention mechanism based on feature similarities. To ensure the prototypical optical flow patterns in 2 are diverse, we introduce the loss of contrastive walks Lstet
Slot attention mechanism
for semantic segmentation 102 60
[022. During inference, we keep 2 frozen and leverage the activations of flow snapshot neurons for action recognition, denoted as M = =()) where M E R See Appendix, Sec. B. and Appendix, Sec. B.2 for mathematical formulations.
Unlike [108] that use slot-bonded optical flow sequences M1 2 X for downstream tasks, we highlight two advantages of using M. First, the dimensionality of flow similarities (K) is typically much smaller than that of slot- dlot-bonded optical flow sequences (D), effectively addressing overfitting concerns and reducing computational overhead. Second, by leveraging flow similarities, we benefit from slot activations that encode prototypical flow patterns distilled from raw optical flow sequences. This filtration process helps eliminate noise and irrelevant information from the slot-bonded flow sequences, enhancing the model's 's robustness.
2.3 Motion Invariant Neurons
The temporal dynamics in video clips can vary significantly in the speed and temporal order of the actions. Here, we introduce motion invariant neurons that capture the accumulative motion magnitudes independent of frame orders. Specifically, define the optical flow (0-at O%:n.!

---

for patch n in Im moving from frame m to I, along and axes. Every patch-level optical flow in Ô can be projected into four motion components along tu, -t, ty and -y axes. Without loss of generality, we show that for patch in I,, all its optical flow motions over 7 frames along the +r axis are aggregated:
{o if 0, otherwise.
Σ Om
O >0 where 1u>0
By repeating the same operations along -I, +y, y axes for all the N patches over T frames, obtain the motion invariant matrix M E RTxNx4 A stack of self-attention blocks followed by global average pooling layer fuse information along 4 motion components and then along temporal dimension T. To encourage the model to learn motion invariant features for action recognition, we introduce cross-entropy loss Linvor to supervise the predicted action against the ground truth y
2.4 Multi-scale Feature Fusion and Training
Processing videos at multiple temporal resolutions allows the model to capture both rapid motions and subtle details, as well as long-term dependencies and broader contexts. We use the subscript in Ô, 2, and M to indicate the temporal resolution. Instead of computing Ö₂ between consecutive frames only, we increase the stride sizes to 2, 4, and 8. For example, O4 E R²*NN2x(7//-11 denotes patch-level flows computed between every 4 frames. Note that we maintain a stride size of 1 for M to ensure motion invariance to temporal orders across the entire video, as subsampling frames would not adequately capture this attribute.
For action recognition, the activations of flow snapshot neurons are concatenated across multiple temporal resolutions as M1.2,4 [M] M2, M4, MM]] and then used for the fusion of motion information. The concatenated data is first processed through series of self-attention blocks that operate across 4K slot dimensions, followed by a global average pooling over these dimensions. We then repeat the same fusion process over 7 time steps. Time embeddings, similar to the sinusoidal positional embeddings in (98). are applied across frames. These embeddings help incorporate temporal context into the learned features. The resulting integrated motion feature vector is used for action recognition, with a cross-entropy loss Lftow to supervise the predicted action against V
While M1,2.4.8 captures the detailed temporal dynamics, M learns robust features against variations in temporal orders. MP combines feature vectors from M1.2,1 and M with fully connected layer and outputs the final action label. A cross-entropy loss Luse is used to balance the contributions from 21,2,4.8 and M. The overall loss is: L = aL + LLioww + Linvar + L Lfuse where the loss weight a = 10. See Appendix, Tab. Tab.[S4for for the effect of a.
Implementation Details. Our model is trained on Nvidia RTX A5000 and A6000 GPUs, and optimized by Adam optimizer
[65] with cosine annealing scheduler (64] starting from the initial learning rate 10 Data loading is expedited by FFCV [56]. All videos are downsampled to T = 32 frames. We use random crop of 224 x 224 pixels with horizontal flip for training, and central crop of 224 x 224 pixels for inference. We use the same set of
for our model in all
the datasets. More training details are in Appendix, Sec. C
3 Experiments
3.1
Our Biological Motion Perception (BMP) Dataset with Human Behavioral Data
Following the works in vision science, psychology, and neuroscience [33 89
9
533
6
16
37
45
[7] we introduce the BMP dataset, comprising 10 action classes (Appendix, Sec. specifically chosen for their strong temporal dynamics and minimal reliance visual cues. Studies [107 112
32 (55 indicate that current action recognition models often rely on static features. Point-light displays reduce confounding factors, such as colors, sketches, and body limbs, by minimizing visual information, thereby highlighting the ability to perceive motion. This selection criterion ensures that specific objects or scene contexts, such as a soccer ball green grass, do not bias the Al models toward recognizing the action as "playing soccer". This approach aligns with our research focus on generalization in motion perception. In the BMP dataset, there are three types of visual stimuli.
Natural RGB videos (RGB): We incorporated 9,492 natural RGB videos from the NTU RGB+D 120 dataset 62 and applied a 7-to-3 ratio for dividing the data into training and testing splits. Joint
5

---

videas (J): We applied Alphapose [26] to identify human body joints in the test set of our RGB videos. The joints of a moving human are displayed as light points, providing minimal visual information other than the joint positions and movements of these joints over time. Sequential position actor videos (SP): SP videos are generated to investigate scenarios where local inter-frame motion signals are eliminated [6]. Light points are positioned randomly between joints rather than on the joints themselves. In each frame, every point is relocated to another randomly selected position on the limb between the two joints with uniform distribution. This process ensures that no individual point carries the valid local image motion signal of limb movement. Nonetheless, the sequence of static postures in each frame still conveys information about body form and motion.
We investigated fundamental properties of motion perception by manipulating various design parameters of these stimuli. We adopted the experiment naming convention [type] + [condition] to denote the experimental setup for the stimulus types and the specific manipulation conditions applied to them. For instance, [Joint-3-Framen from the Joint video type. Next, we introduce the fundamental properties of motion perception and the manipulations performed to examine these properties.
indicates that three frames are uniformly sampled
Temporal order (TO):
we reverse (Reversal) or randomly shuffle (Shuffle) video frames for two types of videos, RGB and Joint above. Temporal resolution (TR): We alter the temporal resolution (4 by uniformly downsampling 32 frames to
To disrupt the temporal order
frames, labelled as 4- 4-Frames and 3-Frames, respectively. For models that need a fixed frame count, each downsampled frame is replicated multiple times before advancing to the next frame in the sequence, until we reach the necessary quantity. Amount of visual information (AVI): We quantify the amount of visual information based on the number of light points (P) in Joint videos (45). Specifically, we included conditions: 5, 6, 10, 14, 18, and 26 light points. Lifetime of visual information (LVI): In SP videos, we manipulate the number of consecutive frames during which each point remains at specific limb position before being randomly reassigned to a different position. Following [6]. we refer to this parameter as the "lifetime of visual information" (LT), and include LT values of 1, 2, and 4. A longer lifetime implies that each dot remains at the same limb position for longer duration, thereby conveying more local image motion information. Invariance to camera views (ICV): Neuroscience research [43) has revealed brain activity linked to decoding both within- and cross-view action recognition. To probe view-dependent generalization effects, we sorted video clips from Joints into three categories based on camera views: frontal, 45 and 90 views. See Appendix, Sec. A.3 for more implementation details.
Human psychophysies experiments: We conducted human psychophysics experiments schematically illustrated in Appendix, Sec. using Amazon Mechanical Turk (MTurk) [8] A total of 90 subjects were recruited. For data quality controls, we implemented checks during the experiment. Following a set of filtering criteria, we retained data from 88 subjects See Appendix, Dec.Door details.
3.2 Video Action Recognition Datasets and Baselines in Computer Vision
To evaluate the ability of all Al models to recognize a wider range of actions in natural RGB vidcos, we include two standard video action recognition datasets in computer vision. NTU RGB+D 60 [87] comprises 56,880 video clips featuring 60 action classes. NW-UCLA [100] contains 1,494 videos featuring 10 action classes. Both datasets are split randomly, with 7:3 ratio for the training and test sets. All Al models are trained on RGB videos and tested on Joint videos of these datasets curated with Alphapose [26] described above. We compute the top-1 action recognition accuracy of all AI models. In addition to our MP model, we include six competitive baselines below for comparison: ResNet3D (88 adapts the architecture of ResNet (3 by substituting the 2D convolutions with the 3D convolutions. I3D 11] extends a pre-trained 2D-CNN on static images to 3D by duplicating
2D filters along temporal dimensions and fine-tuning on videos. X3D (88) expands 2D-CNN progressively along spatial and temporal dimensions with the architecture search. R(2+1)D [96]
factories the 3D convolution kernels into spatial and temporal kernels. Slow [29] is a two-stream architecture that processes temporal and spatial cues separately. MVIT (25) is a transformer based model that expands feature map sizes along the temporal dimension while reducing them along the spatial dimension. Furthermore, we include three more competitive baselines (E2-S-X3) S-33 [4]. VideoMAE (94), TwoStream-CNe remain consistent with the baselines above. All baselines except for TwoStream- are pre-trained on Kineties 400 47. TwoStream-CNN is pre-trained on ImageNet [19]. As an upper bound, we also
(90) and present results in Appendix, Sec. E The findings
train the MP directly on Joint SP videos and the results in Appendix, Sec. E

---

Hunas
MI2-130
X30
100
MP (Ours)
80
Top-1 Accurscy (%)
Top-1 Accuracy
co
20
NGB
ROB ff
Jana
Fand
Jar-at
Joraar
RG9
J:00
J-100
J-140
J-1or
1:
Figure 3: Temporal orders and resolutions
Figure 4: Our model demonstrates human-like robustness under reduced visual information. Top-1 action recognition accuracy is a function of the number of points (P) in Joint (J) videos. Results from RGB test videos are at the leftmost. The colored shaded region represents the standard error across all action classes.
matter in generalization performance on RGB and Joint videos. Stimuli encompass RGB and Joint (J) videos. Short forms include R (reversal). S (shuffle), F (frames), and P (points) in Sec. Error bars indicate the standard error of the top-1 accuracy across different action classes.
4 Results
4.1
Our model achieves human-level performance without task-specific retraining
We explore five key properties of motion perception and their influence on video action recognition tasks between humans and our model (MP). Detailed comparisons between humans and all Al models across all BMP tasks are provided in the Appendix, Sec. L
Temporal order and temporal resolution matter more in Joint videos than natural RGB videos. In Fig. both humans and MP exhibit higher top-1 accuracy in RGB videos (RGB) compared to Joint videos (J-6P) due the increased difficulty of generalization in Joint videos from their minimal motion and visual information. Error bars in Fig.3 repreesen the standard error of top-1 accuracy across different action classes. However, despite this challenge, the performance on Joint videos remains significantly above chance (1/10). This suggests that both humans and MP have remarkable abilities recognize actions based solely on their motion patterns in point-light displays.
Moreover, we observed that shuffled (S) or reversed (R) temporal orders significantly impair recognition performance in both humans and MP. This effect is more pronounced in Joint videos (J-6P-R and J-6P-S) compared to RGB videos (RGB-R and RGB-S). The minimal motion information available in Joint videos makes them particularly susceptible to disruptions in temporal orders. The same behavioral patterns are also captured by MP. Interestingly, shuffling has a lesser impact on human performance compared to reversing RGB videos (RGB-R versus RGB-S, p-value 0.05). However, this pattern is reversed in Joint videos (J-6P-R versus J-6P-S, p-value < 0.05). When considering actions such as sitting down versus standing up, reversing orders may alter the temporal context in RGB videos. The effect of temporal continuity outweighs the effect of temporal context in Joint videos.
We conjectured that temporal resolution matters in video action recognition. Indeed, a decrease in top-1 I accuracy with decreasing temporal resolutions is observed in both RGB and Joint videos (compare 32, 4, and 3 frames). However, this effect is more pronounced in Joint videos (J-6P, J-6P-4F, J-)P-3F) compared to RGB videos (RGB, RGB-4F, 3GB-3F). Surprisingly, even in the most challenging J-6P-3F, both humans and MP achieve top-1 accuracy of over 40% significantly above chance. This suggests that both humans and MP are robust to changes in temporal resolutions.
Minimal amount of visual information sufficient for recognition. In Fig. both humans and all Al models exhibit comparable performances in RGB videos (RGB). Interestingly, the accuracy drop from J-6P to J-5P indistinguishable for humans (p-value 0.05). Similarly, there was a wide range of the number of points that led to robust action recognition for MP (J-26P to J-5P). However,
7

---

its absolute accuracy is much lower than humans. Therefore, we introduce an enhanced version of the MP model (En-MP) by extending across multiple DINO blocks, in addition to the last DINO block. Detailed implementation of the En-MP is provided in Appendix, Sec. Surprisingly. the En-MP outperforms our original MP significantly and performs competitively well as humans. These observations suggest that minimal visual representation in J-5P is sufficient for action recognition on humans and our En-MP. Conversely, traditional AI models show a significant performance decline moving from J-26P to J-5P. These findings suggest that existing AI models struggle with reduced visual information.
Humans and MP are robust to the disrupted local image motion. Aligned with [6 76 in Fig. 5 top-1 accuracy in SP videos improves with an increased number of points for humans, trend that MP replicates. Interestingly, both humans and MP do not show obvious increased performance with lifetimes of points more than frame, potentially due to the loss of form information from fewer dots. This indicates that humans and MP capture not just local motions but also dynamic posture changes.
Camera views have minimal impacts on recognition. Neuroscience studies 143 97 46, (68)
provide evidence that both vievpoint-dependen and viewpoint-invarian representations for action recognition are encoded in primate brains. In Fig. 7 MP exhibits slightly higher accuracy on frontal and 45 views compared to the 90" (profile) view by 2.5%, which mirrors human performance patterns in [46.. However, we note that human behaviors in our study differ slightly from [46] (see Appendix, Sec. G Moreover, MP significantly outperforms the second- best model, MViT, by 18%. This suggests that MP not only captures numan-like viewpoint-depente maintains superior accuracy among AI models across different camera views.
representations but also
4.2 Comparisons among Al models in BMP tasks and standard computer vision datasets
MP aligns with human behaviors more closely than all the competitive baselines in BMP tasks. In Fig. 6 we reported the correlation coefficients between each Al model and human performance across all conditions in the BMP dataset. The results demonstrate that our MP exhibit significantly higher correlation with human performance compared to the baselines, indicating that our MP align more closely with human performance. In addition to the correlation coefficients, we also present the error pattern consistency (31) between models and humans (Appendix, Sec. H Appendix, Fig. (S4). Results suggest that our En-MP achieves the highest error pattern consistency with humans the trial level.
MP significantly outperforms all the competitive baselines in BMP tasks. In Appendix, Fig. S3 we present the absolute accuracy of all models and humans. The slopes near I in our MP model indicate that it performs on par with humans and suraasees all baselines across the five BMP dataset properties. Despite explicitly modelling motion and visual information in separate streams, the SlowFast model struggles with temporal order. Likewise, MViT, which leverages transformer architectures, fails to generalize across different temporal resolutions.
MP outperforms all the competitive baselines on Joint videos from two standard computer vision datasets. In Tab. [ MP, relying solely on patch-level optical flows as inputs, performs above chance. This suggests that motion information alone is sufficient for accurate action recognition. Morcover, MP performs better than existing models on Joint videos in these datasets, highlighting that MP learns to capture better motion representations from RGB videos during training. In Appendix, Sec.[ we provide results and discussions when we explicitly feed pixel-level optical flows as inputs to these baselines during training and test these models the Joint videos. Our results show that MP still outperforms these models, suggesting that MP learns generalizable motion representations beyond optical flow patterns. In Appendix, Sec. J also present visualizations of petch-level optical flow examples, demonstrating its ability to semantically segment the movements of a person.
4.3 Ablation studies reveal key components in our model
To investigate how different components of MP contribute to the generalization performance on Joint and SP videos, we conducted ablation studies (Tab. [ Appendix, Tab. SS4 and Appendix, Tab. S55). Removing the pathway involving motion invariant neurons leads to drop in accuracy when video frames are shuffled or reversed (A1, Tab. 2 and Appendix, Fig. (3). For example, MP outperforms its ablated model without motion invariant neurons in the following experiments: 62.32% vs 49.58% in RGB-R; 61 .34% vs 38.00 88.03% in RGB-S, 38.69% vs 36.03% in J-6P-R, and 32.65% % vs 25.28% in J-6P-S. Similarly, removing the pathway involving flow snapshot neurons also leads to significant
8

---

1.0
100
Human MP (Ours)
O
8P
0.8
80
Top -1 Accuracy (%)
Correlation
60
0.4
0.2
0.0
ResNet3D
130
RG+11
SlowFast
X3D
MVT
MP (Ours)
Lifetime (Frames)
Figure Both humans and our model can recognize actions in SP videos without local motions. Performance varies depending on the persistence of visual information, with stimuli having and 8 points (P) of the actors (Sec. 3.1
Figure 6: Our MP model shows significantly stronger correlation with human performance compared to all baselines. The correlation between the model and human performance across all BMP conditions is presented.
100
MP
Method
Top-1 Acc. (%)
NTU RGB+D 60
NW-UCLA
Top-1 Accuracy (%)
80
ResNet3D
6.89
10,99
I3D
4.46
8.74
60
X3D
2.00
16.59
R(2+1)D 5.18
10.54
SlowFast 6.26
10.09
MVIT
6.56
10.09
MP (ours)
20.38
42.83
frontal view
45 view
50" view
Figure 7: Humans and AI models show minimal difference in generalization across camera views J-6P videos (Sec. 3.1). Error bars indicate the standard error.
Table 1: Our model outperforms all existing models on Joint videos (J-6P) from two standard computer vision datasets. See Sec. 3.2 for dataset descriptions. Best is in bold.
accuracy drop (A2). Both ablation studies highlight the importance of two pathways for action recognition.
Instead of multi-scale feature fusion, we utilize a single-scale pathway flow snapshot neurons (A3). The decrease of 10.57% in accuracy J-6P videos implies that multi-scale flow snapshot neurons capture both fine fine-grained and high-level temporal patterns, essential for action recognition. In A4, the model takes patch-level optical flows as inputs and directly utilizes them for feature fusion and action classification without flow snapshot neurons. There is a significant drop 14.33% in accuracy, implying that flow snapshot neurons capture meaningful prototypical flow patterns, and their activations are critical for recognition. Threshold y controls the level of noise or errors small motions. As expected, removing 7 leads to impaired recognition accuracy (A5).
Data augmentation common technique to enhance model generalization 114 49). It typically includes temporal augmentations like random shuffling and reversal in A6. Surprisingly, MP's generalization performance impaired, indicating that randomizing frame orders during training disrupts motion perception and action understanding. We also present ablation results in RGB videos, with similar conclusions drawn, albeit to lesser extent due to increased visual information and reduced reliance motion.
In addition to the main ablation results discussed here, we also provide the extra ablation studies and model analysis in the Appendix. Specifically, we include the following studies: 1. the removal of the time embedding our MP (Sec. 2.4, Appendix, Tab, Tab.555): 2. the removal of the loss term Lostot in our MP (Sec. 2.2, Appendix, Tab. S5 3. the choice of the reference frame for calculating path-level
optical flows in our MP (Sec. 2.1. Appendix, Tab. S5); 4. the pixel-level optical flows downscaled to the size of the patch-level optical flows in MP (Appendix, Sec. K 5. the replacement of

---

P-MIN
P-ESN
SS
2
Thres. 7
T A
Top-1 Acc. (%)
Ablation ID
Slous
RGB
J-6P
SP-8P-ILT
93.47
64.54
49.23
x
2
x
86.17
42.52
25.91
3
x
96.07
58.43
30.83
4
96.84
54.67
35.43
5
96.70
42.38
29.42
6
93.33
58.32
31.95
Full Model (ours)
96.45
69.00
49.68
Table 2: Ablation reveals critical components in our model. Top-1 accuracy is reported on RGB videos, Joint videos with 6 points (J-6P), and SP videos with 8 points and lifetime of (SP-8P-1LT), From left to right, the ablated components are: the pathway with Motion Invariant Neurons (P-MIN), the pathway with Flow Snapshot Neurons (P-FSN), the single-scale branch with Flow Snapshot Neurons Mi (SS FSN), Slots 2, threshold (Sec. 2.1). and data augmentation by randomly shuffling and reversing training frames within the same video. Best is in bold.
the feature extractor DINO with ResNet in our MP (Appendix, Sec. K); the comparison of the number of trainable parameters among all the models (Appendix, Sec. K); 7. the analysis of key frames predicted by our MP on an example video in Appendix, Sec. the importance of our model designs and demonstrated the generalization ability of our model.
K All these studies emphasized
5 Discussion
We introduce Motion Perceiver (MP) as a generalization model trained on natural RGB videos, capable of perceiving and identifying actions of living beings solely from their minimal motion patterns on point-light displays, even without prior training on such stimuli. Within MP, flow snapshot neurons learn prototypical flow patterns through competitive binding, while motion invariant neurons ensure robustness to variations in temporal orders. The fused activations from both neural populations enable action recognition.
To evaluate the generalization capabilities of all AI models, we curated dataset comprising 63k stimuli across 24 BMP conditions using point-light displays inspired by neuroscience. Psychophysics experiments on this dataset were conducted, providing human behavioral data for comparison with computational models. While Al models can surpass human performance in numerous tasks, current AI models for action recognition still fall short of human capabilities many BMP conditions. By focusing solely on motion information, MP achieves superior generalization performance among all AI models and demonstrates a strong alignment with human behavioral responses. All baselines pre-traine trainrd large-scale video datasets, whereas MP uses feature extractors pre-trained on naturalistic images. Remarkably, despite lacking video pre-training, our MP outperforms all baselines.
Our work takes an initial step toward bridging artificial and biological intelligence in BMP. First, it raises intriguing questions in neuroscience, such as the neural basis of motion invariant neurons, which are crucial when video frames are shuffled or reversed. Bio-inspired architectures can help test specific neuroscience hypotheses, while insights from neuroscience can, in turn, guide the design of more advanced Al systems. Second, our work paves the way for real-world applications requiring robust motion recognition, such as in low-light conditions where visual information is limited.
The 10 action classes in our BMP dataset were selected for their rich temporal information. We observe significant variations in action recognition accuracy across various action classes for both human participants and AI models. In the future, the BMP dataset could be expanded to include more complex actions. Both neuroscience and computer vision use various biological motion stimuli; here, we focus on point-light displays, but future work could explore other visual stimuli, such as motion patterns in noisy backgrounds. While our model shows promising generalization on BMP stimuli, it does not account for attention or top-down influences. Moreover, since it only uses patch-level optical flows, integrating information from the ventral (form perception) and dorsal (motion perception) pathways remains an open challenge. Further discussion on the social impact of our work can be found in Appendix, Sec. M
10

---

Acknowledgements
This research supported by the National Research Foundation, Singapore under its Al Singapore and its NRFF award NRF-NRFFIS-2023-0
Programme (AISG Award No: We also acknowledge Mengmi Zhang's Startup Grant from Agency for Science, Technology. and Research (A (AAR), Startup Grant from Nanyang Technological University, and Early Career Investigatorship from Center for Frontier Al Research (CFAR), A*STAR.
References
[1] Ralph Adolphs. The neurobiology of social cognition. Current opinion in neurobiology,
1 11():2211-399 2001.
[2] Richard John Andrew. Evolution of facial expression: Many human expressions can be traced back to reflex responses of primitive primates and insectivores. Science, 142(3595):1034-111 1041, 1963.
[3] Federico Angelini, Zeyu Fu, Yang Long, Ling Shao, and Syed Mohsen Naqvi. 2d pose-based real-time human action recognition with oeclusionhhandlingg IEEE Transactions on Multimedia, 12333)1433-144 1446, 2019.
[4] Anurag Amnb, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. Proceedings of the IEEE/CVF international conference compuler vision, pages 6836-6846, 2021.
[5] Simon Baker, Daniel Scharstein, James P Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International journal of computer vision, 92:1-31, 2011.
[6] JA Beintema and Markus Lappe. Perception of biological motion without local image motion. Proceedings of the National Academy of Sciences, 99())5561-55633 2002.
[7] Sheila Brownlow, Amy Dixon, Carrie A Egbert, and Rebecca D Radcliffe. Perception of movement and dancer characteristics from point-light displays of dance. The Psychological Record, 47:411 12222, 1997.
[8] Michael Buhrmester, Tracy Kwang, and Samuel Gosling. Amazon's mechanical turk: A source of inexpensive, yet high-quality, data? Perspectives psychological science, 6(1):3-5, 2011.
[9] Roxanne I. Canosa. Simulating biological motion perception using recurrent neural network. FLAIRS, pages 617-622, 2004.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[11] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2299-6308. 2017.
[12] Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jackwon Yoo. Ruxin Chen, and Jian Zheng. Temporal attentive aligmment for large-scale video domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6321-6330. 2019.
[13] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:110m 1078, 2014.
[14] Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang. Shuffle and attend: Video domain adaptation. Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 778-695. Springer, 2020.
11

---

[15] Arridhana Ciptadi, Matthew S Goodwin, and James M Rebg. Movement pattern histogram for action recognition and retrieval. In Computer Vision-ECCV 2014: 13th European Conference. Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13, pages 695-710. Springer, 2014.
[16] James E Cutting and Lynn T Kozlowski. Recognizing friends by their walk: Gait perception without familiarity cues. Bulletin of the psychonomic society, 9:353-356, 1977.
[17] Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M Nasrabadi. Supermix: Supervising the mixing data augmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13794-13803, 2021.
[18] Charles Darwin. The expression of the emotions in man and animals, new york: D. Appleton and Company, 1872.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Kai Li, and Li Fei-Fei. Imagenet: A large-scare scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, leee, 2009.
[20] Winand Dittrich, Tom Troscianko, Stephen EG Lea, and Dawn Morgan. Perception of emotion from dynamic point-light displays represented in dance. Perception, 25(6):727-738, 1996.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.11922 2020.
[22] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd Van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems, 35:28940 28954, 2022.
[23] Jörg-Peter Ewert. Neurocthology of releasing mechanisms: prey-catching in toads. Behavioral and Brain Sciences, 10(3):337-368, 1987.
[24] Jörg-Peter Ewert. The release of visual behavior in toads: stages of paraale//hierarchical information processing. In Visuomotor coordination: Amphibians, comparisons, models, and robots, pages 12-110..500 Springer, 1989.
[25] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824 6835, 2021.
[26] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[27] Ali Farhadi and Mostafa Kamali Tabrizi. Learning to recognize activities from the wrong view point. In Computer Vision-ECCV 2008: 10th European Conference Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10, pages 1541666. Springer, 2008.
[28] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages 203-213, 2020.
[29] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEFF/CVF international conference computer vision, pages 6202-6211, 2019.
[30] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1933-1941, 2016.
12

---

[31] Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying tritilb--trial behaviour of enns and humans by measuring error consistency. Advances in Neural Information Processing Systems, 133800133992 2020.
[32] Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
[33] Martin A Giese and Tomaso Poggio. Neural mechanisms for the recognition of biological movements. Nature Reviews Neuroscience, 4(3):179-192 2003.
[34] Martin Alexander Giese and Tomaso Poggio. Biologically plausible neural model for the recognition of biological motion and actions. 2002.
[35] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[36] Mark Hamilton, Zhoutong Zhang. Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint 3xXiv:203300811 2022.
[37] Simon Hanisch, Evelyn Muschter, Admantini Hatzipanayioti, Shu-Chen Li, and Thorsten Strufe. Understanding person identification through gait. arXiv preprint wrXiv:2203.04179 2022.
[38] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE international conference computer vision workshops, pages 3154-3160, 2017.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] Minui Hong, Jinwoo Choi, and Gunhee Kim. Stylemix: Separating content and style for enhanced data augmentation. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 14862-14870, 2021.
[41] Weiming Hu, Dan Xie, Zhouyu Fu, Wenrong Zeng, and Steve Maybank. Semantic-hased surveillance video retrieval. IEEE Transactions on image processing. 16(4): 100-1116811888 2007.
[42] Filip llie, Thomas Pock, and Richard P Wildes. Is appearance free action recognition possible? In European Conference on Computer Vision, pages 156-173, 173. Springer, 2022.
[43] Leyla Isik, Andrea Tacchetti, and Tomaso Poggio. A fast, invariant representation for human action in the visual system. Journal of neurophysiology, 119(2):631 2018.
[44] Vincent Jacquot, Zhuofan Ying, and Gabriel Kreiman. Can deep learning recognize subtle human activities? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14244-14253, 2020.
[45] Gunnar Johansson. Visual perception of biological motion and model for its analysis. Perception & psychophysics, 14:201-211, 1973.
[46] Daniel Jokisch, Irene Daum, and Nikolaus F Troje. Self recognition versus recognition of others by biological motion: Viewpoint-depender effects. Perception, 35(7):911-920) 2006.
[47] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloc Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint
705.06950. 2017.
[48] Taeoh Kim, Jinhyung Kim, Minho Shim, Sangdoo Yun, Myunggu Kang, Dongyoon Wee, and Sangyoun Lee. Exploring temporally dynamic data augmentation for video recognition. arXiv preprint ar0iv: 15015, 2022.
13

---

[49] Taeoh Kim, Hyeongmin Lee, MyeongAh Cho. Ho Seong Lee, Dong Heon Cho, and Sangyoun Lee. Learning temporally invariant and localizable features via data augmentation for video recognition. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part HI 16, pages 386 403.. Springer, 2020.
[50] Kaleab Kinfu and René Vidal. Analysis and extensions of adversarial training for video classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 3416-3425, 2022.
[51] Thomas Kipf, Gamaleldin F Elsayed. Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111. 2021.
[52] Hema S Koppula and Ashutosh Saxena. Anticipating human activities using object affordances for reactive robotic response. IEEE transactions on pattern analysis and machine intelligence, 38(1):14-29, 2015.
[53] Lynn T Kozlowski and James E Cutting, Recognizing the sex of a walker from dynamic point-light display. Perception & psychophysics, 21:575-580, 1977.
[54] Zhengzhong Lan, Ming Lin, Xuanchong Li, Alex G Hauptmamn, and Bhiksha Raj. Beyond gaussian pyramid: Multi-skip feature stacking for action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 204-212, 2015.
[55] Joachim Lange and Markus Lappe. model biological motion perception from configural form cues. Journal of Neuroscience, 26(1 11:2)144229999 2906, 2006.
[56] Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Mądry. Ffev: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12011- 2023.
[57] Kang Li and Yun Fu. Prediction of human activity by discovering temporal sequence patterns. IEEE transactions on pattern analysis and machine intelligence, 1644-167, 2014.
[58] Ziqi Li and Dongsheng Li. Action recognition of construction workers under occlusion. Journal of Building Engineering, 45:1033,,, 2022.
[59] Yuanze Lin, Xun Guo, and Yan Lu. Self-supervieed video representation learning with meta-contrastive network. In Proceedings of the IEEE/CVE international conference on computer vision, pages 8239-8244 8249, 2021.
[60] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint wwXiv:2001.02400) 2020.
[61] Honghai Liu, Zhaojie Ju, Xiaofei Ji, Chee Seng Chan, Mehdi Khoury, Honghai Liu, Zhaojie Ju, Xiaofei Ji, Chee Seng Chan, and Mehdi Khoury. A view-invariant action recognition based multi-view space hidden markov models. Human Motion Sensing and Recognition: A Fuzzy Qualitative Approach, pages 251-2612 2017.
[62] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+ 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, 42(10):2884-7001 2019.
[63] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with 2020.
slot attention. Advances in neural information processing systems, 33:1
[64] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03080038,, 2016.
[65] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01001, 2017.
14

---

[66] Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding, Vdt: Gereral-purpose video diffusion transformers via mask modeling. In The Twelfth International Conference on Learning Representations, 2023.
[67] Hongjing Lu, Structural processing in biological motion perception. Journal of Vision, 10(12):13-13, 2010.
[68] George Mather and Linda Murdoch. Gender discrimination in biological motion displays based on dynamic cues. Proceedings of the Royal Society of London. Series B: Biological Sciences, 258(1353) 1994.
[69] Desmond Morris. The reproductive behaviour of the zebra finch (poephila guttata), with special reference to pseudofemale behaviour and displacement activities. Behaviour, 6(1):271 1954.
[70] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine- aine-grained action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 122-132, 2020.
[71] Peter Neri, M Concetta Morrone, and David C Burr. Seeing biological motion. Nature, 395(6705):894-89)) 896, 1998.
[72] Boxiao Pan, Zhangjie Cao, Ehsan Adeli, and Juan Carlos Niebles. Adversarial cross-domain action recognition with co-attention. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 1 11815-11822, 2020.
[73] Marina A Pavlova. Biological motion processing as a hallmark social cognition. Cerebral Cortex, 22(5):981-995 2012.
[74] Mingtao Pei, Yunde Jia, and Song-Chun Zhu. Parsing video events with goal inference and intent prediction. In 201 International Conference on Computer Vision, pages 487-494. IEEE, 2011.
[75] Xiaojiang Peng, Changqing Zou, Yu Qiao, and Qiang Peng. Action recognition with stacked fisher vectors. In Computer Eision-ECCT 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 581-595 Springer, 2014.
[76] Yujia Peng, Hannah Lee, Tianmin Shu, and Hongjing Lu. Exploring biological motion perception in two-stream convolutional neural networks. Vision Research, 178:28-40, 2021.
[77] Yujia Peng, Steven Thurman, and Hongjing Lu. Causal action: A fundamental constraint on perception and inference about body movements. Psychological science, 28(6):798-877, 2017.
[78] Frank E Pollick, Jim W Kay, Katrin Heim, and Rebecca Stringer. Gender recognition from point-light walkers. Journal of Experimental Psychology: Human Perception and Performance, 31(6): 2005.
[79] Roi Pony, Itay Naeh, and Shie Mannor. Over-the-air adversarial flickering attacks against video recognition networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 115-524, 2021.
[80] Brian Russell, Andrew McDaid, William Toscano, and Patria Hume. Moving the lab into the mountains: a pilot study of human activity recognition in unstructured environments. Sensors, 21(2):654, 2021.
[81] Michael Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In 2011 international conference on computer vision, pages 1036-1043. IEEE, 2011.
[82] Michael S Ryoo, Thomas J Fuchs, Lu Xia, Jake K Aggarwal, and Larry Matthies. Robot-centric activity prediction from first-person videos: What will they do to me? In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, pages 295-302, 2015.
15

---

[83] Jaeyeong Ryu, Ashok Kumar Patil, Bharatesh Chakravarthi, Adithya Balasubramanyam. Soungsill Park, and Youngho Chai. Angular features-based human action recognition system for real application with subtle unit actions. IEEE Access, 10:9645-9657, 2022.
[84] Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi. Vibhav Vineet, and Yogesh S Rawat. A large-scale robustness analysis of video action recognition models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4698-14408, 2023.
[85] Martin Schrimpf, Jonas Kubilius, Michael Lee, Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413 (113):4133444 423, 2020.
[86] Laura Sevilla-Lara, Yiyi Liao, Fatma Güney, Varun Jampani, Andreas Geiger, and Michael J Black. On the integration of optical flow and action recognition. In Pattern Recognition: 40th German Conference, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings 40, pages 281-297. Springer, 2019.
[87] Amir Shahroudy, Jun Liu, Tian-Tsong Ng. and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010-1019, 2016.
[88] Connor Shorten and Taghi M Khoshgoftaar, survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[89] Julia Siemann, Anne Kroeger, Stephan Bender, Muthuraman Muthuraman, and Michael Siniatchkin. Segregated dynamical networks for biological motion perception in the mu and beta range underlie social deficits in autism. Diagnostics, 14(4):408, 2024.
[90] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition videos. Advances in neural information processing systems, 27, 2014.
[91] Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1390-1399, 2018.
[92] Jaeyong Sung, Colin Ponce, Bart Selman, and Ashutosh Saxena. Unstructured human activity detection from rgbd images. In 2012 IEEE international conference on robotics and automation, pages 842-849. IEEE, 2012.
[93] Steven M Thurman and Hongjnn Lu. Perception of social interactions for spatially scrambled biological motion. PloS one, 9(1 e) 1 12539, 2014.
[94] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 35:10078-10093, 2022.
[95] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 1489-4497, 2015.
[96] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference Computer Vision and Patlern Recognition, pages 6450-6459, 2018.
[97] Nikolaus Troje, Cord Westhoff, and Mikhail Lavrov. Person identification from biological motion: Effects of structural and kinematic cues. Perception & Psychophysics, 67:667-675 2005.
[98] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
16

---

[99] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision, pages 3551-3558. 2013.
[100] Jiang Wang, Xiaohan Nie, Yin Xia, Ying Wu, and Song-Chun Zhu. Cross-view action modeling, learning and recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2649-2656, 2014.
[101] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[102] Ziyu Wang, Mike Zheng Shou, and Mengmi Zhang. Object-centric learning with cyclic walks between parts and whole. Advances in Neural Information Processing Systems, 36, 2024.
[103] Daniel Weinland, Mustafa Ōzuysal, and Pascal Fua. Making action recognition robust to occlusions and viewpoint changes. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part III 11, pages 635-648. Springer, 2010.
[104] Zhen Xing, Qi Dai, Han Hu, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang. Syformer: Semi-supervised video transformer for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18816-18826, 2023.
[105] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8121-8130, 2022.
[106] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[107] Daniel LK Yamins, Ha Hong, Charles Cadieu, Ethan A Solomon, Darren Seibert, and James DiCarlo. Performanceoopimmized hicrarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111223):819-8624, 2014.
[108] Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7177-7188, 2021.
[109] Quanzeng You and Hao Jiang. Action4d: Online action recognition in the crowd and clutter. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1857-11866, 2019.
[110] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032. 2019.
[111] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. Advances in Neural Information Processing Systems, 36, 2024.
[112] Mengmi Zhang, Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Gabriel Kreiman. Finding any waldo with zero-shot invariant and efficient visual search. Nature communications, 91):33730 2018.
[113] Sipeng Zheng, Shizhe Chen, and Qin Jin. Few-shot action recognition with hierarchical matching and contrastive learning. In European Conference on Computer Vision, pages 297-313. Springer, 2022.
[114) Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
17

---

A Biological Motion Perception (BMP) Dataset
A.1 Example Videos in the BMP Dataset
We provide one example RGB video and its corresponding BMP videos under different conditions where the subject is performing a "sit down" action. The naming convention of these BMP videos follows the same as in Sec. 3.1 See the example video at the link
A.2 Action Classes in the BMP Dataset
There are 10 action classes in the BMP dataset: pick up, throw, sit down, stand up, kick something, jump up, point to something, nod head/bow, falling down and arm circles.
A.3 More Implementation Details of the BMP Dataset
In Sec. 3.1 we vary the amount of visual information based on the number of point lights on Joint videos. generate Joint videos, Alphapose (26) is used as the tool to detect the joints of a human body from RGB videos. After that, we use these detected joints to generate BMP stimulus on point-light displays. The specific positions light points correspond to certain joint locations:
J-26P:
point each on the head, nose, jaw, and abdomen, point each on one eye, one ear, one shoulder, one elbow, one hand, one hip and one knee, and points on two feet with points on each foot;
J-18P:
point each on the head, nose, jaw, and abdomen, point each on one ear, one shoulder, one elbow, one hand, one hip, one knee and one ankle:
J-14P: 1 point each the nose, and abdomen, 1 point each one shoulder, one elbow, one hand, one hip, one knee and one ankle;
J-10P: point each on the nose, and abdomen, point each on one shoulder, one hand, one hip and one ankle;
.J-6P:
point each on the nose, and abdomen, point each on one hand and one ankle;
J-5P: 1 point each on the nose, point each on one hand and one ankle.
We also looked into the invariance property to camera views in Sec. 3.1 The video clips in J-6P are categorized based on the viewpoints in the videos: the frontal view, 45" view and 90 view, which respectively labelled as J-6P-0V, J-6P-45V and J-6P-90V in short form. 45" view and 90 view are rotated either clockwise or counterclockwise from the frontal view.
B Mathematical Formulations of Flow Snapshot Neurons
B.1 Slot Attention Module
As explained in Sec. 2.2. the slot attention module aims to obtain flow snapshot neurons
2 E K to capture prototypical moments from the patch-level optical flow Ô E R where
$ = T N and D = 2 x (T - 1). Formally, based on the cross-attention mechanism 198), slots Ż serve the query, Ô contribute to both the key and the value, and q(-), ((-), v(()) are the linear transformation employed to project inputs and slots into a common dimension B, which can be formulated as:
where J:= J:=- JD M (()T E RSS
1 MO)
attni =
(3)
K
tha
h:= (=(U..(()) (r(0)RRKX) where U1
i.j
Et= attn LJ
It is worth noting that the superscript T is the transpose function, and the attention matrix U is normalized make sure that no parts of the input are overlooked. Next, following the work (63
on slot attention, the slots are iteratively refined recurrently based on the Gated Recurrent Unit (GRU) [3 to maintain smooth update. Let 2o represent the initial slot base, with parameters
18

---

What is the action in this video?
pick up
kick something
throw
jump up
arm circles
point to something
sit down
head/bow
stand up
falling down
Figure S1: Schematic of human psychophysies experiments on Amazon Mechanical Turk (MTurk). every trial, subjects are presented with a video and list of ten options. After the video plays only once, the subject has to choose one action among ten options that best describe the action in the video.
initialized by randomly sampling from the Xavier uniform distribution 35 We denote the hidden state time step as hp. and slots at time step p and p + as Zp and 7pt1. Next, we get:
2o+1
GRU (Zp: (2p.
(5)
After P iterations, the final slot features are represented as 2 = 2p. We empirically set P 3
B.2 Contrastive Walk Loss
To encourage diversity in prototypical optical flow patterns in 2, we use the contrastive loss between Ô and / introduced in [102..
Loiot (2,0
(6)
where Q(a.t) is defined as the normalized pairwise feature similarities between a and b Sec. 2.1 We the temperature μ of 0.05 in Q(2,0) and Q(0.2)- CE(.;) stands for the cross-entropy loss and I E IR KxK is the identity matrix. The effect of is covered in Appendix, Tab. S4
C More Training Details
Our model is trained on the training sets of RGB videos in BMP, NTU RGBD+60 and NW-UCLA datasets respectively. BMP: Videos resized to 224 398 pixels for 50 epochs training with batch size of 48. NTU RGB+D 60: Videos are resized to 224 x 398 pixels and trained for 100 epochs with batch size of 128. NW-UCLA: Videos resized to 224 300 pixels and trained for 100 epochs using a batch size of 16.
D Human Psychophysics Experiments
Fig. S1 demonstrates the schematic of human psychophysics experiments on Amazon Mechanical
Turk Turkk) g We recruited a total 90 human subjects. We collect data from a total of 12,600 trials.
19

---

Top-1 1 Acc. (%)
# Trainable Parameters (M)
RGB
J-6P
SP-8P-ILT
ResNet3D
31.7
SlowFast
33.7
E2S-X3D 42
98.7
10.2
10.7
I3D
27.2
X3D
3.0
VideoMAE 194
90.0
9.9
9.9
R(2+1)D
27.3
MVIT
36.1
TwoStream- CNNN [90]
97.0
15.7
10.4
MP(ours)
57.5
MP(ours)
96.5
69.0
49.7
Table S1: Results of more baselines and our motion perceiver (MP) in BMP tasks. Our MP demonstrates superior performance than baselines on RGB videos, Joint videos with 6 points (J-6P), and SP videos with 8 points and a lifetime of 1 (SP-8P-11.) Top-1 accuracy (%) is reported. Best is in bold.
Table S2: Number of trainable parameters million (M) for baselines and our MP. Although our model is larger than the baselines, its superior performance is not result of the larger number of parameters.
In each trial, participants are shown video randomly selected from the BMP dataset and are then asked to perform a forced 1-out-of- 10 choice test to identify the action in the video. Videos from all conditions are randomly selected and presented in random order. The stimuli are uniformly distributed across BMP conditions, with no long-tailed distribution. These action classes are commonly performed in our daily life and free from cultural bias, as psychophysics experiments show humans can recognize them with nearly 100% accuracy on RGB videos. Almost all subjects are from the US, and did not collect demographic data. All the experiments are conducted with the subjects' informed consent and according to protocols approved by the Institutional Review Board of institution. Each subject was compensated.
To control data quality, two checks are implemented in the experiment: (1) Four pre-selected videos in the RGB condition are used the dummy trials, with two vidcos representing the "sit down" action class and the other two representing the "stand up" action class. Since these four videos are meant for quality controls. There is no ambiguity in classifying the action classes in these four videos. These four dummy trials are randomly dispersed with the rest of the actual trials in one experiment. Participants who fail to recognize correct actions in any of the four videos are excluded for data analysis. 88 out of the total 90 participants passed the tests in the dummy trials. (2) Each participant is allowed to participate in the experiment only once. All the trials in one experiment always unique. This is to prevent the subjects from memorizing the answers from the repeated trials.
E More Baselines Comparisons
Besides the comparison of State-of-the-Art methods discussed in Sec. 3.2 this section offers comparisons with three additional baseline methods. E2-S-X3D 42) is a two-stream architecture processing optical flow and spatial information from RGB frames separately. VideoMAE [94), recent baseline trained on Kinetics in self-supervised learning manner. TwoStream-CN: [90] incorporates the spatial networks trained from static frames and temporal networks learned from multi-frame optical flow. The results in Appendix, Tab, ST indicate that our method significantly surpasses all baselines under the J-6P and SP-8P- ILT conditions, highlighting the superior generalization capability of our MP model on the BMP dataset.
we explore the upper bound performance of MP on Joint and SP videos. First, we directly train our MP model on J-6P and test it on J-6P. Its accuracy on J-6P is 95% whereas human performance is 86%. Surprisingly. we found this model trained on J-6P also achieves 55% accuracy in RGB and 71% in SP-SP-ILT, which are far above chance. This implies that our model has generalization ability across multiple modalities. Second, we also train our model on SP-8P-ILT and test it on all three modalities: 43% in RGB, 69% in J-6P, and 93% in SP-8P-1LT The reasonings and conclusions stay the same as the model directly trained on J-6P. Note that although our model achieves very high accuracy on the in-domain test set (train on J-6P, test on J-6P and train on SP-SP-ILT, test on SP-8P-ILT)
Moreover, as stated in Sec. 3.2
overall performance over all three modalities (RGB, J-6P, and SP-SP-1LTT) is still lower than humans (74% vs 88%). This emphasises the importance of studying model generalisation in BMP. There is still performance gap between Al models and humans in BMP.
20

---

Enhanced Motion Perceiver (En-MP)
is worth noting that our current MP model (see Sec. is applied solely the feature maps from the final attention block of DINO (block 12). We have observed that our MP model can also be effectively applied early and intermediate-level blocks of DINO. The final prediction is then generated by fusing features across these three blocks (blocks 1, 7, and 12). We refer to this improved model as the Enhanced Motion Perceiver (En-MP).
G Viewpoint Discrepancy
As mentioned in Sec. 4.1 human behaviors in our study reveal a divergent trend compared to the human performance pattern described in [46]. We point out two reasons: first, the tasks differ between our study and [46]. Our study focuses on action recognition while the work [46) focuses walking pattern discrimination. Second, we found the performance variations across different camera views depend on the action classes. As shown in Appendix, Fig. S2 humans exhibit the best performance when viewing "arm circles" actions from the frontal view, because the movement of two arms will not overlap in the frontal view. For the videos from the action class "point something", the view is the best viewpoint since it results in longer movement trajectories the arm than other viewpoints. Therefore, although humans achieve the best average accuracy on the 45% viewpoint in our experiment, does not imply that humans always perform the best on the 45 viewpoint in all action classes.
100
Frontal view
45 view
90" view
Top-1 Accuracy (%)
80
60
40
20
throw
sit down
stand up
kick something
jump up
point to something
nod head/bow
falling down
arm circles
pick
Figure S2: Human performance across camera views within different action classes on J-6P videos in BMP dataset. The labels on the x-axis are the 10 action classes in the BMP dataset (Appendix, Sec. A.2).
H Alignment Between Models and Humans
The alignment between models and humans can be assessed in three aspects: (1) The correlation coefficients between all the Al models and human performance across all the BMP conditions. The results in Fig, [6 show that our MP has the highest correlation with human performance compared with baselines. (2) The absolute accuracy in action classes across BMP properties, which reported in Appendix, Fig. S3 We averaged the accuracies within the same property. It is observed that all AI models demonstrated lower accuracy than humans. However, among all models, our MP and En-MP outperform the rest. For example, despite explicitly modelling both motion and visual
21

---

100
0.30
ResNet3D
13D
*
TO
0.25
Model Top-1 1 Accuracy (%)
R(2+1)D
TR
89
SlowFast
O
LVI
Error Consistency
0.20
X3D
ICV
MVIT
MP (Ours)
0.15
En-MP (Ours)
0.10
0.05
20
0.00
ResNet3D
13D
R(2+1)D
SlowFast
X3D
MVIT
MP (Ours)
En-MP (Ours)
0
20
100
Human Top-1 Accuracy (%)
Figure S3: Our MP and En-MP significantly surpass all existing models both Joint and SP videos. A correlation plot between model and human performances is provided, with accuracy averaged over conditions within each property on the same type of stimulus (Sec. 3.1). The black dash diagonal serves as a reference.
Figure S4: Our MP and En-MP show higher error consistency with human performance compared to all existing models. The Error Pattern Consistency 31 between each AT model and human performance across all BMP conditions is reported.
information in two separate streams, the SlowFast model still falls short in temporal orders. Similarly, MVIT, which utilizes transformer architectures, fails to generalize to various temporal resolutions. See Appendix, Tab. [S6 for detailed comparisons. (3) The enror pattern consistency between AI models and humans using the metric introduced in (3I] The results Appendix, Fig. [4 reveal that the error patterns from our MP and En-MP are more consistent with human error patterns than all the baselines at the trial level.
Baselines with Optical Flow as Input
As mentioned in Sec. 4.2 the MP model explicitly uses patch-level optical flow to extract temporal information for classification, whereas the baselines do not. Thus, here we introduce augmented baselines where they are trained on pixel-level optical flows. Specifically, we use GMFlow [105] (16]
to extract the pixel-level optical flow from RGB videos in the BMP dataset, and then convert them into three-channel videos based the Middlebury color code (5). Taking these optical flow videos in the training set as the input to baselines. the performances on optical flow videos the testing set and J-6P are shown in Appendix, Tab. S3 It is evident that all baselines achieve high accuracy on optical flow videos in the testing set, indicating that they effectively learn to recognize actions from optical flow inputs. However, their best performances on J-6P significantly worse compared to the MP model (30.83% vs 69.00% in Appendix, Tab. (ab. suggesting that our MP learns more generalizable motion representations from patch-level optical flows.
ResNet3D 38
I3D II
R(2+1)D [96]
SlowFast 29
X3D
28
MViT 25
OF 99.23
99.02
98.67
99.12
98.35 9.87
99.19 30.83
J-6P
9.94
9.87
9.38
9.90
Table S3: Results of baselines with optical flow input. Top-1 accuracy reported on optical flow videos (OF) and Joint videos with 6 points (J-6P). Best in bold.
J Visualization of Patch-Level Optical Flow
As outlined in Sec. 4.2 we provide visualization of patch-level optical flow in vector field plots across example video frames for "stand up" action in Appendix, Fig. S5 We can see that patch-level
22

---

optical flow mostly happens in moving objects (the person performing the action) and captures high-level semantic features. Hence, they are more robust to perturbations in the pixel levels and more compute-efficiet.
(a)
(b)
(c)
(d)
Figure S5: Visualization of patch-level optical flow in vector field plots across sequence of example video frames from "stand up" action class in the BMP dataset. Optical flow from (a) Frame I; (b) Frame 8; (c) Frame 16; (d) Frame 31 to Frame 32.
K More Ablation Studies
To investigate the impact of additional hyperparameters and key components in our model, we have conducted further ablation studies. Note that we use the same set of hyper-parameters for our model across all the datasets.
K
Q
T
Top-1 Acc. (%)
Full Model
8
10
1
100
0.10
0.01
0.02
0.20
RGB
95.40
96.24
96.14
96.66
96.21
98.17
96.84
96.77
96.35
96.45
J-6P
67.77
68.96
68.15
66.92
66.57
40.27
22.19
68.29
67.63
69.00
SP-SP-1LT
45.79
48.77
46.35
47.51
45.08
26.44
18.96
47.09
46.70
49.68
Table $4: Ablation of key hyper-parameters in our model. Top-1 accuracy is reported on RGB videos, Joint videos with 6 points (J-6P), and SP videos with 8 points and a lifetime of (SP-8P-1LT), ILT). From left right, the ablated hyper-paramett
are: the number of slots K (Sec. (Sec. the weight of Lator (Sec. 2.4) temperature T in the patch-level optical flow (Sec. 2.1 and temperature in Lalot (Appendix, Sec. B.2 Best is in bold.
Temperature in the patch-level optical flow: Appendix, Tab. S4 demonstrates that higher temperature r used for computing patch-level optical flows hurt performances, compared with our full model where of flows and lower temperature T is beneficial for clustering flows with similar movement patterns.
0.001. This implies that temperature controls the smoothness of the sequence
Number of slots K: As shown in Appendix, Tab. S4 there is non- mon-monoto performance trend versus the number of slots. The number of slots controls the diversity of temporal information captured from the lath-level optical flow. An increase in the number of temporal slots lead to redundant temporal clues, while a decrease might result in a lack of sufficient temporal clues.
The weight of Tssto:: The parameter a regulates the importance of Tistot in comparison to other loss functions. A small value of a can impede the diversity in temporal slots, while a large a might adversely affect the optimization of features extracted from flow snapshot neurons and motion invariant neurons. From Appendix, Tab. SS4, it can be seen that when a is either small (a = 1) or large (a 100), there will be slight degradation in performance compared with our full model with a a= 10.
Temperature μ in Lsiot: The temperature controls the sharpness of distribution over slots. High temperatures will negatively impact the convergence of slot feature extraction, while low temperatures can produce excessively sharp distributions, potentially impairing the stability of the optimization process during training. Appendix, Tab. (54 indicates that configuring μ at either 0.02 or 0.20 are non-optimized options compared with our full model when μ is 0.05.
23

---

Top -1 Acc. (%)
Ablation ID
Time Emb.
Elot
1st Frame Ref.
RGB
J-6P
SP-8P-1L1
1
95.19
68.12
47.12
2
X
X
96.52
67.84
49.12
V
V
3
91.40
58.60
47.16
Full Model
X
96.45
69.00
49.68
Table S5: Ablation of the time embedding (Sec. 2.4). the loss term Lstot (Sec. 2.2 and the referenced frame for calculating path-level optical flow (51e. 1 in our model. Top-1 accuracy is reported on RGB videos, Joint videos with 6 points (J-6P), and SP videos with 8 points and a lifetime of (SP-SP-1LT). Best is in bold.
Model components: The ablation of the time embedding (Sec. 2.4 and the loss term Lotot (Sec. 2.2 in our model is shown in Appendix, Tab. SS As illustrated in Sec. 2.4 time embeddings are appended to the activations flow snapshot neurons. Aligning with [66], time embeddings provide useful temporal context (A1). The loss Lstor on contrastive walks encourages the slots to capture diverse prototypical flow patterns. Consistent with [022, removal of Lotot (A2) leads to degraded generalisation performance, indicating the importance of contrastive walk loss between slots and optical flows. Furthermore, our MP relies solely on the first frame as the reference for computing optical flow (A3), the performance will degrade significantly. This is because optical flows estimated by computing the similarity between feature maps from video frames. The errors in feature similarity matching might be carried over in computing optical flows. Using multiple frames as references for computing optical flows eliminates such errors.
We also introduce an MP variation using pixel-level optical flow downscaled to the size of patch-level optical flow as input. Our MP model outperforms this model variation: 96.5% vs 68.8% in RGB, 69.0% vs 12.6% in J-6P, and 49.7% vs 9.4% in SP-8P-1LTT Hence, this implics that DINO captures semantic features that are more effective and robust for optical flow calculation than downscaled pixel-level flows.
Downscale pixel-level flows:
Feature Extractor DINO with ResNet:In our MP. using ViT-based DINO has demonstrated its effectiveness in biological motion perception. We then conducted an experiment where ViT was replaced with the classical 2D convolutional neural network (2D-CNN) ResNet50, pre-trained on ImageNet, as feature extractor for video frames. The results show that while the performance of our MP with ResNet50 lower than with ViT, it still exceeds chance levels. Specifically, the accuracy of DINO-VIT (ours) versus DINO-Res is: 96.45% vs 80.37% in RGB, 69.00% vs 40.34% in J-6P, and 49.68% vs 40.03% in SP-8P- Additionally, it outperforms baselines using 3D-CNN backbones, such ResNet3D, I3D, and R(2+1)D. This suggests that our MP effectively generalizes in BMP, regardless of the feature extraction backbone used.
Number of Trainable Parameters As mentioned in Sec. 4.3 we present the number of trainable parameters for all baselines and our MP model Appendix, Tab. [52) While our model is larger than the baselines, its superior performance is not attributed to its size. To validate this, We included SlowFast--Pe Rest-R1 variant with 61.9M parameters, which underperforms compared to our model. Specifically, the performance of our model versus SlowFast-Ressell 69.0% vs 39.4% % in J-6P and 49.7% vs 12.6% in SP-8P- -LT. addition, we list the number trainable parameters in millions (M) for each model part of our MP: Flow Snapshot Neuron (0.07M), Motion Invariant Neuron (0M) and Feature Fusion (57.5M). Compared to DINO with 85.8M parameters for image processing, our MP model, appended to DINO, only requires slightly more than half of its size. Yet, it leverages DINO features from static images to generalize to recognize actions from a sequence of video frames.
is 96.5% vs 99.3% in RGB,
Key Frame Analysis As indicated Sec. to explore how the start, end, or development of the action would influence action recognition, we analyze the effect of which frames are essential for the "pick up" action class one example video. Briefly, we randomly selected X frames among 32 frames, duplicated the remaining frames to replace these selected frames, and observed the accuracy drops, where X = 11,22,,,,2,,8831,, When multiple frames are replaced, the performance drop implics the importance of the development of these frames. In total, we performed 1000 times of

---

0.724
Averrgge Acc Accurscy
0.707
10
11
13
14
15
16
17
18
19
20
21
22
23
21
25
20
27
28
29
30
31
32
Framo Index
Figure S6: Frame analysis for the "pick up" action class in one example video from the BMP dataset. The color in the heat map indicates the video frames which would impact the action recognition accuracy of the model the most. Blue colors represent higher importance.
random frame selections per X and presented the visualization of frame importance by averaging all the accuracy drops over all the random frame selections. The visualization results in Appendix, Fig. S6 suggest that the fourth and seventh frames are essential for the pick-up class recognition.
L Detailed Results of All BMP Conditions
The detailed results all BMP conditions are shown in Appendix, Tab, S6 It is evident that, in comparison to baselines, motion perceiver (MP) only delivers enhanced performance but also more accurately mirrors human performance most BMP conditions. Especially in the case of SP-8P-1LTT, model significantly outperforms the top baseline by substantial margin of 29.39%. These findings emphasize that our MP model is more effective at recognizing biological motion, demonstrating greater alignment with human behavioral data than existing AI models.
ResNer3D
38
I3D I
R(2+1)D 96
SlowFast 29
X3D 28
MViT 25
MPu(s
Human
RGB
98.
98.74
99.02
98.63
98.67
99.02
96.45
98.00
RGB-R
70.51
68.82
76.23
62.18
66.50
68.19
62.32
61.83
RGB-S
53.12
41.57
38.76
22.68
41.50
51.65
61.34
76.00
RGB-4F
68.89
76.62
77.91
60.11
75.53
32.44
68.82
88.00
RGB-3F
61.24
62.89
64.61
65.98
64.92
31.88
68.43
84.33
J-26P
27.95
75.35
33.39
77.98
67.21
76.90
72,16
92.50
J-18P
26.12
76.51
34,66
75.74
67.80
75.46
73.17
J-14P
25.77
68.15
35.22
72.30
66.68
74.61
70.37
J-10P
20.75
56.04
26.90
70.54
61.03
68.05
71.73
J-6P
18.54
28.41
18.57
$5.72
48.38
$2.00
69.00
86.33
J-5P
18.19
27.14
17.17
$1.23
40.41
47.86
65.52
86.17
J-6P-R
17.70
23.53
16.26
35,04
31,00
32.37
38.69
$6.67
J-6P-S
26.69
19.56
18.43
24.30
20.93
20.89
32,65
47.83
J.6P.4F
16.19
10.96
15.80
21.70
20.40
12.29
41.12
63.17
J-6P-3F
16.22
13.83
11.27
18.22
17.03
43.71
55.67
J-6P-0V
19.83
31,36
17.86
59,50
$1,30
52.23
70.20
84,00
J-6P-4SV
17.98
30.46
18.73
58.88
53.18
55.11
69.64
91.50
J-6P-90V
17.78
23.43
19.14
48.85
40.79
48.74
67.15
90.83
SP.4P-1LTT
12.96
21.91
10.50
19.10
14.12
10.64
32.76
56.33
SP-4P-2LT
10.57
22.02
15.06
26.09
17.77
18.40
35.01
45.36
SP-4P-I.T
12.64
23.07
19.31
19.77
19.59
27.53
34.76
52.50
IP-
15.59
18.43
10.53
20.29
19.91
15.06
49.68
80.33
SP-SP-2LT 13.45
33.85
17.49
21.00
22.54
31.07
49.33
SP-8P-4LT 17.35
38.97
27.39
21.31
26.23
46.63
50.49
74.47
Table S6: Detailed results of baselines, motion perceiver (MP) and human in all BMP conditions. Top-1 1 accuracy (%) is reported. There are 24 conditions in total. Best is in bold. The second best is underlined.
25

---

M
Social Impact
The Motion Perceiver, with its human-like ability to perceive biological motion, brings both promising advancements and concerning implications for society. In the sports and fitness field, it provides valuable tools for improving athletic performance, injury prevention, and personalized training. However, the same capabilities may raise privacy concerns, as the technology could potentially be misused for unauthorized surveillance or tracking of individuals' movements public spaces. This invasion of privacy might extend to analyzing behavior patterns and gait. Furthermore, there is a risk of discrimination the method is employed contexts like employment screening or law enforcement profiling, where certain movement patterns might be unfairly associated with specific demographics or lead to biased judgments.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction (Sec. clearly state the claims reflecting the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: discuss the limitations of the work in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
There is no theoretical proof in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our model architecture is described fully and clearly in the paper. The steps taken to create the BMP dataset are also described. Data and code have been released publicly.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer:[Yes Yes]
Justification: Data and code have been released publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ 'pipsscc/puullc//guu Sotp:///ggiddeeeeCouuuuubbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper-paramette how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Sec. 2.c.2.4andd and Sec. cover the experiments settings and details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars indicating standard error are added in all the relevant plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The computed resources are reported in Sec. 2.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neurippttttt httpa//wwp.p...p..ao..cccccaacaa
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts of our work in Appendix, Sec. M
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss the risk for misuse of our work in Appendix, Sec. M
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers that produced the code packages datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of the BMP dataset are presented in Sec. 3.1
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The details of crowdsourcing and research with human subjects are presented
in Sec. 3.1 and Appendix, Sec. D
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: IRB approvals obtained.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "eeZqyjlfWd", "reviewer": "Reviewer_nybp", "summary": "This paper proposes a new biologically inspired architecture for action recognition. The motion perceiver computes a patch-level optical flow from DINO features which is then processed in a two-stream architecture with one pathway using slot-attention to recognize different motion patterns and the other one integration over motion to get a time-independent motion signal. The paper also contributes a new dataset which provides multipe point light versions of the videos together with human performance data. The model is trained on RGB videos and evaluated on the point light videos. The authors show that the model's performance is well aligned with human performances on the different stimulus versions and that it outperforms other ML models on the least-information stimuli.", "questions": "* How many trainable parameters has the final model, and how are they distributed over the model parts?\n* given that the DINO based patch-level optical flow seems to be a major pillar of the model, I would love to get a better idea of how it performs on some example videos or images. It could be nice to have some vector field plots in the Appendix or a video in the Supplementary Material\n* Caption Figure 1: \"AI models are trained to learn to recognize actions\": Is this on purpose? I would say they are either trained to recognize actions or they learn to recognize action. Being trained to learn to recognize would imply some meta learning.\n* l98 \"Considering Ft as a 2D grid of patches in N = H × W where H and W are the height and width of Ft \" this sentence seems to be missing a subject\n* l127: \"This dense optical flow estimation approach at the patch level captures a richer set of motion dynamics, providing a more comprehensive analysis of movements throughout the video\" I find it interesting that the patch-level optical flow is defined relative to a fixed reference frame instead of, e.g. always the first frame and then actually uses all frames as reference. Did the authors check how much does this help compared to only using e.g. the first frame? Right now this sentence doesn't seem to have any supporting evidence\n* l158: \" Every patch-level optical flow in Ô can be projected into four orthogonal motion components along +x, −x, +y and −y axes\": I think I'm missing something here, but how are motions in +x and -x orthogonal?\n* l163 \"we obtain motion invariant matrix\": \"we obtain THE motion invariant matrix\"?\n* l270 \" Interestingly, shuffling has a lesser impact on human performance compared to reversing RGB videos (RGB-R versus RGB-S)\" technically, the two performances might be identical within the margin of error if I'm not mistaken.\n* Figure 4: given the size of the error bars in Figure 3, I think it would be nice to also have error bars in this figure. Since it would make the figure harder to read, I think it would be enough to have a larger version in the appendix with error bars that can be referred here.\n* l322: I agree that ablations A1 and A2 show that both pathways are important, but they seem to be hardly of similar importance. Removing the motion invariant pathway results in a drop of performance, while removing the flow snapshot neuron pathway essentially results in model performance collapsing. So I would say that the flow snapshot neuron pathway is the crucial part, and invariance helps further.\n* l329 \"In A4, the model takes patch-level optical flows as inputs and directly utilizes them for feature fusion and action classification without flow snapshot neurons\" Unless I'm missreading the table, A4 has flow shapshot neurons?\n* l349 \"Psychophysics experiments on this dataset were conducted, providing human behavioral data as an upper bound\": Why upper bound? I don't see why models couldn't outperform humans in principle.\n* references to the appendix could be a bit more clear, e.g. \"see Appendix, Sec B\" instead of \"see Sec B\"", "limitations": "Limitations are well discussed in the main paper. Societal impact is not discussed because \"There are no societal impacts, to the best of our knowledge\" (l980), with which I would slightly disagree. There is always potential societal impact from building more human like models (good as well as bad). But this is mostly a nitpicking point that doesn't affect my review of the paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* very interesting study linking neuroscience, psychophysics and deep learning\n* Interesting modelling architecture with multiple relevant and significant contributions:\n  * patch based optical flow on DINO features is a nice idea and seems to work very well\n  * slot attention on the optical flow data is also a nice idea\n  * motion invariant neurons which integrate over motion patterns\n* well-designed dataset\n* convincing and interesting results that are well presented and discussed\n* interesting ablation study with very good discussion that goes beyond a simple table\n* very well written paper with clear structure.", "weaknesses": "* I think at least one relevant comparison is missing: Illic et al, \"Is Appearance Free Action Recognition Possible\" (ECCV 2022, https://f-ilic.github.io/AppearanceFreeActionRecognition) asks a very related question and also uses somewhat similar stimuli (white noise stimuli, but also different kinds of dot stimuli) and also introduce a dataset to that end (AFD). Illic et al claim that their twostream architecture can solve the appearance free stimuli and at least on the homepage they also mention dot stimuli close to the ones used in the present paper. However, so far I couldn't find any dot stimuli results in the actual paper. Beyond this paper, I think there are more two-stream architecture models that might be relevant to include. This is overall the main reason that kept me from increasing the rating of the paper.\n* the claim \"Our model demonstrates human-like robustness to minimal visual information\" (Figure 4) seems to strong for me. For most of the stimulus types, MP is not even the best competitor, only for J-6P and J5-P it outperforms all other models. But even then the performance drop compared to RGB seems to be 3 times more than for humans. The same holds for l282 \"There is a slight decrease in accuracy when transitioning from RGB to J-6P for both humans and MP\", where humans seem to be around 90% and MP around 70%. Interestingly, Figure 4 suggests that most of the performance drop comes from going from RGB to the point light display, because the drop happens already at J-26P, after which the performans of MP mostly stays constant.\n* I'm missing more discussion of when and where \"classical AI models fail or succeed and the alignment with humans\". I think there is a bit to be learned from Appendix F. Especially, I would find it interesting to compute corellations with human scores across the different stimuli. Figure 3 suggests that MP is well aligned with human BMP. But I would like to know if it is also more aligned than other models (which I think would be a very strong additional result).\n* notation sometimes a bit convoluted (M, \\hat M, \\tilde M, ...). Maybe sometimes it's worth using more descriptive names like \"M_\\text{invariant}\"\n\n\n**Update**: After the rebuttal, I increased my review from 7 to 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[nybp.1 - missing AFD and two-stream models]** Thanks! We will cite and discuss this paper in the final version. Complementary to the AFD dataset in Illic et al., we introduce the BMP dataset containing stimuli on point-light displays, also commonly studied in psychology and neuroscience. \n\nIn addition to the two-stream SlowFast baseline already included in our paper, we add one more two-stream network E2-S-X3D in Illic et al. for comparison on our BMP dataset. It underperforms our MP model by a large margin (see Tab R1). This demonstrates the importance of FSN and MIN in our model. \n\n**[nybp.2 - Fig4 and claim in l282]** Yes, the reviewer is correct. Note that our current MP is only applied on the feature maps from the last attention block of DINO (block 12). We found that our MP model can also be applied across early and middle-level blocks of DINO. The final prediction is made based on the feature fusion across these three blocks (blocks 1, 7 and 12), with only a 7.9% accuracy difference on the J-6P compared to humans (see Tab R3). It also outperforms the second-best MViT baseline across all the number of joints. We will add this enhanced MP model in the final version and revise the claims in Fig 4 and l282.\n\n**[nybp.3 - alignment and correlation]** The alignment between models and humans can be assessed in three aspects: (1) The absolute accuracy in all action classes across all BMP conditions, which is reported in Sec 4.2 and Fig 6. The slope of close to 1 in our model in Fig 6 suggests that our model performs competitively well as humans in all five properties in BMP. (2) The correlation values for all the models across all the BMP conditions. The results in Tab R5 show that our MP has the highest correlation with human performance compared with baselines. (3) The error pattern consistency between models and humans using the metric introduced in [p]. The results in Tab R6 indicate that our model achieves the highest error consistency with humans at the trial level. \n\n**[nybp.4 - notations]** We will make the notations clearer in the final version.\n\n**[nybp.5 - model parameters]** We list the number of trainable parameters in million (M) for each model part of our MP: FlowSnapshot Neuron (0.07M), Motion Invariant Neuron (0M), Feature Fusion (57.5M), \nCompared to DINO with 85.8 million (M) parameters for image processing, our MP model, appended to DINO, only requires slightly more than half of its size. Yet, it leverages DINO features from static images to generalize to recognize actions from a sequence of video frames. \n\n**[nybp.6 - visualization of optical flow]** See Fig R2 in the rebuttal PDF for a visualization of patch-level optical flow in vector field plots across example video frames for “stand up” action. We can see that patch-level optical flow mostly happens in moving objects (the person performing the action) and captures high-level semantic features. Hence, they are more robust to perturbations in the pixel levels and more compute-efficient. \n\n**[nybp.7 - minor in language]** Thanks. We will fix it:\" AI models are trained to recognize actions\".\n\n**[nybp.8 - minor in language]** Thanks. We will fix the grammar mistakes.\n\n**[nybp.9 - first frame as reference frame]** As the review suggested, we add an ablation study where our MP only uses the first frame as the reference frame to compute optical flow. Our MP significantly outperforms this ablated version. Top-1 accuracy for MP and ablated MP are: 96.5% vs 91.4% in RGB; 69% vs 58.7% in J-6P, and 49.7% vs 47.2% in SP-8P-1LT. Optical flows are estimated by computing the similarity between feature maps from video frames. The errors in feature similarity matching might be carried over in computing optical flows. Using multiple frames as references for computing optical flows eliminates such errors.  \n\n**[nybp.10 - Orthogonal motion]** Thanks. We will remove “orthogonal”. Briefly, an optical flow vector can be decomposed into x and y axes. For example, the optical flow vector (3,5) has a magnitude of 3 on the +x axis and 5 on the +y axis, while a magnitude of 0 along the -x and -y axis.\n\n**[nybp.11 - Typo]** Thanks. We will fix it. \n\n**[nybp.12 - Statistical test]** Thanks. We performed the statistical tests (two-tailed t-test) in human performance between RGB-R and RGB-S. The p-value is 0.465, above 0.05. This implies that the accuracy of shuffling frames is not statistically different from reversing frames. We will revise the claim in l270. Moreover, we will perform statistical tests and report p-values for other result comparisons in the final version. \n\n**[nybp.13 - Error bars in Fig4]** Thanks. We will add the enlarged version of Fig 4 in the Appendix with error bars included. \n\n**[nybp.14 - Effect of Motion Invaraint Neurons]** We agree with the reviewer that the effect of Motion Invariant neurons is not well reflected in this ablation study. However, its effect is much more prominent when video frames are shuffled or reversed. For example, our model outperforms its ablated model without MIN in the following experiments: 62.3% vs 49.8% in RGB-R; 61.3% vs 38.0% in RGB-S, 38.7% vs 36.1% in J-6P-R, and 32.7% vs 25.5% in J-6P-S. We will emphasize this point in the final version.\n\n**[nybp.15 - A4 in Table 2]** Yes, the reviewer is correct. There is no flow snapshot neuron. We will fix it in A4, Tab 2.\n\n**[nybp.16 - Humans as upper bound]** We acknowledge that AI models can outperform humans in many tasks. However, in our BMP tasks, we argue that current AI models for action recognition remain inferior to humans across many BMP conditions, as shown in our experiments. We will clarify this point in the final version.\n\n**[nybp.17 - References]** Thanks. We will change them.\n\n**[nybp.18 - Societal impact]** Thanks. In the final version, we will expand our discussion on the societal impacts. This includes positive impacts, such as in sports and fitness; and negative impacts, such as privacy invasion and discrimination.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6IRoGqDnHC", "reviewer": "Reviewer_buWx", "summary": "The paper introduces the Motion Perceiver (MP), a novel AI model designed to improve the generalization of action recognition in biological motion perception (BMP) tasks. It leverages patch-level optical flows and introduces flow snapshot neurons that learn and store prototypical motion patterns and motion-invariant neurons that maintain the consistency of motion recognition across temporal sequences. The authors create a comprehensive BMP benchmark dataset with 62,656 video stimuli across 24 BMP conditions and demonstrate that MP surpasses existing models by 29% in top-1 action recognition accuracy. The MP model’s performance closely aligns with human behavior in BMP tasks, providing significant insights into the underlying mechanisms of BMP and paving the way for more generalizable AI models in action recognition.", "questions": "Most of my questions are described in Weaknesses. Some additional or specific ones:\n\n1. What's the runtime of the proposed method compared to baselines?\n2. Have you tried using the pixel-level optical flow downscaled to patch level and compared that to the patch flow?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: The Motion Perceiver (MP) model, which uses patch-level optical flows and novel neuron mechanisms (flow snapshot neurons and motion-invariant neurons), is a creative and original approach to improving action recognition in BMP tasks. The BMP benchmark dataset is also a significant contribution that provides a new standard for evaluating both human and AI performance in BMP tasks.\n\n- Quality: The paper demonstrates a well-designed and rigorous methodology, including detailed descriptions of the MP model architecture, the BMP dataset, and the experimental setup. The extensive evaluation of the MP model against existing AI models and its comparison with human performance provide strong evidence supporting the paper’s claims. \n\n- Clarity: The paper is well-written and clearly explains the technical details of the MP model and the experimental methodology. Figures and tables are effectively used to illustrate the key concepts and results. The paper also situates its contributions within the broader context of existing research in AI and motion perception, providing a clear understanding of the novelty and significance of the work.", "weaknesses": "- Limited Contextualization: Although the paper does a good job explaining the MP model and its benefits, it could benefit from a more detailed comparison with prior work. Specifically, a deeper discussion on how the proposed method improves upon existing models in terms of architectural innovations and performance metrics would provide clearer context. Particularly, lines 56-57 state that “In contrast to many existing pixel-level optical flow models [83, 84, 79, 91, 71], ….”, but some of the references are actually representation-level flow models, e.g., [71]. More comprehensive comparisons to those prior works (theoretically and/or empirically) would make the claim more convincing.\n\n- Dataset Limitations and Biases: The paper introduces a comprehensive BMP benchmark dataset, but it would be beneficial to address potential limitations and biases within this dataset. For example, discussing the diversity of the actions, the representation of different demographics, and the potential impact of these factors on the model’s performance would provide a more balanced view of the dataset’s strengths and limitations. Most importantly, the authors designed two scenarios (joint videos and sequential position actor videos), and claim they are good benchmarks for recognizing motion patterns. This claim is not well justified. For example, why not use sketches, optical flows, or other types of motion representation?\n\n- Results Not Supportive Enough: In Fig. 4, it seems that the proposed method only outperforms on J-6P and J-5P, and is inferior on other levels. In this case, it is hard to claim that the proposed method has better generalization. While the J-6P and J-5P are most abstract and challenging, there is no evidence that those two levels are the best evaluators for generalization on action recognition. Additionally, the comparison methods are mostly dated; more recent baselines are needed. There is no mention/control on the efficiency and computation cost as well, making the comparisons less meaningful.\n\n- Contribution: The biggest weakness is the overall contribution of the paper. While the paper is strong in proposing a new neural architecture and a benchmark, the proposed task itself is not justified with theoretical or practical value. There is no valid proof that joint and SP videos are good representations of biological motion (compared to other motion representations). More specifically, the joints are from the skeleton and keypoints of a human body, which limits its generalization to other objects. Regarding human action recognition itself, it is plausible that having a keypoint modality in the training procedure could potentially solve the problem of distribution shift (RGB to abstract motion representation). While the idea is well-motivated, the approach does not strongly support the claim; the methodology, the proposed benchmark, and empirical results will all benefit a lot from further study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[buWx.1-More comparisons to prior works]** Thanks. We will remove [71] from this sentence in the final version. Moreover, we added two more new baselines [a] and [c]. In [a], E2-S-X3D is a two-stream architecture processing optical flow and spatial information from RGB frames separately. In [c], TwoStream CNN incorporates the spatial networks trained from static frames and temporal networks learned from multi-frame optical flow. Results in Tab R1 show that our method outperforms both baselines, demonstrating the superior generalization ability of our MP model on the BMP dataset.\n\n**[buWx.2-Dataset limitation and bias]** We introduced the 10 action classes in Appendix A1.1 and explained our dataset split in Sec 3.1. Stimuli are uniformly distributed across BMP conditions, with no long-tailed distribution in training or testing. These action classes are common and free from cultural bias, as psychophysics experiments show humans can recognize them with nearly 100% accuracy. Almost all subjects are from the US, and we did not collect demographic data. We are open to testing potential biases if specified.\n\nTab R4 shows human and model performance in per-category accuracy. We observed variations across BMP conditions. For example, both humans and our model have low accuracy for the stand-up class in the TO property, as altering frame order can confuse it with actions like sitting down or falling down. \n\nWe benchmarked humans and models using Joint and SP videos, which are established stimuli for studying human motion perception in vision science, psychology, and neuroscience. We followed the designs from notable works [30,82,9,46,6,16,34,40,64,69,89] in the References to leverage their discoveries and compare our findings.\n\nWe are unclear about the specific \"sketches\" the reviewer refers to. Studies [i-k] indicate that current action recognition models often rely on static features.  Point-light displays reduce these confounding factors by minimizing visual information, thereby highlighting the ability to perceive motion. The reviewer also suggested using optical flows as stimuli, but their suitability for studying motion perception is unclear due to potential colour-based confounding. \n\nFinally, we do not claim Joint or SP videos with point-light displays are the only stimuli for studying biological motion perception. Like CIFAR-10's role in object recognition decades ago, our work introduces initial datasets for biological motion perception, encouraging community contributions.\n\n**[buWx.3-Results not supportive enough]** Yes, the reviewer correctly points out our model's performance in Fig 4. However, Fig 4 shows only one aspect of generalization in Joint videos. Tab S3 shows our model beats baselines in 17 of 24 BMP properties, with some gaps as large as 30%.\n\nOur current MP model is applied only to feature maps from DINO's last attention block (block 12). We found it can also be applied across early and middle blocks (e.g., blocks 1, 7 and 12), with final predictions based on feature fusion across these blocks. As shown in Tab R3, it outperforms the second-best MViT baseline across all Joint cases. We will add this enhanced MP model in the final version.\n\nAs requested by the reviewer, we added three more baselines. Two are introduced in **[buWx.1 - More comparisons to prior works]**. Additionally, we added VideoMAE [b], a recent baseline trained on Kinetics in a self-supervised learning manner. Our model also significantly outperforms VideoMAE, as shown in Tab R1.\n\nWe also added Tab R2, listing the number of trainable parameters for all the models. Our model size is larger than baselines, but its superior performance is not due to size. We included a SlowFast-ResNet101 variation with 61.9M parameters, which performs worse than ours. The performance of ours versus SlowFast-ResNet101 are: 96.5% vs 99.3% in RGB, 69.0% vs 39.4% in J-6P, and 49.7% vs 12.6% in SP-8P-1LT. We also discussed runtime efficiency comparisons for all models. See **[buWx.5-Runtime comparison]** for more details.\n\n**[buWx.4-Theoretical and practical value]** The stimulus design in our BMP experiments is inspired by well-established works in vision science, psychology, and neuroscience. See more details in **[buWx.2-Dataset limitation and bias]**. We do not claim that joint videos and SP videos are the only ways to study the generalization of motion perception. There are definitely other types of stimuli testing the generalization of motion perception for other objects, such as random noise animated with motion extracted from real objects shown in [a]. \n\nThe reviewer suggested training the models on RGB videos with the keypoint modality, but this would defeat our study's purpose. We argue that a model's generalization ability should arise from well-designed architectures, visual inputs, and learning rules. While data augmentation is an effective engineering approach in computer vision [d-h], we aim to develop AI models that generalize to BMP stimuli as humans do, without augmenting visual inputs during training.    \n\n**[buWx.5-Runtime comparison]** All the baselines are pre-trained on Kinetics 400, whereas the DINO architecture in our MP is pre-trained on ImageNet. This difference makes fair runtime comparisons challenging. As expected, our MP has a longer runtime than the baselines due to the pre-trained dataset differences. Nonetheless, it is noteworthy that our MP outperforms all baselines despite not being pre-trained on any video datasets. \n\n**[buWx.6-Downscale pixel-level flows]** As suggested by the reviewer, we conducted an MP variation using pixel-level optical flow downscaled to the size of patch-level optical flow as input. Our MP model outperforms this model variation: 96.5% vs 68.8% in RGB, 69.0% vs 12.6% in J-6P, and 49.7% vs 9.4% in SP-8P-1LT. This implies that DINO captures semantic features that are more effective and robust for optical flow calculation than downscaled pixel levels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sk6pOfk2Bm", "reviewer": "Reviewer_WFYf", "summary": "The paper introduces a novel, neuroscience-inspired approach to performing biological motion perception from videos. The videos are part of a dataset that the authors also introduce, depicting 10 different actions. The videos range from fully RGB frames to point-light displays that only cover the joints. While humans have been very successful at solving such tasks, even with limited data, AI models struggle. For this reason, the authors propose an elegant approach that involves obtaining optical flow from feature maps of an ImageNet-pretrained network and creating a system of slots to organize important features for the task. Additionally, they introduce features that account for scale and invariance. Overall, the method shows significant improvement over the tested baselines.", "questions": "I add them in the weakness section.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel dataset designed to advance the field of action recognition in videos. This benchmark dataset challenges most of the baseline methods in the field. Additionally, the paper presents an elegant approach to combining features over time, inspired by loosely but interestingly applied neuroscience insights. This approach appears to effectively motivate a method that performs better under almost all conditions of the proposed dataset.", "weaknesses": "My main question/concern is whether the baselines are fairly compared with the proposed method. It might be worth adding learning curves to indicate whether the models have effectively converged. Additionally, including a parameter count to express whether the method is more or less computationally expensive than the baselines would provide useful insight.\n\nI might have missed it but it seems that the inspiration to take  ViT as a feature extraction is not well motivated. Perhaps worth mentioning if there is some rationale that  harmonizes with neuroscience or if its a engineering choice. \n\nAnother question/concern is that the connection back to neuroscience seems weak. However, there might be a potential opportunity arising from this approach. The ablation studies suggest a certain dependency on time resolution. It might be beneficial to show which frame in the sequence is the most important and how the start, end, or development of the action is crucial for action detection. This could tie back to neuroscience and provide testable hypotheses, thereby strengthening the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[WFYf.1 - Fair comparison and learning curves]** All the baselines are action recognition models pre-trained on Kinetics 400. In contrast, the DINO architecture in our MP is pre-trained on ImageNet. All the models are then fine-tuned on RGB videos in the training set of our BMP dataset. We would like to emphasize that it is remarkable that our MP can still outperform all the baselines even though it has never been pre-trained on any video datasets.\n\nAs requested by the reviewer, we present the learning curves of our MP and MVIT: the loss of our MP converges to near zero after around 7000 iterations with a top-1 accuracy of 97% in the validation set. Similarly, the loss of MVIT converges to near zero after around 1250 iterations with top-1 accuracy of around 97% as well. We note that MViT converges faster than our MP due to the difference in the pre-trained datasets. Both models achieve near-perfect performance in RGB videos; however, our model significantly surpasses MViT in joint videos and SP videos as shown in Table S3 in Appendix F. This emphasizes that our MP is better at generalization in BMP.   \n\n**[WFYf.2 - Parameter count]** As also requested by the reviewer, we now add Tab R2 listing the number of trainable parameters for all the models. Note that our model is larger than the baselines in the paper. However, we argue that the superior performance of our model is not a result of the larger number of parameters. To demonstrate this, we included a variation of SlowFast-ResNet101 containing 61.9M parameters. Despite its large size, its generalisation performance in BMP is still inferior to ours. The performance of ours versus SlowFast-ResNet101 are: 96.45% vs 99.26% in RGB, 69.00% vs 39.43% in J-6P, and 49.68% vs 12.64% in SP-8P-1LT.\n\n**[WFYf.3 - Motivation of ViT]** The reviewer is correct. We used ViT because it is one of the state-of-the-art architectures for image processing with the best performances in several computer vision tasks. It is purely an engineering choice.\n\nFollowing up on the reviewer’s advice, we also conducted an experiment where we replaced ViT with the classical 2D-convolutional neural network (2D-CNN) ResNet50 pre-trained on ImageNet as a feature extractor from video frames. Evidence in neuroscience [l-o] has suggested that 2D-CNN models are bio-plausible models capable of predicting behaviours and neural responses in primates. Results show that the performance of our MP with ResNet50 is lower than our original model with ViT but still higher above chance. The accuracy of DINO-ViT (ours) versus DINO-ResNet50 are: 96.45% vs 80.37% in RGB, 69.00% vs 40.34% in J-6P, and 49.68% vs 40.03% in SP-8P-1LT. Moreover, it outperforms the baselines with 3D-CNN as backbones, such as ResNet3D, I3D and R(2+1)D. This suggests that our MP is effective at generalization in BMP regardless of its feature extraction backbones.    \n\n\n**[WFYf.4 - Connection to neuroscience]** The stimulus design in our paper is inspired by neuroscience works. A non-exclusive list of neuroscience papers using point-light displays and studying different BMP conditions are [30,82,9,46,6,16,34,40,64,69,89] in the References. In our work, we also introduced a computational model for BMP. We did not make any claims that our model is biologically plausible as we agree with the reviewer that the model is only loosely inspired by neuroscience.\n\nThe reviewer also raises this interesting question on how the start, end, or development of the action would influence action recognition. To address this question, we analyze the effect of which frames are essential for the pick-up action class in one example video. Note that we are unable to systematically and rigorously test all action classes due to the limited time of the rebuttal.\n\nBriefly, we randomly selected X frames among 32 frames, duplicated the remaining frames to replace these selected frames, and observed the accuracy drops, where X = [1,8,16,24,28,31]. When multiple frames are replaced, the performance drop implies the importance of the development of these frames. In total, we performed 1000 times of random frame selections per X and presented the visualization of frame importance by averaging all the accuracy drops over all the random frame selections. See Fig R1 in the rebuttal PDF. The visualization results suggest that the fourth and seventh frames are essential for the pick-up class recognition.\n\nIn addition to the question raised by the reviewer, our approach also raises other intriguing neuroscience questions, such as what are the neural basis for motion-invariant neurons that are crucial when video frames are shuffled or reversed. Our work takes an initial step in connecting artificial and biological intelligence in BMP. These bio-inspired architectures can help validate certain hypotheses in neuroscience, and insights from neuroscience can inform the design of better AI systems. We will include these discussion points in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwdiXyALgQ", "reviewer": "Reviewer_eynA", "summary": "This paper uses AI models to investigate how humans can accurately predict actions based solely on minimal action patterns without prior training. The authors propose a new motion recognition model called Motion Perceiver (MP). This new model first extracts patch-level semantic features from images using a DINO-pretrained ViT model, then computes patch-level optical flow from these features, and develops two pathways featuring motion-invariant representations and slot-based representations from these optical flows. They show that this new model yields more human-like generalization performance on recognizing actions from videos of joints and sequential positions than existing AI models. Their new model also shows some human-like behavior patterns when performing action recognition. They further present ablation studies showing which components in this new model are critical.", "questions": "See the weakness points.", "limitations": "The authors discussed the limitations of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper contains solid work on collecting/constructing training and testing datasets, developing a new computational model, training existing baselines on the datasets, comparing the new model to these existing baselines, and showing the influence of different architectural designs in the new model. The authors have also collected human psychophysics data through MTurk on their minimal motion stimuli, which is useful for the community.\n\nThe new model proposed by the authors outperforms the existing baselines by significant and sometimes large margins. Both the new model and baselines are evaluated on various scenarios to test their generalizability performance.\n\nThe paper is well written, with a lot of details in the appendix and clear descriptions in the main text.", "weaknesses": "1.\tThis paper needs stronger baselines. The authors train their baselines on the small amount of natural RGB videos collected by them. However, humans perceive a lot of videos during their development in the real world. This means a stronger and more human-like baseline is finetuning from a pretrained video model using self-supervised learning algorithms such as VideoMAE, instead of starting from random initializations. Moreover, the fact that their new model (MP) uses a DINO-pretrained visual encoder makes the existing baselines even weaker, since all the baselines are trained from scratch on the small amount of videos. Fixing this would make the comparison much more reasonable and fairer.\n\n2.\tThe authors also need to show how good models directly trained on these joints or sequential points videos can perform on this task. It is important to get a sense about how far away this new model is compared to models directly trained on within-domain videos. This can also help tell whether human is better than these models or not.\n\n3.\tSuppose this work aims to get a more human-like model through this human-like learning curriculum compared to just training a model on the joint or sequential point videos. In that case, the authors need to show whether their new model is quantitatively more human-like compared to the other models. This can include explaining per-category action-recognition performance or the error patterns when recognizing actions.\n\nI appreciate the authors' responses to these points. The first point is well explained. Please also remember to include this training detail of the baseline models in the final version. The additional experiments on the second point are also convincing. I raise my score from 6 to 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[eynA.1 - Unfair comparison with baselines and VideoMAE]** All the baselines are action recognition models pre-trained on Kinetics 400. In contrast, the DINO architecture in our MP is pre-trained on ImageNet. All the models are then fine-tuned on RGB videos in the training set of our BMP dataset. We would like to emphasize that it is remarkable that our MP can still outperform all the baselines even though it has never been pre-trained on any video datasets.\n\nAs requested by the reviewer, we now include VideoMAE [b] for comparison with our MP. The model is trained on Kinectics with a self-supervised learning method. Just like other baselines, the performance of VideoMAE is still inferior to our model as shown in Tab R1. We will include this experiment in the final version.  \n\n**[eynA.2 - Train on joint or SP videos]**  That is a great suggestion! We now directly train our model on J-6P and test on J-6P. Its accuracy on J-6P is 95.4% whereas human performance is 86%. Surprisingly, we found this model trained on J-6P also achieves 55% accuracy in RGB and 71% in SP-8P-1LT, which are far above chance. This implies that our model has generalization ability across multiple modalities. Similarly, we also train our model on SP-8P-1LT and test it on all three modalities: 42.7% in RGB, 69.3% in J-6P, and 93% in SP-8P-1LT. The reasonings and conclusions stay the same as the model directly trained on J-6P.\n\nNote that although our model achieves very high accuracy on the in-domain test set (train on J-6P, test on J-6P and train on SP-8P-1LT, test on SP-8P-1LT), its overall performance over all three modalities (RGB, J-6P, and SP-8P-1LT) is still lower than humans (73.6% vs 88.2%). This emphasises the importance of studying model generalisation in BMP. There is still a performance gap between AI models and humans in BMP.    \n\n**[eynA.3 - human-model alignment]**  The reviewer asked us to compare the human-model alignment between our model and models trained directly on J-6P or SP-8P-1LT. However, this would defeat the purpose of our study. We argue that the generalization ability of a model should arise from well-designed architectures, visual inputs, and learning rules. Augmenting data to tackle the generalisation problem is one engineering and effective approach in computer vision [d-h]. However, just like how humans have been exposed to only RGB naturalistic videos and meanwhile can generalize to multiple types of BMP stimuli, we are interested in developing AI models capable of generalizing to BMP like the way humans do without any data augmentations on the visual inputs during training. \n\nThe alignment between models and humans can be assessed in three aspects: (1) the absolute accuracy of a model matches with human accuracy in all the action classes across all BMP conditions, (2) the relative change in accuracy across all BMP conditions is consistent in terms of correlation values, (3) the error pattern consistency between models and humans is high. We present the results in these three aspects for our model and all the baselines.    \n\nSimilar to Fig 6, we now present Fig R3 in the rebuttal PDF where each dot on the scatter plot indicates the absolute accuracy of a model for every action class for all BMP conditions. If a model and humans achieve the same accuracy in the same action class across all five fundamental properties in BMP, the slope of the fitted linear line is 1 and all the markers would be at the diagonal line. The slope of close to 1 in our model in Fig R3 suggests that our model performs competitively well as humans in all action classes in BMP.\n\nTo evaluate the relative change in accuracy across all BMP conditions, we now compute the correlation values for all the models and present the results in Tab R5. Our MP shows the highest correlation with human performance compared with all the baselines. \n\nLastly, we use the error consistency metric introduced in [p] to evaluate all models’ error consistency with humans and report the results in Tab R6. It turns out that our model achieves the highest error consistency with humans. This implies that humans and our model tend to make similar mistakes at the trial levels. \n\nOverall, the absolute accuracy in Fig R3, the correlation value in Tab R5, and the error consistency score in Tab R6 suggest that our model aligns with humans much more than all the baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers' feedback. Results are provided in the tables here, and we encourage reviewers to refer to the PDF file containing additional figures. To differentiate these new figures and tables in the rebuttal from those in the main text, we have prefixed them with "R" in the rebuttal. For example, Fig R1 and Tab R1 correspond to Fig 1 in the rebuttal PDF and Tab 1 in this Author Rebuttal.  We have included a point-by-point response for each of the four reviewers.

**Table R1: Results of new baselines, MViT (the baseline method in our paper), and our motion perceiver (MP). Top-1 accuracy (%) is reported. Best is in bold.**
|  | RGB | J-6P | SP-8P-1LT |
|---|---|---|---|
| E2S-X3D [a] | 98.7 | 10.2 | 10.7 |
| VideoMAE [b] | 90.0 | 9.9 | 9.9 |
| MViT | **99.0** | 52.0 | 15.1 |
| TwoStream-CNN [c] | 97.0 | 15.7 | 10.4 |
| MP(ours) | 96.5 | **69.0** | **49.7** |

**Table R2:Number of trainable parameters in million (M) for baselines and our motion perceiver (MP).**
|  | ResNet3D | I3D | R(2+1)D | SlowFast | X3D | MViT | MP(ours) |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Trainable Param (M) | 31.7 | 27.2 | 27.3 | 33.7 | 3.0 | 36.1 | 57.5 |


**Table R3: Performance of MViT, MP (block 12) and enhanced MP (blocks 1, 7 and 12) in terms of the amount of visual information. Top-1 action recognition accuracy is reported. Best is in bold.**

||RGB|J-26P|J-18P|J-14P|J-10P|J-6P|J-5P|
|-|-|-|-|-|-|-|-|
|MViT|**99.0**|76.9|75.5|74.6|68.1|52.0|47.9|
|MP(ours)|96.5|72.2|73.2|70.4|71.7|69.0|65.5|
|Enhanced MP|97.8|**80.0**|**80.6**|**78.1**|**81.0**|**78.5**|**76.7**|

**Table R4:  Human and our MP performances in per-category accuracy. Top-1 action recognition accuracy is presented in the format: Human Performance/MP Performance**.
|   | **pick up** | **throw** | **sit down** | **stand up** | **Kick something** | **Jump up** | **Point to something** | **Nod head/bow** | **Falling down** | **Arm circles** |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| **RGB** | 97.5/96.0 | 96.7/90.9 | 100.0/95.6 | 100.0/98.6 | 98.3/95.9 | 100.0/100.0 | 98.3/93.8 | 96.7/95.5 | 93.3/99.3 | 100.0/99.3 |
| **J-6P** | 93.3/56.4 | 75.0/76.2 | 76.7/64.2 | 85.0/70.1 | 95.0/73.2 | 95.0/91.3 | 81.7/44.7 | 91.7/68.2 | 75.0/53.4 | 95.0/93.2 |
| **SP-8P-1LT** | 81.7/49.1 | 68.3/57.0 | 80.0/57.7 | 85.0/63.7 | 90.0/28.5 | 86.7/84.1 | 66.7/4.1 | 55.0/37.7 | 96.7/58.0 | 93.3/60.1 |
| **LVI** | 66.9/44.3 | 55.3/50.5 | 50.6/45.1 | 60.3/51.7 | 78.1/26.0 | 83.6/74.2 | 40.6/3.7 | 36.1/23.4 | 89.2/48.0 | 81.7/44.1 |
| **AVI** | 95.6/53.3 | 78.9/73.2 | 81.1/67.9 | 86.7/75.0 | 95.6/73.0 | 97.8/90.7 | 83.3/44.3 | 86.7/72.0 | 80.0/50.2 | 97.8/89.9 |
| **TO** | 77.1/56.6 | 42.9/68.7 | 26.3/2.1 | 11.7/5.2 | 90.8/77.8 | 87.1/92.8 | 74.2/38.8 | 77.5/59.3 | 25.4/29.1 | 92.9/58.1 |
| **TR** | 75.4/57.2 | 41.3/36.9 | 79.2/80.5 | 85.4/83.6 | 83.8/63.8 | 60.0/27.6 | 80.4/53.4 | 72.5/67.4 | 85.8/70.7 | 64.2/12.2 |
| **ICV** | 96.7/56.3 | 80.0/76.2 | 83.3/64.2 | 85.0/70.1 | 96.7/72.9 | 96.7/91.1 | 87.2/44.4 | 85.6/68.3 | 81.1/53.4 | 95.6/93.5 |
|  |  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |

**Table R5: Correlation values for all the models across all the BMP conditions. Best in bold.**
|  | ResNet3D | I3D | R(2+1)D | SlowFast | X3D | MViT | MP(ours) |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Correlation | 0.37 | 0.53 | 0.39 | 0.71 | 0.71 | 0.66 | **0.89** |

**Table R6: Error consistency results of all the models across all the BMP conditions. Best in bold.**
|  | ResNet3D | I3D | R(2+1)D | SlowFast | X3D | MViT | MP(ours)|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| error consistency | 0.135 | 0.175 | 0.110 | 0.236 | 0.209 | 0.219 | **0.240** |

**Reference list:**

[a] Filip et al., Is appearance free action recognition possible? ECCV 2022. 

[b] Zhan et al., Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS 2022.

[c] Karen et al., Two-stream convolutional networks for action recognition in videos. NeurIPS 2014.

[d] Yun et al., Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV 2019.

[e] Dabouei et al., Supermix: Supervising the mixing data augmentation. CVPR 2021.

[f] Hong  et al., Stylemix: Separating content and style for enhanced data augmentation. CVPR 2021.

[g] Zhong et al., Random erasing data augmentation. AAAI 2020.

[h] Kim et al., Learning temporally invariant and localizable features via data augmentation for video recognition. ECCV 2020 Workshops.

[i] Kowal et al., A deeper dive into what deep spatiotemporal networks encode: Quantifying static vs. dynamic information. CVPR 2022.

[j] Choi et al., Why can't i dance in the mall? learning to mitigate scene bias in action recognition. NeurIPS 2019.

[k] He et al., Human action recognition without human. ECCV 2016 Workshops.

[l] Yamins et al., Performance-optimized hierarchical models predict neural responses in higher visual cortex. PNAS 2014.

[m] Zhang et al., Finding any Waldo with zero-shot invariant and efficient visual search. Nat. Commun.2018.

[n] Geirhos et al., Generalisation in humans and deep neural networks. NeurIPS 2018.

[o] Schrimpf et al. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron 2020.

[p] Geirhos et al., Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. NeurIPS 2020.

[q] Simonyan et al., Two-stream convolutional networks for action recognition in videos. NeurIPS 2014.

Meta-review: All reviewers agreed that this paper is suitable for publication. The authors used a classic psychophysics paradigm in human vision to motivate the development of a new vision architecture. The problem and approach will be inspiring to cognitive scientists, and the benchmark will be useful for cognitive scientists and potentially the computer vision community to better gauge the alignment (or lack thereof) of there models with human vision.
Decision: Accept (poster)