Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception

Authors: shuangpeng  han
ziyu  wang
mengmi  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 11:33:57

Abstract: Biological motion perception (BMP) refers to humans' ability to perceive and recognize the actions of living beings solely from their motion patterns, sometimes as minimal as those depicted on point-light displays. While humans excel at these tasks \textit{without any prior training}, current AI models struggle with poor generalization performance. To close this research gap, we propose the Motion Perceiver (MP). MP solely relies on patch-level optical flows from video clips as inputs. During training, it learns prototypical flow snapshots through a competitive binding mechanism and integrates invariant motion representations to predict action labels for the given video. During inference, we evaluate the generalization ability of all AI models and humans on 62,656 video stimuli spanning 24 BMP conditions using point-light displays in neuroscience. Remarkably, MP outperforms all existing AI models with a maximum improvement of 29\% in top-1 action recognition accuracy on these conditions. Moreover, we benchmark all AI models in point-light displays of two standard video datasets in computer vision. MP also demonstrates superior performance in these cases. 
More interestingly, via psychophysics experiments, we found that MP recognizes biological movements in a way that aligns with human behaviors. Our data and code are available at https://github.com/ZhangLab-DeepNeuroCogLab/MotionPerceiver.

Main paper: 
 our proposed motion perceiver (mp): 
 Our proposed model, Motion Perceiver (MP), aims to learn robust action-discriminative motion features from natural RGB videos and generalize these capabilities to recognize actions from BMP stimuli on point-light displays. MP takes the inputs of only patch-level optical flows from a video V, which comprises T frames denoted as {I 1 , I 2 , ..., I t , ..., I T }. While visual features from videos are typically useful for action recognition, our research focuses on extracting and learning motion information alone from natural RGB videos. Finally, MP outputs the predicted actions from a predefined set of labels Y. See Fig. 2 for the model architecture.Pixel-level optical flow has been a common approach to modelling temporal dynamics in videos [90, 91, 86, 99] . Different from these works, we use the frozen ViT [21] , pre-trained on ImageNet [19] with DINO [10] , to extract feature map F t ∈ R N ×C from I t and compute its optical flow relative to other frames. Consider F t as a 2D grid of patches in N = H × W , where H and W are the height and width of F t . N represents the number of patches in the feature map and C is the feature dimension per patch. A unique spatial location for each patch can be defined by its 2D coordinates. We concatenate the X and Y coordinates of all patches in F t to form the patch locations G ∈ R N ×2 .Two reasons motivate us to design patch-level optical flows. First, the empirical evidence [36] suggests that current unsupervised feature learning frameworks produce feature patches with semantically consistent correlations among neighboring patches. Therefore, patch-level optical flows convey meaningful motion information about semantic objects in a video. Second, pixel-level optical flows can be noisy due to motion blur, occlusions, specularities, and sensor noises. Feature maps obtained from feed-forward neural networks often mitigate these low-level perturbations and provide more accurate estimations of optical flows.Next, we introduce how the patch-level optical flow for video V is computed. Without loss of generality, given any pair of feature sets a ∈ R m×d and b ∈ R n×d , where m and n are the numbers of patches in the feature sets and d is the feature dimension, the adjacency matrix Q a,b between a and b can be calculated as their normalized pairwise feature similarities:Q (a,b) = e f (a)f (b) T /τ n e f (a)f (b) T /τ ∈ R m×n ,where the superscript T is the transpose function, f (•) is the l 2 -normalization, and τ is the temperature controlling the sharpness of distribution with its smaller values indicating sharper distribution. We set temperature τ = 0.001 and the influence of τ is analyzed in Appendix, Tab. S4.Using the patch features of I t as the reference, the optical flow between any consecutive frames I m and I m+1 is defined asEQUATIONĜIi→Ij represents the positions of all patches from F i of I i after transitioning to F j of I j . Essentially, the positions of patches with shared semantic features on F i tend to aggregate towards the centroids of corresponding semantic regions on F j . Consequently, the optical flow O It m→m+1 indicates the movement from I m to I m+1 for patches that exhibit similar semantic features as those in I t .For all m ∈ {1, 2, ..., T -1} and t ∈ {1, 2, ..., T }, we compute O It m→m+1 and concatenate them to obtain the patch-level optical flowÔ ∈ R T ×N ×2×(T -1) for video V as Ô = [O I1 1→2 , ..., O I T (T -1)→T ].Since small optical flows might be susceptible to noise or errors, the optical flows in Ô with magnitudes less than γ = 0.2 are set to 10 -6 to maintain numerical stability. Unlike [90] where optical flows are computed only between two adjacent frames, our method introduces Ô, where we compute a sequence of optical flows across all T frames for any reference patch of a video frame. This dense optical flow estimation approach at the patch level captures a richer set of motion dynamics, providing a more comprehensive analysis of movements throughout the video.Drawing on findings from neuroscience that highlight biological neurons selective for complex optic flow patterns associated with specific movement sequences in motion pathways [33] , we introduce "flow snapshot neurons." These neurons are designed to capture and represent prototypical moments or "slots" in movement sequences. Mathematically, we define K = 6 flow snapshot neurons or slots Ẑ ∈ R K×D , where D = 2 × (T -1) is the feature dimension per slot. Ẑ contains learnable parameters randomly initialized with the Xavier uniform distribution [35] . The impact of K is discussed in Appendix, Tab. S4. [63] separates and organizes different elements of an input into a fixed number of learned prototypical representations or "slots." The learned slots have been useful for semantic segmentation [102, 60, 111, 51, 22] . Here, we follow the training paradigm and implementations of the slot attention mechanism in [63] and apply it to Ô. Specifically, each slot in Ẑ attends to unique optical flow sequence patterns in Ô through a competitive attention mechanism based on feature similarities. To ensure the prototypical optical flow patterns in Ẑ are diverse, we introduce the loss of contrastive walks L slot [102] . During inference, we keep Ẑ frozen and leverage the activations of flow snapshot neurons for action recognition, denoted as M = f ( Ô)f ( Ẑ) T , where M ∈ R T ×N ×K . See Appendix, Sec. B.1 and Appendix, Sec. B.2 for mathematical formulations.Unlike [108] that use slot-bonded optical flow sequences M Ẑ ∈ R T ×N ×D for downstream tasks, we highlight two advantages of using M . First, the dimensionality of flow similarities (K) is typically much smaller than that of slot-bonded optical flow sequences (D), effectively addressing overfitting concerns and reducing computational overhead. Second, by leveraging flow similarities, we benefit from slot activations that encode prototypical flow patterns distilled from raw optical flow sequences. This filtration process helps eliminate noise and irrelevant information from the slot-bonded flow sequences, enhancing the model's robustness.The temporal dynamics in video clips can vary significantly in the speed and temporal order of the actions. Here, we introduce motion-invariant neurons that capture the accumulative motion magnitudes independent of frame orders. Specifically, we define the optical flow (O x,n,It m→t , O y,n,It m→t )for patch n in I m moving from frame I m to I t along x and y axes. Every patch-level optical flow in Ô can be projected into four motion components along +x, -x, +y and -y axes. Without loss of generality, we show that for patch n in I t , all its optical flow motions over T frames along the +x axis are aggregated:1 T T m=1 O x,n,It m→t 1 O x,n,I t m→t >0 where 1 x>0 = 1 if x > 0 0 otherwise (2)By repeating the same operations along -x, +y, -y axes for all the N patches over T frames, we obtain the motion invariant matrix M ∈ R T ×N ×4 . A stack of self-attention blocks followed by a global average pooling layer fuse information along 4 motion components and then along temporal dimension T . To encourage the model to learn motion invariant features for action recognition, we introduce cross-entropy loss L invar to supervise the predicted action against the ground truth Y.Processing videos at multiple temporal resolutions allows the model to capture both rapid motions and subtle details, as well as long-term dependencies and broader contexts. We use the subscript in Ô, Ẑ, and M to indicate the temporal resolution. Instead of computing Ô1 between consecutive frames only, we increase the stride sizes to 2, 4, and 8. For example, Ô4 ∈ R T ×N ×2×(T /4-1) denotes patch-level flows computed between every 4 frames. Note that we maintain a stride size of 1 for M to ensure motion invariance to temporal orders across the entire video, as subsampling frames would not adequately capture this attribute.For action recognition, the activations of flow snapshot neurons are concatenated across multiple temporal resolutions as M1,2,4,8 = [ M1 , M2 , M4 , M8 ] and then used for the fusion of motion information. The concatenated data is first processed through a series of self-attention blocks that operate across 4K slot dimensions, followed by a global average pooling over these dimensions. We then repeat the same fusion process over T time steps. Time embeddings, similar to the sinusoidal positional embeddings in [98] , are applied across frames. These embeddings help incorporate temporal context into the learned features. The resulting integrated motion feature vector is used for action recognition, with a cross-entropy loss L f low to supervise the predicted action against Y. While M1,2,4,8 captures the detailed temporal dynamics, M learns robust features against variations in temporal orders. MP combines feature vectors from M1,2,4,8 and M with a fully connected layer and outputs the final action label. A cross-entropy loss L f use is used to balance the contributions from M1,2,4,8 and M . The overall loss is: L = αL slot + L f low + L invar + L f use , where the loss weight α = 10. See Appendix, Tab. S4 for the effect of α.Implementation Details. Our model is trained on Nvidia RTX A5000 and A6000 GPUs, and optimized by AdamW optimizer [65] with cosine annealing scheduler [64] starting from the initial learning rate 10 -4 . Data loading is expedited by FFCV [56] . All videos are downsampled to T = 32 frames. We use a random crop of 224 × 224 pixels with horizontal flip for training, and a central crop of 224 × 224 pixels for inference. We use the same set of hyper-parameters for our model in all the datasets. More training details are in Appendix, Sec. C. 
our biological motion perception (bmp) dataset with human behavioral data: 
 Following the works in vision science, psychology, and neuroscience [33, 89, 9, 53, 6, 16, 37, 45, 71, 76, 97] , we introduce the BMP dataset, comprising 10 action classes (Appendix, Sec. A.2) specifically chosen for their strong temporal dynamics and minimal reliance on visual cues. Studies [107, 112, 32, 85] indicate that current action recognition models often rely on static features. Point-light displays reduce confounding factors, such as colors, sketches, and body limbs, by minimizing visual information, thereby highlighting the ability to perceive motion. This selection criterion ensures that specific objects or scene contexts, such as a soccer ball on green grass, do not bias the AI models toward recognizing the action as "playing soccer." This approach aligns with our research focus on generalization in motion perception. In the BMP dataset, there are three types of visual stimuli.Natural RGB videos (RGB): We incorporated 9,492 natural RGB videos from the NTU RGB+D 120 dataset [62] and applied a 7-to-3 ratio for dividing the data into training and testing splits. Joint videos (J): We applied Alphapose [26] to identify human body joints in the test set of our RGB videos. The joints of a moving human are displayed as light points, providing minimal visual information other than the joint positions and movements of these joints over time. Sequential position actor videos (SP): SP videos are generated to investigate scenarios where local inter-frame motion signals are eliminated [6] . Light points are positioned randomly between joints rather than on the joints themselves. In each frame, every point is relocated to another randomly selected position on the limb between the two joints with uniform distribution. This process ensures that no individual point carries the valid local image motion signal of limb movement. Nonetheless, the sequence of static postures in each frame still conveys information about body form and motion.We investigated 5 fundamental properties of motion perception by manipulating various design parameters of these stimuli. We adopted the experiment naming convention [type] + [condition] to denote the experimental setup for the stimulus types and the specific manipulation conditions applied to them. For instance, [Joint-3-Frames] indicates that three frames are uniformly sampled from the Joint video type. Next, we introduce the fundamental properties of motion perception and the manipulations performed to examine these properties.To disrupt the temporal order [84] , we reverse (Reversal) or randomly shuffle (Shuffle) video frames for two types of videos, RGB and Joint above. Temporal resolution (TR): We alter the temporal resolution [84] by uniformly downsampling 32 frames to 4 or 3 frames, labelled as 4-frames and 3-frames, respectively. For models that need a fixed frame count, each downsampled frame is replicated multiple times before advancing to the next frame in the sequence, until we reach the necessary quantity. Amount of visual information (AVI): We quantify the amount of visual information based on the number of light points (P) in Joint videos [45] . Specifically, we included conditions: 5, 6, 10, 14, 18, and 26 light points. Lifetime of visual information (LVI): In SP videos, we manipulate the number of consecutive frames during which each point remains at a specific limb position before being randomly reassigned to a different position. Following [6] , we refer to this parameter as the "lifetime of visual information" (LT), and include LT values of 1, 2, and 4. A longer lifetime implies that each dot remains at the same limb position for a longer duration, thereby conveying more local image motion information. Invariance to camera views (ICV): Neuroscience research [43] has revealed brain activity linked to decoding both within-view and cross-view action recognition. To probe view-dependent generalization effects, we sorted video clips from Joints into three categories based on camera views: frontal, 45 • , and 90 • views. See Appendix, Sec. A.3 for more implementation details.Human psychophysics experiments: We conducted human psychophysics experiments schematically illustrated in Appendix, Sec. D, using Amazon Mechanical Turk (MTurk) [8] . A total of 90 subjects were recruited. For data quality controls, we implemented checks during the experiment. Following a set of filtering criteria, we retained data from 88 subjects. See Appendix, Sec. D for details.To evaluate the ability of all AI models to recognize a wider range of actions in natural RGB videos, we include two standard video action recognition datasets in computer vision. NTU RGB+D 60 [87] comprises 56,880 video clips featuring 60 action classes. NW-UCLA [100] contains 1,493 videos featuring 10 action classes. Both datasets are split randomly, with a 7:3 ratio for the training and test sets. All AI models are trained on RGB videos and tested on Joint videos of these datasets curated with Alphapose [26] described above. We compute the top-1 action recognition accuracy of all AI models. In addition to our MP model, we include six competitive baselines below for comparison: ResNet3D [38] adapts the architecture of ResNet [39] by substituting the 2D convolutions with the 3D convolutions. I3D [11] extends a pre-trained 2D-CNN on static images to 3D by duplicating 2D filters along temporal dimensions and fine-tuning on videos. X3D [28] expands 2D-CNN progressively along spatial and temporal dimensions with the architecture search. R(2+1)D [96] factories the 3D convolution kernels into spatial and temporal kernels. SlowFast [29] is a two-stream architecture that processes temporal and spatial cues separately. MViT [25] is a transformer-based model that expands feature map sizes along the temporal dimension while reducing them along the spatial dimension. Furthermore, we include three more competitive baselines (E2-S-X3D [42] , VideoMAE [94] , TwoStream-CNN [90] ) and present results in Appendix, Sec. E. The findings remain consistent with the baselines above. All baselines except for TwoStream-CNN are pre-trained on Kinetics 400 [47] . TwoStream-CNN is pre-trained on ImageNet [19] . As an upper bound, we also train the MP directly on Joint or SP videos and the results are in Appendix, Sec. E.  
our model achieves human-level performance without task-specific retraining: 
 We explore five key properties of motion perception and their influence on video action recognition tasks between humans and our model (MP). Detailed comparisons between humans and all AI models across all BMP tasks are provided in the Appendix, Sec. L.Temporal order and temporal resolution matter more in Joint videos than natural RGB videos. In Fig. 3 , both humans and MP exhibit higher top-1 accuracy in RGB videos (RGB) compared to Joint videos (J-6P) due to the increased difficulty of generalization in Joint videos from their minimal motion and visual information. Error bars in Fig. 3 represent the standard error of top-1 accuracy across different action classes. However, despite this challenge, the performance on Joint videos remains significantly above chance (1/10). This suggests that both humans and MP have remarkable abilities to recognize actions based solely on their motion patterns in point-light displays. Moreover, we observed that shuffled (S) or reversed (R) temporal orders significantly impair recognition performance in both humans and MP. This effect is more pronounced in Joint videos (J-6P-R and J-6P-S) compared to RGB videos (RGB-R and RGB-S). The minimal motion information available in Joint videos makes them particularly susceptible to disruptions in temporal orders. The same behavioral patterns are also captured by MP. Interestingly, shuffling has a lesser impact on human performance compared to reversing RGB videos (RGB-R versus RGB-S, p-value < 0.05). However, this pattern is reversed in Joint videos (J-6P-R versus J-6P-S, p-value < 0.05). When considering actions such as sitting down versus standing up, reversing orders may alter the temporal context in RGB videos. The effect of temporal continuity outweighs the effect of temporal context in Joint videos.We conjectured that temporal resolution matters in video action recognition. Indeed, a decrease in top-1 accuracy with decreasing temporal resolutions is observed in both RGB and Joint videos (compare 32, 4, and 3 frames). However, this effect is more pronounced in Joint videos (J-6P, J-6P-4F, J-6P-3F) compared to RGB videos (RGB, RGB-4F, RGB-3F). Surprisingly, even in the most challenging J-6P-3F, both humans and MP achieve top-1 accuracy of over 40%, significantly above chance. This suggests that both humans and MP are robust to changes in temporal resolutions.Minimal amount of visual information is sufficient for recognition. In Fig. 4 , both humans and all AI models exhibit comparable performances in RGB videos (RGB). Interestingly, the accuracy drop from J-6P to J-5P is indistinguishable for humans (p-value > 0.05). Similarly, there was a wide range of the number of points that led to robust action recognition for MP (J-26P to J-5P). However, its absolute accuracy is much lower than humans. Therefore, we introduce an enhanced version of the MP model (En-MP) by extending it across multiple DINO blocks, in addition to the last DINO block. Detailed implementation of the En-MP is provided in Appendix, Sec. F. Surprisingly, the En-MP outperforms our original MP significantly and performs competitively well as humans. These observations suggest that a minimal visual representation in J-5P is sufficient for action recognition on humans and our En-MP. Conversely, traditional AI models show a significant performance decline moving from J-26P to J-5P. These findings suggest that existing AI models struggle with reduced visual information.Humans and MP are robust to the disrupted local image motion. Aligned with [6, 76] , in Fig. 5 top-1 accuracy in SP videos improves with an increased number of points for humans, a trend that MP replicates. Interestingly, both humans and MP do not show obvious increased performance with lifetimes of points more than 1 frame, potentially due to the loss of form information from fewer dots. This indicates that humans and MP capture not just local motions but also dynamic posture changes.Camera views have minimal impacts on recognition. Neuroscience studies [43, 97, 46, 68] provide evidence that both viewpoint-dependent and viewpoint-invariant representations for action recognition are encoded in primate brains. In Fig. 7 , MP exhibits a slightly higher accuracy on frontal and 45 • views compared to the 90 • (profile) view by 2.5%, which mirrors human performance patterns in [46] . However, we note that human behaviors in our study differ slightly from [46] (see Appendix, Sec.G). Moreover, MP significantly outperforms the second-best model, MViT, by 18%. This suggests that MP not only captures human-like viewpoint-dependent representations but also maintains superior accuracy among AI models across different camera views.MP aligns with human behaviors more closely than all the competitive baselines in BMP tasks. In Fig. 6 , we reported the correlation coefficients between each AI model and human performance across all conditions in the BMP dataset. The results demonstrate that our MP and En-MP exhibit a significantly higher correlation with human performance compared to the baselines, indicating that our MP and En-MP align more closely with human performance. In addition to the correlation coefficients, we also present the error pattern consistency [31] between models and humans (Appendix, Sec. H, Appendix, Fig. S4 ). Results suggest that our MP achieves the highest error pattern consistency with humans at the trial level.MP significantly outperforms all the competitive baselines in BMP tasks. In Appendix, Fig. S3 , we present the absolute accuracy of all models and humans. The slopes near 1 in our MP model indicate that it performs on par with humans and surpasses all baselines across the five BMP dataset properties. Despite explicitly modelling motion and visual information in separate streams, the SlowFast model struggles with temporal order. Likewise, MViT, which leverages transformer architectures, fails to generalize across different temporal resolutions.MP outperforms all the competitive baselines on Joint videos from two standard computer vision datasets. In Tab. 1, MP, relying solely on patch-level optical flows as inputs, performs above chance. This suggests that motion information alone is sufficient for accurate action recognition. Moreover, MP performs better than existing models on Joint videos in these datasets, highlighting that MP learns to capture better motion representations from RGB videos during training. In Appendix, Sec. I, we provide results and discussions when we explicitly feed pixel-level optical flows as inputs to these baselines during training and test these models on the Joint videos. Our results show that MP still outperforms these models, suggesting that MP learns generalizable motion representations beyond optical flow patterns. In Appendix, Sec. J, we also present visualizations of patch-level optical flow examples, demonstrating its ability to semantically segment the movements of a person.To investigate how different components of MP contribute to the generalization performance on Joint and SP videos, we conducted ablation studies (Tab. 2, Appendix, Tab. S4 and Appendix, Tab. S5). Removing the pathway involving motion invariance neurons leads to a drop in accuracy when video frames are shuffled or reversed (A1, Tab. 2 and Appendix, Fig. S3 ). For example, MP outperforms its ablated model without MIN in the following experiments: 62.32% vs 49.58% in RGB-R; 61.34% vs 38.03% in RGB-S, 38.69% vs 36.03% in J-6P-R, and 32.65% vs 25.28% in J-6P-S. Similarly, removing the pathway involving flow snapshot neurons also leads to a significant accuracy drop (A2). Both ablation studies highlight the importance of two pathways for action recognition. Instead of multi-scale feature fusion, we utilize a single-scale pathway of flow snapshot neurons (A3). The decrease of 10.57% in accuracy on J-6P videos implies that multi-scale flow snapshot neurons capture both fine-grained and high-level temporal patterns, essential for action recognition.In A4, the model takes patch-level optical flows as inputs and directly utilizes them for feature fusion and action classification without flow snapshot neurons. There is a significant drop of 14.33% in accuracy, implying that flow snapshot neurons capture meaningful prototypical flow patterns, and their activations are critical for recognition. Threshold γ controls the level of noise or errors in small motions. As expected, removing γ leads to impaired recognition accuracy (A5).Data augmentation is a common technique to enhance AI model generalization [88, 110, 17, 40, 114, 49] . It typically includes temporal augmentations like random shuffling and reversal in A6. Surprisingly, MP's generalization performance is impaired, indicating that randomizing frame orders during training disrupts motion perception and action understanding. We also present ablation results in RGB videos, with similar conclusions drawn, albeit to a lesser extent due to increased visual information and reduced reliance on motion.In addition to the main ablation results discussed here, we also provide the extra ablation studies and model analysis in the Appendix. Specifically, we include the following studies: frames predicted by our MP on an example video in Appendix, Sec. K. All these studies emphasized the importance of our model designs and demonstrated the generalization ability of our model. 
discussion: 
 We introduce Motion Perceiver (MP) as a generalization model trained on natural RGB videos, capable of perceiving and identifying actions of living beings solely from their minimal motion patterns on point-light displays, even without prior training on such stimuli. Within MP, flow snapshot neurons learn prototypical flow patterns through competitive binding, while motion-invariant neurons ensure robustness to variations in temporal orders. The fused activations from both neural populations enable action recognition.To evaluate the generalization capabilities of all AI models, we curated a dataset comprising 63k stimuli across 24 BMP conditions using point-light displays inspired by neuroscience. Psychophysics experiments on this dataset were conducted, providing human behavioral data for comparison with computational models. While AI models can surpass human performance in numerous tasks, current AI models for action recognition still fall short of human capabilities in many BMP conditions. By focusing solely on motion information, MP achieves superior generalization performance among all AI models and demonstrates a strong alignment with human behavioral responses. All baselines are pre-trained on large-scale video datasets, whereas our MP uses feature extractors pre-trained on naturalistic images. Remarkably, despite lacking video pre-training, our MP outperforms all baselines.Our work takes an initial step toward bridging artificial and biological intelligence in BMP. First, it raises intriguing questions in neuroscience, such as the neural basis of motion-invariant neurons, which are crucial when video frames are shuffled or reversed. Bio-inspired architectures can help test specific neuroscience hypotheses, while insights from neuroscience can, in turn, guide the design of more advanced AI systems. Second, our work paves the way for real-world applications requiring robust motion recognition, such as in low-light conditions where visual information is limited.The 10 action classes in our BMP dataset were selected for their rich temporal information. We observe significant variations in action recognition accuracy across various action classes for both human participants and AI models. In the future, the BMP dataset could be expanded to include more complex actions. Both neuroscience and computer vision use various biological motion stimuli; here, we focus on point-light displays, but future work could explore other visual stimuli, such as motion patterns in noisy backgrounds. While our model shows promising generalization on BMP stimuli, it does not account for attention or top-down influences. Moreover, since it only uses patch-level optical flows, integrating information from the ventral (form perception) and dorsal (motion perception) pathways remains an open challenge. Further discussion on the social impact of our work can be found in Appendix, Sec. M.A.1 Example Videos in the BMP DatasetWe provide one example RGB video and its corresponding BMP videos under different conditions where the subject is performing a sit-down action. The naming convention of these BMP videos follows the same as in Sec. 3.1. See the example video at the link.There are 10 action classes in the BMP dataset: pick up, throw, sit down, stand up, kick something, jump up, point to something, nod head/bow, falling down and arm circles.In Sec. 3.1, we vary the amount of visual information based on the number of point lights on Joint videos. To generate Joint videos, Alphapose [26] is used as the tool to detect the joints of a human body from RGB videos. After that, we use these detected joints to generate BMP stimulus on point-light displays. The specific positions of light points correspond to certain joint locations:• J-26P: 1 point each on the head, nose, jaw, and abdomen, 1 point each on one eye, one ear, one shoulder, one elbow, one hand, one hip and one knee, and 8 points on two feet with 4 points on each foot; • J-18P: 1 point each on the head, nose, jaw, and abdomen, 1 point each on one ear, one shoulder, one elbow, one hand, one hip, one knee and one ankle; • J-14P: 1 point each on the nose, and abdomen, 1 point each on one shoulder, one elbow, one hand, one hip, one knee and one ankle; • J-10P: 1 point each on the nose, and abdomen, 1 point each on one shoulder, one hand, one hip and one ankle; • J-6P: 1 point each on the nose, and abdomen, 1 point each on one hand and one ankle; • J-5P: 1 point each on the nose, 1 point each on one hand and one ankle.We also looked into the invariance property to camera views in Sec. 3.1. The video clips in J-6P are categorized based on the viewpoints in the videos: the frontal view, 45°view and 90°view, which are respectively labelled as J-6P-0V, J-6P-45V and J-6P-90V in short form. 45°view and 90°view are rotated either clockwise or counterclockwise from the frontal view.As explained in Sec. 2.2, the slot attention module [63] aims to obtain flow snapshot neurons Ẑ ∈ R K×D to capture prototypical moments from the patch-level optical flow Ô ∈ R S×D where S = T × N and D = 2 × (T -1). Formally, based on the cross-attention mechanism [98] , slots Ẑ serve as the query, Ô contribute to both the key and the value, and q(•), k(•), v(•) are the linear transformation employed to project inputs and slots into a common dimension B, which can be formulated as: attn i,j := e Ji,j K l e J i,lwhereEQUATIONEQUATIONIt is worth noting that the superscript T is the transpose function, and the attention matrix U is normalized to make sure that no parts of the input are overlooked. Next, following the work [63] on slot attention, the slots are iteratively refined recurrently based on the Gated Recurrent Unit (GRU) [13] to maintain a smooth update. Let Ẑ0 represent the initial slot base, with parameters initialized by randomly sampling from the Xavier uniform distribution [35] . We denote the hidden state at time step p as h p , and slots at time step p and p + 1 as Ẑp and Ẑp+1 . Next, we get:EQUATION)After P iterations, the final slot features are represented as Ẑ = ẐP . We empirically set P = 3 [63] .To encourage diversity in prototypical optical flow patterns in Ẑ, we use the contrastive loss between Ô and Ẑ introduced in [102] .EQUATIONwhere Q (a,b) is defined as the normalized pairwise feature similarities between a and b in Sec. 2.1.We use the temperature µ of 0.05 in Q ( Ẑ, Ô) and Q ( Ô, Ẑ) . CE(•,•) stands for the cross-entropy loss and I ∈ R K×K is the identity matrix. The effect of µ is covered in Appendix, Tab. S4. Fig. S1 demonstrates the schematic of human psychophysics experiments on Amazon Mechanical Turk (MTurk) [8] . We recruited a total of 90 human subjects. We collect data from a total of 12,600 trials.Top-1 Acc. (%) RGB J-6P SP-8P-1LT E2S-X3D [42] 98.7 10.2 10.7 VideoMAE [94] 90.0 9.9 9.9 TwoStream-CNN [90] In each trial, participants are shown a video randomly selected from the BMP dataset and are then asked to perform a forced 1-out-of-10 choice test to identify the action in the video. Videos from all conditions are randomly selected and presented in random order. The stimuli are uniformly distributed across BMP conditions, with no long-tailed distribution. These action classes are commonly performed in our daily life and free from cultural bias, as psychophysics experiments show humans can recognize them with nearly 100% accuracy on RGB videos. Almost all subjects are from the US, and we did not collect demographic data. All the experiments are conducted with the subjects' informed consent and according to protocols approved by the Institutional Review Board of our institution. Each subject was compensated.To control data quality, two checks are implemented in the experiment: (1) Four pre-selected videos in the RGB condition are used as the dummy trials, with two videos representing the "Sit Down" action class and the other two representing the "Stand Up" action class. Since these four videos are meant for quality controls. There is no ambiguity in classifying the action classes in these four videos. These four dummy trials are randomly dispersed with the rest of the actual trials in one experiment. Participants who fail to recognize correct actions in any of the four videos are excluded for data analysis. 88 out of the total 90 participants passed the tests in the dummy trials. (2) Each participant is allowed to participate in the experiment only once. All the trials in one experiment are always unique. This is to prevent the subjects from memorizing the answers from the repeated trials.Besides the comparison of State-of-the-Art methods discussed in Sec. 3.2, this section offers comparisons with three additional baseline methods. E2-S-X3D [42] is a two-stream architecture processing optical flow and spatial information from RGB frames separately. VideoMAE [94] , a recent baseline trained on Kinetics in a self-supervised learning manner. TwoStream-CNN [90] incorporates the spatial networks trained from static frames and temporal networks learned from multi-frame optical flow. The results in Appendix, Tab. S1 indicate that our method significantly surpasses all baselines under the J-6P and SP-8P-1LT conditions, highlighting the superior generalization capability of our MP model on the BMP dataset.Moreover, as stated in Sec. 3.2, we explore the upper bound performance of our MP on Joint and SP videos. First, we directly train our MP model on J-6P and test it on J-6P. Its accuracy on J-6P is 95% whereas human performance is 86%. Surprisingly, we found this model trained on J-6P also achieves 55% accuracy in RGB and 71% in SP-8P-1LT, which are far above chance. This implies that our model has generalization ability across multiple modalities. Second, we also train our model on SP-8P-1LT and test it on all three modalities: 43% in RGB, 69% in J-6P, and 93% in SP-8P-1LT. The reasonings and conclusions stay the same as the model directly trained on J-6P. Note that although our model achieves very high accuracy on the in-domain test set (train on J-6P, test on J-6P and train on SP-8P-1LT, test on SP-8P-1LT), its overall performance over all three modalities (RGB, J-6P, and SP-8P-1LT) is still lower than humans (74% vs 88%). This emphasises the importance of studying model generalisation in BMP. There is still a performance gap between AI models and humans in BMP.It is worth noting that our current MP model (see Sec. 2) is applied solely to the feature maps from the final attention block of DINO (block 12). We have observed that our MP model can also be effectively applied to early and intermediate-level blocks of DINO. The final prediction is then generated by fusing features across these three blocks (blocks 1, 7, and 12). We refer to this improved model as the Enhanced Motion Perceiver (En-MP).As mentioned in Sec. 4.1, human behaviors in our study reveal a divergent trend compared to the human performance pattern described in [46] . We point out two reasons: first, the tasks differ between our study and [46] . Our study focuses on action recognition while the work [46] focuses on walking pattern discrimination. Second, we found the performance variations across different camera views depend on the action classes. As shown in Appendix, Fig. S2 , humans exhibit the best performance when viewing "Arm Circles" actions from the frontal view, because the movement of the two arms will not overlap in the frontal view. For the videos from the action class "Point To Something", the 90°view is the best viewpoint since it results in longer movement trajectories of the arm than other viewpoints. Therefore, although humans can achieve the best average accuracy on the 45°viewpoint in our experiment, it does not imply that humans always perform the best on the 45°viewpoint in all action classes. The alignment between models and humans can be assessed in three aspects: (1) The correlation coefficients between all the AI models and human performance across all the BMP conditions. The results in Fig. 6 show that our MP has the highest correlation with human performance compared with baselines. (2) The absolute accuracy in all action classes across all BMP properties, which is reported in Appendix, Fig. S3 . We averaged the accuracies within the same property. It is observed that all AI models demonstrated lower accuracy than humans. However, among all AI models, our MP and En-MP outperform the rest. For example, despite explicitly modelling both motion and visual Figure S3 : Our MP and En-MP significantly surpass all existing models on both Joint and SP videos. A correlation plot between model and human performances is provided, with accuracy averaged over conditions within each property on the same type of stimulus (Sec. 3.1). The black dash diagonal serves as a reference.Figure S4 : Our MP and En-MP show higher error consistency with human performance compared to all existing models. The Error Pattern Consistency [31] between each AI model and human performance across all BMP conditions is reported. information in two separate streams, the SlowFast model still falls short in temporal orders. Similarly, MViT, which utilizes transformer architectures, fails to generalize to various temporal resolutions. See Appendix, Tab. S6 for detailed comparisons. (3) The error pattern consistency between AI models and humans using the metric introduced in [31] . The results in Appendix, Fig. S4 reveal that the error patterns from our MP and En-MP are more consistent with human error patterns than all the baselines at the trial level.As mentioned in Sec. 4.2, the MP model explicitly uses patch-level optical flow to extract temporal information for classification, whereas the baselines do not. Thus, here we introduce augmented baselines where they are trained on pixel-level optical flows. Specifically, we use GMFlow [105, 106] to extract the pixel-level optical flow from RGB videos in the BMP dataset, and then convert them into three-channel videos based on the Middlebury color code [5] . Taking these optical flow videos in the training set as the input to baselines, the performances on optical flow videos in the testing set and J-6P are shown in Appendix, Tab. S3. It is evident that all baselines achieve high accuracy on optical flow videos in the testing set, indicating that they effectively learn to recognize actions from optical flow inputs. However, their best performances on J-6P are significantly worse compared to the MP model (30.83% vs 69.00% in Appendix, Tab. S6), suggesting that our MP learns more generalizable motion representations from patch-level optical flows.ResNet3D [38] I3D [11] R(2+1)D [96] SlowFast [29] X3D [28] MViT [ Table S3 : Results of baselines with optical flow as input. Top-1 accuracy is reported on optical flow videos (OF) and Joint videos with 6 points (J-6P). Best is in bold.As outlined in Sec. 4.2, we provide a visualization of patch-level optical flow in vector field plots across example video frames for "stand up" action in Appendix, Fig. S5 . We can see that patch-level optical flow mostly happens in moving objects (the person performing the action) and captures high-level semantic features. Hence, they are more robust to perturbations in the pixel levels and more compute-efficient. (a) (b) (c) (d)To investigate the impact of additional hyperparameters and key components in our model, we have conducted further ablation studies. Note that we use the same set of hyper-parameters for our model across all the datasets. Table S4 : Ablation of key hyper-parameters in our model. Top-1 accuracy is reported on RGB videos, Joint videos with 6 points (J-6P), and SP videos with 8 points and a lifetime of 1 (SP-8P-1LT).From left to right, the ablated hyper-parameters are: the number of slots K (Sec. 2.2) the weight α of L slot (Sec. 2.2), temperature τ in the patch-level optical flow (Sec. 2.1) and temperature µ in L slot (Appendix, Sec. B.2). Best is in bold.Temperature τ in the patch-level optical flow: Appendix, Tab. S4 demonstrates that higher temperature τ used for computing patch-level optical flows hurt performances, compared with our full model where τ = 0.001. This implies that temperature τ controls the smoothness of the sequence of flows and lower temperature τ is beneficial for clustering flows with similar movement patterns.As shown in Appendix, Tab. S4, there is a non-monotonic performance trend versus the number of slots. The number of slots controls the diversity of temporal information captured from the path-level optical flow. An increase in the number of temporal slots can lead to redundant temporal clues, while a decrease might result in a lack of sufficient temporal clues.The weight α of L slot : The parameter α regulates the importance of L slot in comparison to other loss functions. A small value of α can impede the diversity in temporal slots, while a large α might adversely affect the optimization of features extracted from flow snapshot neurons and motion invariant neurons. From Appendix, Tab. S4, it can be seen that when α is either small (α = 1) or large (α = 100), there will be a slight degradation in performance compared with our full model with α = 10.Temperature µ in L slot : The temperature µ controls the sharpness of distribution over slots. High temperatures will negatively impact the convergence of slot feature extraction, while low temperatures can produce excessively sharp distributions, potentially impairing the stability of the optimization process during training. Appendix, Tab. S4 indicates that configuring µ at either 0.02 or 0.20 are non-optimized options compared with our full model when µ is 0.05. Key Frame Analysis As indicated in Sec. 4.3, to explore how the start, end, or development of the action would influence action recognition, we analyze the effect of which frames are essential for the pick-up action class in one example video. Briefly, we randomly selected X frames among 32 frames, duplicated the remaining frames to replace these selected frames, and observed the accuracy drops, where X = [1, 8, 16, 24, 28, 31] . When multiple frames are replaced, the performance drop implies the importance of the development of these frames. In total, we performed 1000 times of random frame selections per X and presented the visualization of frame importance by averaging all the accuracy drops over all the random frame selections. The visualization results in Appendix, Fig. S6 suggest that the fourth and seventh frames are essential for the pick-up class recognition.The detailed results of all BMP conditions are shown in Appendix, Tab. S6. It is evident that, in comparison to baselines, our motion perceiver (MP) not only delivers enhanced performance but also more accurately mirrors human performance in most BMP conditions. Especially in the case of SP-8P-1LT, our model significantly outperforms the top baseline by a substantial margin of 29.39%. These findings emphasize that our MP model is more effective at recognizing biological motion, demonstrating greater alignment with human behavioral data than existing AI models.ResNet3D [38] I3D [11] R(2+1)D [96] SlowFast [29] X3D [28] MViT [25] The Motion Perceiver, with its human-like ability to perceive biological motion, brings both promising advancements and concerning implications for society. In the sports and fitness field, it provides valuable tools for improving athletic performance, injury prevention, and personalized training. However, the same capabilities may raise privacy concerns, as the technology could potentially be misused for unauthorized surveillance or tracking of individuals' movements in public spaces. This invasion of privacy might extend to analyzing behavior patterns and gait. Furthermore, there is a risk of discrimination if the method is employed in contexts like employment screening or law enforcement profiling, where certain movement patterns might be unfairly associated with specific demographics or lead to biased judgments. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We discuss the risk for misuse of our work in Appendix, Sec. M.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original papers that produced the code packages or datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
training testing: 
 Temporal order Temporal resol.Amount/lifetime of visual info.Invariance to camera view 180 ! Figure 1 : Humans excel at biological motion perception (BMP) tasks with zero training, while current AI models struggle with poor generalization performance. AI models are trained to recognize actions from natural RGB videos and tested using BMP stimuli on point-light displays, which come in two forms: Joint videos, which display only the detected joints of actors in white dots, and Sequential position actor videos (SP), where light points in white are randomly positioned between joints and reallocated to other random positions on the limb in subsequent frames (Sec. 3.1). Note that skeletons, shown in gray in the example video, are not visible to humans or AI models during testing. The generalization performance of both humans and models is assessed after varying five properties in temporal and visual dimensions. See Appendix, Sec. A.1 for example videos.Biological Motion Perception (BMP) refers to the remarkable ability to recognize and understand the actions and intentions of other living beings based solely on their motion patterns [33] . BMP is crucial for tasks such as predator detection [24] , prey selection [23] , courtship behavior [69] , and social communications [18, 2, 1] among primates and humans. In classical psychophysical and neurophysiological experiments [45] , motion patterns can sometimes be depicted minimally, such as in point-light displays where only major joints of human or animal actors are illuminated. Yet, without any prior training, humans can robustly and accurately recognize actions [6, 71, 67] and characteristics of these actors like gender [53, 78] identity [16, 73] , personalities [7] , emotions [20] , social interactions [93] , and casual intention [77] .To make sense of these psychophysical and neurophysiological data, numerous studies [33, 34, 55, 89, 9, 52] have proposed computational frameworks and models in BMP tasks. Unlike humans, who excel at BMP tasks without prior training, these computational models are usually trained under specific BMP conditions and subsequently evaluated under different BMP conditions. However, the extent to which these models can learn robust motion representations from natural RGB videos and generalize them to recognize actions on BMP stimuli remains largely unexplored.In parallel to the studies of BMP in psychology and neuroscience, action recognition on images and videos in computer vision has evolved significantly over the past few decades due to its wide range of real-world applications [15, 41, 81, 82, 57, 74] . The field has progressed from relying heavily on hand-crafted features [99, 75, 54] to employing deep-learning-based approaches [90, 95, 11, 101, 28, 30, 29, 25, 4, 94] . These modern approaches can capture the temporal dynamics and spatial configurations of complex activities within dynamic, unstructured environments [80, 92] . Despite these advancements, existing AI models still struggle with generalization issues related to occlusion [103, 3, 58] , noisy environments [109] , viewpoint variability [27, 61, 100] , subtle human movements [83, 44] and appearance-free motion information [42] . While various solutions have been proposed to enhance AI generalization in action recognition [12, 72, 14, 70, 104, 48, 59, 113, 79, 50] , they do not specifically tackle the generalization challenges in BMP tasks.Here, our objective is to systematically and quantitatively examine the generalization ability of AI models trained on natural RGB videos and tested in BMP tasks. To date, research efforts in neuroscience and psychology [44, 34, 37] have mostly focused on specific stimuli for individual BMP tasks. There is a lack of systematic, integrative, and quantitative exploration that covers multiple BMP properties and provides a systematic benchmark for evaluating both human and AI models in these BMP tasks. To bridge this gap, we establish a benchmark BMP dataset, containing 62,656 video stimuli in 24 BMP conditions, covering 5 fundamental BMP properties. Our result indicates that current AI models exhibit limited generalization performance, slightly surpassing chance levels.Subsequently, to enhance the generalization capability of AI models, we draw inspiration from [33] in neuroscience and introduce Motion Perceiver (MP). MP only takes dense optical flows between any pairs of video frames as inputs and predicts action labels for the given video. In contrast to many existing pixel-level optical flow models [90, 91, 86, 99] , MP calculates dense optical flows at the granularity of patches from the feature maps. In MP, we introduce a set of flow snapshot neurons that learn to recognize and store prototypical motion patterns by competing with one another and binding with dense flows. This process ensures that similar movements activate corresponding snapshot neurons, promoting consistency in motion pattern recognition across patches of video frames. The temporal dynamics within dense flows can vary significantly depending on factors such as the speed, timing, and duration of the actions depicted in video clips. Thus, we also introduce motion-invariant neurons. These neurons decompose motions along four motion directions and integrate their magnitudes over time. This process ensures that features extracted from dense flows remain invariant to small changes and distortions in temporal sequences. Given a reference patch (yellow or green example patches), MP computes its patch-level optical flow (red arrows, Sec. 2.1) on the feature maps extracted from DINO [10] . Subsequently, these flows are processed through flow snapshot neurons (Sec. 2.2) and motion invariant neurons (Sec. 2.3) in two pathways. Activations from both groups of neurons are then integrated for action classification (Sec. 2.4). Time embeddings (T Emb.) are used in the feature fusion process.2. We propose the Motion Perceiver (MP). The model takes only patch-level optical flows of videos as inputs. Key components within MP, including flow snapshot neurons and motion-invariant neurons, significantly enhance its generalization capability in BMP tasks.Our MP model outperforms all existing AI models in BMP tasks, achieving up to a 29% increase in top-1 action recognition accuracy, and demonstrating superior performance in point-light displays of two standard video datasets in computer vision.The behaviors exhibited by the MP model across various BMP tasks demonstrate a high degree of consistency with human behaviors in the same tasks. Network analysis within MP unveils crucial insights into the underlying mechanisms of BMP, offering valuable guidance for the development of generalizable motion perception capabilities in AI models. 
 
 Appendix:  


Decision: All reviewers agreed that this paper is suitable for publication. The authors used a classic psychophysics paradigm in human vision to motivate the development of a new vision architecture. The problem and approach will be inspiring to cognitive scientists, and the benchmark will be useful for cognitive scientists and potentially the computer vision community to better gauge the alignment (or lack thereof) of there models with human vision.

Reviews: review: paper_summary: paper_summary: This paper proposes a new biologically inspired architecture for action recognition. The motion perceiver computes a patch-level optical flow from DINO features which is then processed in a two-stream architecture with one pathway using slot-attention to recognize different motion patterns and the other one integration over motion to get a time-independent motion signal. The paper also contributes a new dataset which provides multipe point light versions of the videos together with human performance data. The model is trained on RGB videos and evaluated on the point light videos. The authors show that the model's performance is well aligned with human performances on the different stimulus versions and that it outperforms other ML models on the least-information stimuli.
strength_weakness: strength_weakness: * I think at least one relevant comparison is missing: Illic et al, "Is Appearance Free Action Recognition Possible" (ECCV 2022, https://f-ilic.github.io/AppearanceFreeActionRecognition) asks a very related question and also uses somewhat similar stimuli (white noise stimuli, but also different kinds of dot stimuli) and also introduce a dataset to that end (AFD). Illic et al claim that their twostream architecture can solve the appearance free stimuli and at least on the homepage they also mention dot stimuli close to the ones used in the present paper. However, so far I couldn't find any dot stimuli results in the actual paper. Beyond this paper, I think there are more two-stream architecture models that might be relevant to include. This is overall the main reason that kept me from increasing the rating of the paper.
* the claim "Our model demonstrates human-like robustness to minimal visual information" (Figure 4) seems to strong for me. For most of the stimulus types, MP is not even the best competitor, only for J-6P and J5-P it outperforms all other models. But even then the performance drop compared to RGB seems to be 3 times more than for humans. The same holds for l282 "There is a slight decrease in accuracy when transitioning from RGB to J-6P for both humans and MP", where humans seem to be around 90% and MP around 70%. Interestingly, Figure 4 suggests that most of the performance drop comes from going from RGB to the point light display, because the drop happens already at J-26P, after which the performans of MP mostly stays constant.
* I'm missing more discussion of when and where "classical AI models fail or succeed and the alignment with humans". I think there is a bit to be learned from Appendix F. Especially, I would find it interesting to compute corellations with human scores across the different stimuli. Figure 3 suggests that MP is well aligned with human BMP. But I would like to know if it is also more aligned than other models (which I think would be a very strong additional result).
* notation sometimes a bit convoluted (M, \hat M, \tilde M, ...). Maybe sometimes it's worth using more descriptive names like "M_\text{invariant}"


**Update**: After the rebuttal, I increased my review from 7 to 8.
questions: questions: * How many trainable parameters has the final model, and how are they distributed over the model parts?
* given that the DINO based patch-level optical flow seems to be a major pillar of the model, I would love to get a better idea of how it performs on some example videos or images. It could be nice to have some vector field plots in the Appendix or a video in the Supplementary Material
* Caption Figure 1: "AI models are trained to learn to recognize actions": Is this on purpose? I would say they are either trained to recognize actions or they learn to recognize action. Being trained to learn to recognize would imply some meta learning.
* l98 "Considering Ft as a 2D grid of patches in N = H × W where H and W are the height and width of Ft " this sentence seems to be missing a subject
* l127: "This dense optical flow estimation approach at the patch level captures a richer set of motion dynamics, providing a more comprehensive analysis of movements throughout the video" I find it interesting that the patch-level optical flow is defined relative to a fixed reference frame instead of, e.g. always the first frame and then actually uses all frames as reference. Did the authors check how much does this help compared to only using e.g. the first frame? Right now this sentence doesn't seem to have any supporting evidence
* l158: " Every patch-level optical flow in Ô can be projected into four orthogonal motion components along +x, −x, +y and −y axes": I think I'm missing something here, but how are motions in +x and -x orthogonal?
* l163 "we obtain motion invariant matrix": "we obtain THE motion invariant matrix"?
* l270 " Interestingly, shuffling has a lesser impact on human performance compared to reversing RGB videos (RGB-R versus RGB-S)" technically, the two performances might be identical within the margin of error if I'm not mistaken.
* Figure 4: given the size of the error bars in Figure 3, I think it would be nice to also have error bars in this figure. Since it would make the figure harder to read, I think it would be enough to have a larger version in the appendix with error bars that can be referred here.
* l322: I agree that ablations A1 and A2 show that both pathways are important, but they seem to be hardly of similar importance. Removing the motion invariant pathway results in a drop of performance, while removing the flow snapshot neuron pathway essentially results in model performance collapsing. So I would say that the flow snapshot neuron pathway is the crucial part, and invariance helps further.
* l329 "In A4, the model takes patch-level optical flows as inputs and directly utilizes them for feature fusion and action classification without flow snapshot neurons" Unless I'm missreading the table, A4 has flow shapshot neurons?
* l349 "Psychophysics experiments on this dataset were conducted, providing human behavioral data as an upper bound": Why upper bound? I don't see why models couldn't outperform humans in principle.
* references to the appendix could be a bit more clear, e.g. "see Appendix, Sec B" instead of "see Sec B"
limitations: limitations: Limitations are well discussed in the main paper. Societal impact is not discussed because "There are no societal impacts, to the best of our knowledge" (l980), with which I would slightly disagree. There is always potential societal impact from building more human like models (good as well as bad). But this is mostly a nitpicking point that doesn't affect my review of the paper.
paper_summary: paper_summary: The paper introduces the Motion Perceiver (MP), a novel AI model designed to improve the generalization of action recognition in biological motion perception (BMP) tasks. It leverages patch-level optical flows and introduces flow snapshot neurons that learn and store prototypical motion patterns and motion-invariant neurons that maintain the consistency of motion recognition across temporal sequences. The authors create a comprehensive BMP benchmark dataset with 62,656 video stimuli across 24 BMP conditions and demonstrate that MP surpasses existing models by 29% in top-1 action recognition accuracy. The MP model’s performance closely aligns with human behavior in BMP tasks, providing significant insights into the underlying mechanisms of BMP and paving the way for more generalizable AI models in action recognition.
strength_weakness: strength_weakness: - Limited Contextualization: Although the paper does a good job explaining the MP model and its benefits, it could benefit from a more detailed comparison with prior work. Specifically, a deeper discussion on how the proposed method improves upon existing models in terms of architectural innovations and performance metrics would provide clearer context. Particularly, lines 56-57 state that “In contrast to many existing pixel-level optical flow models [83, 84, 79, 91, 71], ….”, but some of the references are actually representation-level flow models, e.g., [71]. More comprehensive comparisons to those prior works (theoretically and/or empirically) would make the claim more convincing.

- Dataset Limitations and Biases: The paper introduces a comprehensive BMP benchmark dataset, but it would be beneficial to address potential limitations and biases within this dataset. For example, discussing the diversity of the actions, the representation of different demographics, and the potential impact of these factors on the model’s performance would provide a more balanced view of the dataset’s strengths and limitations. Most importantly, the authors designed two scenarios (joint videos and sequential position actor videos), and claim they are good benchmarks for recognizing motion patterns. This claim is not well justified. For example, why not use sketches, optical flows, or other types of motion representation?

- Results Not Supportive Enough: In Fig. 4, it seems that the proposed method only outperforms on J-6P and J-5P, and is inferior on other levels. In this case, it is hard to claim that the proposed method has better generalization. While the J-6P and J-5P are most abstract and challenging, there is no evidence that those two levels are the best evaluators for generalization on action recognition. Additionally, the comparison methods are mostly dated; more recent baselines are needed. There is no mention/control on the efficiency and computation cost as well, making the comparisons less meaningful.

- Contribution: The biggest weakness is the overall contribution of the paper. While the paper is strong in proposing a new neural architecture and a benchmark, the proposed task itself is not justified with theoretical or practical value. There is no valid proof that joint and SP videos are good representations of biological motion (compared to other motion representations). More specifically, the joints are from the skeleton and keypoints of a human body, which limits its generalization to other objects. Regarding human action recognition itself, it is plausible that having a keypoint modality in the training procedure could potentially solve the problem of distribution shift (RGB to abstract motion representation). While the idea is well-motivated, the approach does not strongly support the claim; the methodology, the proposed benchmark, and empirical results will all benefit a lot from further study.
questions: questions: Most of my questions are described in Weaknesses. Some additional or specific ones:

1. What's the runtime of the proposed method compared to baselines?
2. Have you tried using the pixel-level optical flow downscaled to patch level and compared that to the patch flow?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces a novel, neuroscience-inspired approach to performing biological motion perception from videos. The videos are part of a dataset that the authors also introduce, depicting 10 different actions. The videos range from fully RGB frames to point-light displays that only cover the joints. While humans have been very successful at solving such tasks, even with limited data, AI models struggle. For this reason, the authors propose an elegant approach that involves obtaining optical flow from feature maps of an ImageNet-pretrained network and creating a system of slots to organize important features for the task. Additionally, they introduce features that account for scale and invariance. Overall, the method shows significant improvement over the tested baselines.
strength_weakness: strength_weakness: My main question/concern is whether the baselines are fairly compared with the proposed method. It might be worth adding learning curves to indicate whether the models have effectively converged. Additionally, including a parameter count to express whether the method is more or less computationally expensive than the baselines would provide useful insight.

I might have missed it but it seems that the inspiration to take  ViT as a feature extraction is not well motivated. Perhaps worth mentioning if there is some rationale that  harmonizes with neuroscience or if its a engineering choice. 

Another question/concern is that the connection back to neuroscience seems weak. However, there might be a potential opportunity arising from this approach. The ablation studies suggest a certain dependency on time resolution. It might be beneficial to show which frame in the sequence is the most important and how the start, end, or development of the action is crucial for action detection. This could tie back to neuroscience and provide testable hypotheses, thereby strengthening the paper.
questions: questions: I add them in the weakness section.
limitations: limitations: yes
paper_summary: paper_summary: This paper uses AI models to investigate how humans can accurately predict actions based solely on minimal action patterns without prior training. The authors propose a new motion recognition model called Motion Perceiver (MP). This new model first extracts patch-level semantic features from images using a DINO-pretrained ViT model, then computes patch-level optical flow from these features, and develops two pathways featuring motion-invariant representations and slot-based representations from these optical flows. They show that this new model yields more human-like generalization performance on recognizing actions from videos of joints and sequential positions than existing AI models. Their new model also shows some human-like behavior patterns when performing action recognition. They further present ablation studies showing which components in this new model are critical.
strength_weakness: strength_weakness: 1.	This paper needs stronger baselines. The authors train their baselines on the small amount of natural RGB videos collected by them. However, humans perceive a lot of videos during their development in the real world. This means a stronger and more human-like baseline is finetuning from a pretrained video model using self-supervised learning algorithms such as VideoMAE, instead of starting from random initializations. Moreover, the fact that their new model (MP) uses a DINO-pretrained visual encoder makes the existing baselines even weaker, since all the baselines are trained from scratch on the small amount of videos. Fixing this would make the comparison much more reasonable and fairer.

2.	The authors also need to show how good models directly trained on these joints or sequential points videos can perform on this task. It is important to get a sense about how far away this new model is compared to models directly trained on within-domain videos. This can also help tell whether human is better than these models or not.

3.	Suppose this work aims to get a more human-like model through this human-like learning curriculum compared to just training a model on the joint or sequential point videos. In that case, the authors need to show whether their new model is quantitatively more human-like compared to the other models. This can include explaining per-category action-recognition performance or the error patterns when recognizing actions.

I appreciate the authors' responses to these points. The first point is well explained. Please also remember to include this training detail of the baseline models in the final version. The additional experiments on the second point are also convincing. I raise my score from 6 to 7.
questions: questions: See the weakness points.
limitations: limitations: The authors discussed the limitations of the paper.

Comments: comment: We sincerely appreciate the time and effort you have put into thoughtfully reviewing our paper!

We agree with the three points raised by the reviewer. Indeed, our BMP task is challenging. Many models including Illic et al, find the BMP dataset challenging to generalize. Our proposed model demonstrates robust generalization capabilities in biological motion perception. The results from training our model on Joint or SP videos, along with the outcomes using minimal visual information (J-6P and J-5P), further support this claim.

Finally, your feedback and guidance have been helpful in enhancing our work. Thank you again for your valuable comments and support!
I thank the authors for the detailed answer to my review and for conducting the additional experiments and analyses I'm mostly very happy with the answers. I want to mention:

* **nybp.1 - missing AFD and two-stream models:** Thank you for including the model from Illic et al. The bad performanc of the model on the point light stimuli is interesting and demonstrates that they are are hard and powerful generalization test.
* **eynA.2 - train on joint or SP videos**: I very much like this additional experiment for estimating how well the architecture could solve the point light displays, but especially showing that the model generalizes surprisingly well from point light displays to RGB videos. Although this is clearly not the direction in which humans generalize, I think this result provides additional strong support for the hypothesis that the MP provides a strong and robust mechanism and makes the paper even more interesting.

* **buWx.3-Results not supportive enough**: I agree with the reviewer that it's a bit disappointing that for the higher-frequency point light displays other methods slightly outperform MP (I would have expected other models to fail more dramatically on these stimuli). Outperforming other models even earlier would clearly be even more convincing. But I think the model performance for the more reduced stimuli and on many other stimulus categories still is strong evidence for the generalization capabilities of the model architecture, especially together with the new evidence from eynA.2.


I think the rebuttal has strengthened this paper which I consider a very strong contribution to NeurIPS: It takes inspriation from Neuroscience both in terms of mechanisms as well as datasets to build a computer vision model with high human alignment and impressive generalization capabilities, making it is a very interesting example of NeuroAI. Hence I'm increasing my rating to 8.
We sincerely appreciate the time and effort you put into the review!
We sincerely appreciate the time and effort you put into the review!
We sincerely appreciate the time and effort you have put into thoughtfully reviewing our paper. Your feedback and guidance have been helpful in enhancing our work. Thank you again for your valuable comments and support!
I want to thank  the authors for the new experiments provided, including the selection of frame one that I suggested. I increase one point my score.
Thank you for your constructive feedback! We are glad that our responses have addressed most of your questions.

The reviewer is still not convinced that the proposed BMP design is a good representative of the generalization capability of motion perception. We respectfully disagree with this point and provide the following arguments:

Point-light displays (our BMP designs) are a highly effective tool for testing motion perception generalization because they isolate the motion cues from other visual information, allowing researchers to focus purely on how the brain perceives and interprets movement [30,82,9,46,6,16,34,40,64,69,89]. Here are some key reasons why point-light displays are particularly useful for studying the generalization capability of motion perception:

**1. Minimalistic Representation**

Point-light displays strip away all extraneous visual details, such as texture, colour, and form, and represent motion using just a few points of light corresponding to the major joints of a moving figure. This minimalistic representation ensures that any perception of motion is based purely on the movement of these points, allowing researchers to study the fundamental mechanisms of motion perception.

**2. Focus on Motion Cues**

Since point-light displays lack detailed structural information, the observer's ability to perceive motion relies solely on dynamic cues. This helps researchers understand how motion information alone contributes to the recognition of objects or actions, without the influence of other visual features.

**3. Generalization Across Contexts**

Because point-light displays remove contextual and visual details, they are an excellent way to test whether motion perception can generalize across different contexts. For example, people can still recognize a walking figure from point-light displays even when specific visual details are absent, demonstrating the brain's ability to generalize motion patterns.

**4. Biological Motion Perception**

Point-light displays are particularly effective for studying biological motion perception, which is the ability to perceive complex movements like walking, running, or dancing. These displays can show how well the visual system can recognize and interpret the patterns of movement that are characteristic of living beings, even with minimal visual information.

Overall, the use of point-light displays provides a controlled environment to study the fundamental aspects of motion perception and its generalization across different contexts and conditions.

Finally, we sincerely appreciate the time and effort you have put into thoughtfully reviewing our paper. Your feedback and guidance have been helpful in enhancing our work. Thank you again for your valuable comments and support!
Thanks the authors for the detailed rebuttal. The additional results with new baselines are beneficial. On the contribution side, it is still not convincing to me that the proposed BMP design is a good representative of generalization capability of motion perception, but I think the paper overall (methodology and evaluation framework) does make meaningful contributions to the field towards more generalizable perception. I have increased my score therefore.