Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli
Authors: Matthias Tangemann
Matthias Kuemmerer
Matthias Bethge
Venue: NeurIPS 2024 poster (2024)
Keywords: common fate
motion energy
optical flow
figure-ground segmentation
humans vs machines
Primary area: neuroscience_and_cognitive_science
TL;DR: We use random dot stimuli that remove appearance but not motion information from videos to show substantial differences between human perception and state-of-the-art machine vision.

Abstract: Humans excel at detecting and segmenting moving objects according to the {\it Gestalt} principle of “common fate”. Remarkably, previous works have shown that human perception generalizes this principle in a zero-shot fashion to unseen textures or random dots. In this work, we seek to better understand the computational basis for this capability by evaluating a broad range of optical flow models and a neuroscience inspired motion energy model for zero-shot figure-ground segmentation of random dot stimuli. Specifically, we use the extensively validated motion energy model proposed by Simoncelli and Heeger in 1998 which is fitted to neural recordings in cortex area MT. We find that a cross section of 40 deep optical flow models trained on different datasets struggle to estimate motion patterns in random dot videos, resulting in poor figure-ground segmentation performance. Conversely, the neuroscience-inspired model significantly outperforms all optical flow models on this task. For a direct comparison to human perception, we conduct a psychophysical study using a shape identification task as a proxy to measure human segmentation performance. All state-of-the-art optical flow models fall short of human performance, but only the motion energy model matches human capability. This neuroscience-inspired model successfully addresses the lack of human-like zero-shot generalization to random dot stimuli in current computer vision models, and thus establishes a compelling link between the Gestalt psychology of human object perception and cortical motion processing in the brain.

Code, models and datasets are available at https://github.com/mtangemann/motion_energy_segmentation

Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli
Matthias Tangemann
Matthias Kümmerer
Matthias Bethge
University of Tübingen, Tübingen AI Center matthias. (lasttaame)@bethhgellb.... org
Abstract
Humans excel at detecting and segmenting moving objects according to the Gestals principle of "common fate". Remarkably, previous works have shown that human perception generalizes this principle zero-shot fashion to unseen textures or random dots. In this work, we seek better understand the computational basis for this capability by evaluating a broad range of optical flow models and neuro- science inspired motion energy model for zero-shot figure-ground segmentation of random dot stimuli. Specifically, we use the extensively validated motion en- ergy model proposed by Simoncelli and Heeger 1998 which is fitted neural recordings in cortex area MT. We find that cross section of 40 deep optical flow models trained on different datasets struggle to estimate motion patterns in random dot videos, resulting in poor figure-grount segmentation performance. Conversely, the neuroscience-inse inspired model significantly outperforms all optical flow models on this task. For direct comparison to human perception, we conduct a psy- chophysical study using a shape identification task as a proxy to measure human segmentation performance. All state-of-the-art optical flow models fall short of human performance, but only the motion energy model matches human capability. This catroscienne-inspire model successfully addresses the lack of human- -liee zero-shot generalization to random dot stimuli in current computer vision models, and thus establishes compelling link between the Gestalt psychology of human object perception and cortical motion processing in the brain.
Code, models and datasets available at htttps://// htt.gtttubb.co.oo cow.bb///co//taggaaan///
motiion maotion_energy.oo www.eenseeeeeentteeeeeeee segmentation
Introduction
Motion is powerful cue that humans use to detect and segment visual objects. A striking example camouflaged animals, which are difficult to spot when stationary but become much easier to detect when moving, Motion segmentation in humans is believed to driven by the principle of common which posits that elements that move together, belong together. Remarkably, human perception generalizes this principle in a zero-shot fashion to novel textures or moving random dots. easily detect biological motion from only few moving dots. More recently, Robert et al. [31 introduced random dot stimuli called object kinematograms that preserve the motion in video while ensuring that static appearance cues are uninformative about the video contents (Fig. example video the supplemental material).
fate [47 48 42
For example, the seminal work by Johansson [19] showed that humans
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Segmentation
Motion Estimator type
Motion Energy
Random dots (zero shott
0.8
Optical
Motion Energy
0.6
Original videos (lAdd
Rendom dots, same motion (zero shot)
0.2
0.4
0.6
0.8
Original videos
We compare state-of-the-art optical flow estimators and neuroscience inspired motion energy model on figure-ground segmentation task. For evaluation, we use random dot stimuli with the same motion patterns as the original videos, but for which the appearance of each individual frame is informative (example video in the supplemental material). The neuroscience inspired model generalizes to these stimuli much better than state-of-the-art optical flow models.
Figure 1:
Nevertheless, humans were able to classify the animals and objects in these videos based on motion information alone.
In this work, seek understand the computational basis for appearance- motion perception in humans which enables this zero-shot generalization to random dots. Recent advancements in computer vision models for motion segmentation enable accurate segmentation of moving objects in natural videos based on a combination of optical flow estimation networks with downstream segmentation networks (e.g., (51)). However, it remains untested whether these models generalize in similar way as human perception. Since the motion estimation stage is critical for segmenting moving objects, focus testing a broad range of state-of-the-art optical flow methods in combination with a fixed segmentation network. Our analysis reveals that existing computer vision approaches do generalize in a human-like manner: Many stimuli perform near chance level for random dots.
models on natural
In the primate visual cortex, area MT is known be involved in motion perception and interpretation. Computational models for this area based on motion energy were proposed almost forty years ago 46 and since then have been shown to predict key characteristics of neural firing patterns (37). Instead of matching deep features between two frames, these models rely on spatio-temporal filtering in pixel space combined with post-processing stage resolve ambiguities. We demonstrate that this mechanism can be successfully integrated with deep neural networks for motion segmentation in realistic videos and reaches the performance of early deep learning based optical flow models the original, textured vidcos-which is remarkable considering that the motion energy model was developed to explain the tuning of individual neurons and has several orders of magnitude fewer parameters than typical optical flow networks. Crucially, the motion energy model substantially outperforms all tested optical flow models in zero-shot generalization to moving random dots. In direct comparison with humans in controlled psychophysics study, the motion energy based approach is the only model that can match human capability.
In summary, our paper makes the following contributions:
We show that a broad range of state-of-the-art optical flow methods do not support human-like motion segmentation that generalizes to random dot patterns.
We demonstrate that a classical neuroscience model can be successfully integrated with deep neural networks and generalizes to random dot stimuli.
We conduct psychophysical experiment to directly compare random dot motion segmenta- tion in humans and machines. While state-of-the-arr optical flow models fall short of human performance, the motion energy model can match it.
These results establish compelling link between the Gestalt psychology of human object perception and cortical motion processing in the brain, showing that motion energy approach can overcome lack of human-like zero-shot generalization to random dot stimuli in current computer vision models. Integrating this mechanism with state-of-the-art optical flow methods promising path towards more robust motion estimation models.

---

2
Related Work
Motion energy. Modelling motion perception in humans has been frequently approached using motion energy models. These models exploit the fact that a moving pattern corresponds to oriented edges when considering a video as spatio-temporal volume []. 46]. Several models have been proposed that build on this principle, aiming to explain the tuning properties of neurons found in visual areas V1 and MT 37 15
32 28 With few exceptions [1 38), these models have not been used as a motion estimation models computer vision context. Our work the first to study motion energy models for moving object segmentation.
Optical flow estimation. Optical flow traditionally has been formulated an optimization problem with the goal of finding good matches between two frames [6]. During recent years, optimization based methods have been superseded by deep neural networks that frame optical flow estimation as an end-to-end regression task. FlowNet [11] pioneered this approach with a CNN that optionally includes an explicit temporal matching operation. Following works contributed better training data and proposed coarse-to- co-se-to-fine -ine architectures to predict optical flow [77 39 40 which lead to substantial
performance improvements. More recently, models that iteratively refine high resolution optical flow map 43 [8] and Transformer-based models [36 53 54 have further improved state-of-the-an
Some works have compared optical flow models to human motion perception [56] 41. however not in the context of motion segmentation.
Motion Segmentation. The typical approach to motion segmentation is using optical flow as input for downstream segmentation model. One classic line of work computes point trajectories from optical flow and then clusters the trajectories to segment moving regions 6 geometric approaches to motion segmentation have been combined with deep learning in later work 4 []. More recently. purely deep learning based approaches have been able to improve state-of-the- [1]. To achieve high performance on classical motion segmentation datasets, the optical flow based motion segmentation is typically combined with appearance based segmentation [9, 52). In this work, we evaluate generalization to random dot stimuli for which appearance is not informative, so we focus on purely motion-driven approaches.
Classical
art [44] |9] 22 23
Methods
The aim of this work is to evaluate which computational models match the capabilities of humans for zero-shot motion segmentation of random dot patterns. We follow the standard motion segmentation approach in computer vision and first use a motion model to estimate the motion in an input video, followed by segmentation network that predicts the foreground mask. In order for models to perform well on zero-shot segmentation of random dot patterns, it is critical that the motion estimator used by the model generalizes well to these random dot stimuli. Ideally, the motion estimator would be invariant to changes in texture. Therefore, we focus on the motion estimation stage by evaluating broad range of optical flow models in comparison to a neuroscience inspired motion energy model. As a segmentation model, we use the same segmentation architecture for all motion estimators which we train from scratch for every model.
3.1 Optical Flow Models
We use a range of optical flow models that includes all major deep learning based approaches to optical flow estimation. FlowNet 2.0 [17] was the first CNN based model that reached the performance of classical, optimization based methods. We consider three variants of the model using different combinations of subnetworks. PWC-Net [39] introduced a multi-scale approach that combined operations from classical approaches (such as cost volumes and warping), with components from deep learning. Different from previous models, RAFT 43 not based on a coarse-to- -oosse-o-ffie approach but rather on iterative refinement of high resolution optical flow map derived from multi-scale correspondences. GMA [18] extends the RAFT architecture by introducing a Transformer-baser module to better handle occlusions, which have been shown to be difficult for previous models. More recently, GMFlow [53 54 and FlowFormer++
(36 have been proposed as fully Transformer-basee
architectures for optical flow estimation.
3

---

We use the implementations and checkpoints of these models from the MMFlow library E1, except for FlowFormer++ and GMFlow for which we use the implementations and checkpoints provided by the respective authors For each architecture, we consider checkpoints trained on different datasets that are common in the field, including the FlyingChairs [T], FlyingThingss [24]
Ii
Sintel 7 and
KITTI 23 266. In total, we evaluate 40 optical flow models.
We apply the models to prediet multi-scale optical flow, in order to match the multi-scale features predicted by the motion energy model. All of the optical flow models internally use several scales to predict optical flow. However, this representation is followed by non-trive non-trivial processing to combine motion information across scales, so that using this internal representation directly would most likely lead to inferior performance. Therefore we use the unmodified models and scale the final optical flow prediction to the desired resolutions using bilinear interpolation.
3.2 Motion Energy Model
Motion energy models are based on the insight that a motion pattern in a video corresponds to a spatio-temporal orientation when the video is considered at every pixel can therefore be estimated by using spatiotemporal filters that respond to particular motion direction and speed. This mechanism has important differences from the optical flow models discussed before. All of the optical flow models compute deep features for two frames individually and match these features between two frames estimate motion. The spatio-temporal filters in motion energy models on the other hand operate directly in pixel space. This approach leads to more ambiguous matches, which are typically resolved by considering more than two frames, and postprocessing stage.
z-y-l volume [ [6) The motion
In this study, we build on the influential motion energy model by Simoncelli & Heeger 377 In addition to the oriented filters described above, this model introduced a second stage that implements an intersection of constraints construction 13 2 in order resolve ambiguities of the linear filter responses. This motion energy model can be implemented as a CNN with the architecture shown in Figure 2 We derived the weights of the CNN from the the parameters of the original model and verified that our PyTorch [30] implementation of the motion energy model equals the original MATLAB implementation up to numerical differences. Following the original model, we apply the model for five different input scales that are obtained by repeatedly blurring and downsampling the input by factor of two. To streamline the implementation, we do not scale the activations after every layer and experimentally verified that this change does not affect downstream performance for motion segmentation.
3.3 Segmention model
We use a coarse-to-fine segmentation network to predict per-pixel logits for the respective pixel belonging to the foreground object (Figure ( Input to the segmentation model are the multi-scale motion energy maps multi-scale optical flow maps as predicted by the models described earlier. At each scale, the segmentation model consists of three components: The input projection layer predicts motion features for each scale. The core of the network is refinement CNN that aggregates features across scales, At each scale, the refinement CNN concatenates the motion features from the current scale with the refined representation from all previous scales and predicts the refined representation for the current scale. Finally, the output projection layer predicts the segmentation given the refined representation from the finest scale. All layers except for the output projection are followed by CELU nonlinearity [3] and instance normalization [45. The parameters of the components are shared across the stages, so that the network is essentially a recurrent neural network that integrates information from coarsest to the finest scale in order to predict segmentation.
3.4 Training
All models are trained on a synthetic video dataset that we generated using the Kubric library [1] Each video shows a single moving object in front of a moving background. The 3D objects and
https://www. cns www. wtt..uu..ct///cc///ooeeee.

---

Segmentation
Model
Motion Esimation Modal
Segmentation
Sepmentation
Model
Motion Energy Model
Segmentation Model
Video TelnK
Gonw Rc1xt
Linear
Gonv Txtst
Linear
from presious scale
Conw 1x8x1
Gaussian
Corw 1x13ac
Derivative
Gauestan Fitter
Conw 1x1x2
Filter
Inttar Womm
Conn Concalenrss
Recified Square
Conv
Conr
Channahhomm
Gonu 1x11xr
V1 Blur
Goussian
Cony Tarkx11
Fitter
mext aciée
Charne/Nomm
Figure 2: (top) Our motion segmentation architecture: The motion estimation predicts multi-scale optical flow or motion energy, the segmentation model predicts the moving foreground region. (bottom left) The motion energy model implemented as a CNN. The weights are chosen such that the CNN is equivalent the original model by [37). (bottom right) The segmentation model combines motion features across scale and predicts a binary segmentation at the input resolution.
backgrounds used for dataset generation are scans of everyday objects and scenes, resulting in highly realistic renderings. We used 901 videos for training and 100 test videos, each having 90 frames at 30Hz. The training and test videos used different sampled from the same distribution. The code and hyperparameters for generating videos, as well as the rendered dataset, are publicly available")
of object and backgrounds but are otherwise
For all models, we freeze the weights of the motion estimator and only train the downstream per pixel binary cross segmentation network. As common for binary motion segmentation, [21] with a learning rate entropy to the ground truth masks as loss. We use the Adam optimizer
of le 4 for all models and train for 40.000 steps using batch size of 8. All models
trained on NVIDIA GeForce RTX 2080 Ti GPUs with 12GB of VRAM. Depending on the computational requirements of the motion model, training the segmentation model single GPU takes between and 6 hours.
3.5 Zero-shot evaluation on random dot stimuli
We evaluate models on the original test videos well random dot stimuli generated for all test videos based on the ground truth optical flow. We use the same procedure as [3I] for generating random dot stimuli using 500 dots with lifetime of 8 frames, which matches the dot density and lifetimes.
We apply all models using a shifting window approach for the full length videos, but excluding the first and last four frames that the window is fully contained within the video for all models. For evaluation, we obtain binary prediction by thresholding with 0.5 and measure performance by computing IoU and F-Score for each frame individually and then averaging over the test set.

---

Results
4.1 Zero-Shot Random Dot Segmentation
Table I summarizes the motion segmentation performances achieved when using different motion estimators, both on the i.i.d. test set and the corresponding random dot stimuli. For better overview, we visualize the performances as measured by loU in Figure I and in the appendix.
Original
Random Dots
Motion Estimator
Training Dataset
loU
F-Score
IoU
F-Score
Motion Energy (ours)
0.759
0.845
0.600
0.718
GMFlow
Flying Things 3D
0,885
0.925
0.381
0.493
(2 scales, 6 refinements)
Sintel
0.823
0.874
0.222
0.315
Mixed
0.823
0.874
0.069
0.106
FlowNet2 SD
FlyingChairs
0.828
0.884
0.377
0.499
FlowNet2 CSS
FlyingThings!?
0.837
0.896
0.351
0.469
FlyingChairs
0.735
0.818
0.252
0.359
PWC-Net
ThiiggT
0.729
0.815
0.347
0,469
FlyingChairs
0.544
0.662
0.324
0.442
KITTI
0.600
0.715
0.121
0.193
FlowNet2
FlyingChairs
0.662
0.757
0.278
0,380
FlyingThings3D
0.821
0.877
0.154
0.241
FlowNet2 CS
Flying ThingsDD
0.800
0.867
0.269
0.374
FlyingChairs
0.669
0.760
0.247
0.359
GMFlow (2 scales)
Flying Things 3D
0.704
0.793
0.267
0.373
Sintel
0.733
0.812
0.253
0.351
Mixed
0.743
0.822
0.210
0.300
GMFlow (1 scale)
Mixed
0.843
0.897
0.225
0.308
Flying Things 3D
0.797
0.859
0.174
0.244
GMA (+P)
KITTI
0.520
0.630
0.154
0.224
FlyingChairs
0,646
0.717
0.101
0.142
Mixed
0.895
0.932
0.054
0.078
Flying ThinggTh
0,850
0.894
0.039
0.057
FlowFormee++
Flying Chairs
0.741
0.800
0.144
0.218
Flying Things 3D
0.901
0.935
0.119
0.182
Sintel
0.908
0.942
0.092
0.140
Flying Things 3D
0.902
0.938
0.072
0.113
GMA
KITTI
0.579
0.679
0.143
0.214
FlyingChairs
0.643
0.724
0.134
0.194
FlyingThings3D
0.867
0.909
0.100
0.150
FlyingThings3D Sintel
0.890
0.928
0.089
0.132
Mixed
0.890
0.927
0.077
0.
GMA (P-only)
FlyingChairs
0.663
0.743
0.138
0.207
KITTI
0.567
0.650
0.112
0.159
Mixed
0.881
0.920
0.093
0.141
FlyingThingss
0.867
0.909
0.090
0.142
RAFT
FlyingThings3ID Sintel
0.886
0.924
0.132
0.180
Flying Things3hing
0.869
0.909
0.116
0.172
Mixed
0.885
0.925
0.108
0.145
KITTI
0.506
0.600
0.107
0.147
FlyingChairs
0.647
0.718
0.100
0.147
Table 1: Model performances for the i.i.d. test videos and zero shot to the corresponding random dot
stimuli with the same motion patterns. For all motion estimators, the same segmentation network is used to predict the and ordered by the performance the random dot stimuli.
segmentation. Results are grouped by the motion estimation model
Recent optical flow methods perform strongly on the original videos. FlowFormerse works best on our dataset with an IoU of 90.8%, closely followed by GMA variant that reaches 89.5% IoU.

---

These results parallel the strong performance of recent Transformer-basee architectures on standard optical flow benchmarks. The motion energy based model only achieves performance of 75.9% IoU and lags behind state-of-the-art optical flow models, but performs similar as earlier deep learning based optical flow models. This result remarkable when considering that the motion energy model predates the deep learning models by several decades and has not been tuned for dense, end-to-end motion prediction. Within each model, the checkpoints from the FlyingThings22 dataset tend to perform best for the original videos. The FlyingThings: dataset contains renderings of 3D objects undergoing rigid motion, so arguably it is the most similar dataset compared to the one used this study.
Motion energy generalizes much better to random dots. The motion energy based model reaches an IoU of 60.0%, which outperforms the performance of the second best model by more than 20 percentage points. Strikingly, the FlowFormer++ and GMA models that performed best on the original videos generalize particularly bad to the random dot stimli (IoU < 10%). Overall, more dated optical flow architectures such as FlowNet2 variants and PWC-Net tend to generalize better to random dot stimuli than more recent approaches. An interesting exception is GMFlow, which reached an IoU of 38.1% 1% and performed best among all optical flow models. We do not observe a clear effect of the training dataset.
We visualize model predictions in Figure 3 For the original videos, the quality of the predicted optical flow varies but allows for a clear segmentation of the moving object. The object is also clearly represented in the motion energy maps, with some feature maps responding highly to the background and others to the moving object. The motion energy maps however tend to be noisier than the optical flow predictions, which explains the lower performance of the motion energy model for the clean videos.
The random dot stimuli exhibits the same motion as the original video, so the prediction of an ideal motion estimator would be unchanged. The optical flow methods however fail to properly estimate the motion of the foreground object. While some methods like FlowNet 2.0 and PWC-Net predict highly noisy motion pattern that roughly matches the location of the foreground object, many optical flow estimators fail to detect the foreground motion at all. The motion energy on the other hand looks highly similar for the random dot stimulus and the original video, allowing the motion energy segmentation to generalize well in this case.
4.2 Ablation study
As an ablation study, we evaluated whether the performance of the motion energy segmentation model can be improved by learning the parameters of the motion model. We tested different combinations of layers in the motion energy CNN that are fixed, finetuned or trained from scratch and trained them end-to-end with the segmentation model.
The results in table that the original weights of the model allow for the best generalization to random dots. This remarkable when considering that the weights of the motion energy model have been originally selected to explain the tuning properties of individual neurons, but not for image-computable motion estimation. Some of the configurations however outperformed the original weights on the original videos. So while the network architecture allows for generalization in principle, all our models trained by gradient descent converged to solutions that performed well on the training data but did not generalize.
As further ablation study, we removed or replaced layers of the motion energy model. The results in the supplemental information suggest that the pooling and normalization layers are particularly important for generalization to random dots. More details and further experiments are provided in the supplemental information.
5 Human Machine Comparison
The previous results have revealed differences between different motion estimation models in terms of generalization to random dots. While known that humans can recognize objects in random dot stimuli without prior training (31. the ability to segment objects in moving random dot patterns has not been quantified before. We therefore conduct human subject study in order to directly compare the zhoo generalization to random dots in humans and machines.
7

---

Optiaa sow.
Sang
Oppccal Hoxe
Molton Lawegy
Modton Lawegy
Segmentation
Cale
Figure 3: Example predictions for different motion estimators. The motion pattern in the random dot stimulus is the same in the original video. While the optical flow estimates highly accurate for the original videos, the models struggle with the random dot stimuli that exhibit the same motion. The activations of the motion energy model model however generalize well the random dot stimuli, enabling to detect and segment the foreground object.
Original IoU
Random Dots
VI Linear
V1 Blur
MT Linear
MT Blur
F-Score
IoU
F-Score
fix
fix
fix
fix
0.759
0.845
0.600
0.718
fix
fix
finetune
fix
0.776
0.856
0.563
0.686
finetune
fix
finetune fix
0.804
0.873
0.468
0.599
fix
fix
scratch scratch
0.794
0.873
0.463
0.583
finetune fix scratch scratch
0.827
0.887
0.395
0.508
finetune finetune scratch cratch
0.600
0.717
0.162
0,246
scratch fix scratch fix scratch
0.660
0.752
0.052
0.087
scratch scratch
scratch
0.593
0.702
0.027
0.048
Table 2: Comparison of using the original weights (fix), finetuning the original weights (finetung) training from scratch (scratch) for the layers the motion energy model.
Due to the inherent difficulty in directly evaluating the segmentation perceived by humans, we employed a shape identification task as a surrogate requiring segmentation (Figure 4p. Each trial involved a random target shape and distractor shape from the Infinite dSprites dataset [12]. The random dot stimulus shows the target shape moving linearly across the center of the image with random motion direction and speed. After the video concluded, participants were shown clean renderings of the target and distractor shapes and were required to select the shape that they perceived in the random dot stimulus. Since the shape alternatives were unknown while the random dot stimulus was shown, participants had to segment and memorize the shape compare it to the shape choices afterward. Therefore, performing well on this task necessitates sufficiently good segmentation of the moving shapes within the motion patterns of the random dot stimuli.
random dot video and then

---

Human
Figure 4: We compare humans and machines using random dot shape identification task proxy to measure segmentation in humans. Shown a video of random dots, participants have to respond which of two shapes perceived in the video. Humans outperformed all optical flow based models, but not the motion energy based model for this task. More details are provided in the supplemental material.
We performed the study in controlled vision lab environment, where participants viewed the experiment on VIEWPixx 3D LCD monitor (1920x 1080, 120Hz) with the distance fixed to 65cm using a chin rest. The duration of all videos was 1s at framerate of 30 Hz. Overall, collected data from N=13 subjects, of which we excluded one subject due to insufficient visual acuity (remaining: N=12, 4 female, male). Among the subjects where both trained vision scientists and naive subjects.
We evaluated all models on the same stimuli as human subjects. Given a random dot video, we applied the respective model segment the video and selected the shape option that better matched the prediction as measured by IoU.
The results in Figure show that all models based optical flow are clearly outperformed by humans. Many of the optical flow based models perform near chance level, while some models reached a non-trivial performance. Overall, more recent optical flow models that perform very well on the original videos appear to generalize worse to this task, with GMFlow [53] [4] being a noteable exception. Different from the optical flow models, the motion energy based approach is the only model to match and even outperform human performance. More detailed results the supplemental information show that the motion energy segmentation model performs par with the highest performing participants of the study.
Limitations
To allow for comparing large number of motion estimation models with reasonable computational budget we made compromises for other modeling aspects. We limited the size of the segmentation network to allow for efficient training but performed a control experiment to show that using more sophisticated segmentation network does not improve generalization (see supplemental information). Moreover, we used the same training schedule for all models but ensured that our setting supports all models adequately by visually inspecting the loss curves.
When comparing humans and machines we did not model several factors that are expected to influence human performance, such as the impact of internal noise and attentional lapses. As common in psychophysies experiments, several subjects reported making accidental errors for few examples (50) which negatively affects performance. So even a model that perfectly replicates the motion processing algorithm humans not expected perfectly replicate human behavior our setting.
7 Broader impact
This work is highly interdisciplinary, bridging state-of-thee art computer vision motion segmentation algorithms with the principles of Gestalt psychology and the neuroscience of cortical motion pro- cessing in the brain. By showing that neuroscience-ingse conventional optical flow models zero-shot generalization to random dot stimuli, study high- lights the potential for integrating biological insights into AI systems. Benefits of broader impact include the development more robust and human-like systems, educational value, and the creation of AI systems that are more aligned with human cognition.
motion energy model can outperform

---

8 Discussion
Computational models for motion estimation have long history in both computational neuroscience and computer vision. Shallow models based on spatio-temporal filtering in pixel space have been able to predict neural activity in brain areas related to motion perception [37 32 and are compatible with range of phenomena in human perception [52]. In computer vision, models based on matching deep features between two frames have continuously improved performance over the last years and are successfully applied in range of downstream tasks. Despite these successes, our study reveals a striking gap between deep optical flow networks and human perception: While humans generalize the common fate Gestalt principle to zero-shot segmentation of random moving dots, the optical flow models fail to generalize to these stimuli. Furthermore, we show that a classic motion energy approach can be scaled to realistic videos while matching human generalization capabilitiess
The great success of deep neural networks in computer vision has spawned interest in using DNNs also as a model for human vision, in particular for core object recognition [55]. In the same spirit, deep neural networks might be promising models for human motion perception [56]. While promising. our study parallels findings for core object recognition that show striking differences between human perception and DNNs 49]. For motion perception, however, we show that is possible to combine classical models from computational neuroscience with the scalability of deep learning. Further integration of these modeling traditions is a promising path towards image computable models of human motion perception [41].
While closing the gap between human perception and machine vision is crucial for computational neuroscience, we believe that computer vision likely profits from better alignment with human vision as well. Humans still greatly outperform machines in terms of robustness and efficiency. Our study suggests a substantial entanglement of motion estimation with appearance in DNNs, which might also be linked to the lack of robustness observed in state-of of-the-art motion estimators 33 34] Computational principles that better match human vision should be considered as promising candidates for addressing these issues.
Finally, we argue that deep learning based models as presented in our work have the potential to greatly improve our understanding of motion perception in humans. Low-level mechanisms for motion estimation and higher level processes for motion interpretation have been mostly studied in isolation (27]. In our work we follow a more holistic approach by studying the effects motion detection mechanisms on the perception of moving objects, which offers several unique opportunities. First, it not necessary for most downstream tasks to perfectly estimate the physically correct motion. For example, segmenting moving objects does require precise information about object boundaries while other mistakes are less critical. Studying motion estimation and interpretation jointly allows to better understand viable compromises in estimation accuracy as the basis for more efficient processing. Second, studying end-to-end models of motion estimation and interpretation advances our understanding of how neural mechanisms give rise to behavior. DNNs are particularly promising modeling approach positioned in a "Goldilocks zone" regarding the trade-off between biological plausibility and scalability to natural stimuli and tasks [10. In this vein, our work establishes compelling link between cortical mechanisms for motion estimation and the Gestalt psychology of human object perception.
In the future, this work can be extended in several directions. While scaling remarkably well, the original motion energy model is not able to match the performance of -ate-offtthe-at optical flow methods on natural scenes. We see integrating principles from computational neuroscience with techniques from deep learning as promising path towards closing this gap 41). Moreover, training the parameters of the CNN implementation of the motion energy model jointly with the segmentation model did not lead to generalizable solution. How humans learn generalizable motion perception from data, or to which degree this capability is innate, are important questions for future research. Finally, in the spirit of the neuroconnectionist research programme [10] we see model as an executable hypothesis for motion perception in the human brain. While matching human performance in terms of generalization to moving random dots, this model might well fail to capture other aspects of human motion perception. Further evaluating and extending models of motion perception to capture diverse range of phenomena is exciting path towards holistic understanding of human perception.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the German Research Foundation (DFG)): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP 4, project number: 276693517. The authors thank the International Max Planck Research School for Intelligent Systems for supporting MT. We thank Felix Wichmann, Thomas Klein and all other members of the Wichmann-Lab for supporting and testing the human machine comparison study, and Larissa Höffing for valuable feedback on the manuscript.
References
[1] E. H. Adelson and J. R. Bergen. Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America A, 2(2):284-299) February 1985. doi: 10. 13.1364/ JOSAA.2.000284,
[2] E. H. Adelson and J. A. Movshon. Phenomenal coherence of moving visual patterns. Nature, 30065892):523-525, December 1982. doi: 10. 10.1038/300523a0
[3] J. T. Barron. Continuously Differentiable Exponential Linear Units. arXiv preprint arXiv:1704.00 07483, April 2017. doi: 11t.5/0/aaXXi...4.744440..
[4] P. Bideau and E. Learned-Miller. It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos. In Computer Vision ECCV 2016, pages 433-449, Cham, October 2016. Springer International Publishing. doi: 10. 00.1007/978-3-31-4444488
[5] P. Bideau, A. RoyChowddury, R. R. Menon, and E. Learned-Mlllrr. The Best of Both Worlds: Combining CNNs and Geometric Constraints for Hierarchical Motion Segmentation. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 508-517, June 2018.
[6] T. Brox and J. Malik. Object Segmentation by Long Term Analysis of Point Trajectories. In Computer Vision ECCV 2010, volume 6315 of Lecture Notes in Computer Science, pages 282-295, Berlin, Heidelberg, September 2010. Springer. doi: 10. 10.10077978-3-642-111 1455555-----1155--01111
[7] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A Naturalistic Open Source Movie for Optical Flow Evaluation. In Computer Vision ECCV 2012, volume 7577 of Lecture Notes in Computer Science, pages 611-625, Berlin, Heidelberg, October 2012. Springer. doi: 10.1007/978-3-6023333333
[8] M. Contributors. MMFlow: OpenMMLab Optical Flow Toolbox and Benchmark, 2021.
[9] A. Dave, P. Tokmakov, and D. Ramanan. Towards Segmenting Anything That Moves. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) Workshops, October 2019.
[10] A. Doerig, R. P. Sommers, K. Seeliger, B. Richards, Ismael, G. W. Lindsay, K. P. Kording, Konkle, M. J. van Gerven, N. Kriegeskorte, and T. C. Kietzmann. The neurcconectioniist research programme. Nature Reviews Neuroscience, (4(7):4314500 July 2023. doi: 10.1038/ -215833-23--07755...
[11) A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, and Brox. FlowNet: Learning Optical Flow With Convolutional Networks. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 2758-2766, December 2015.
[12) S. Dziadzio, Ç. Yıldiz, G. M. van de Ven, T. Trzciáski, T. Tuytelaars, and M. Bethge. Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization. In 3rd Conference on Lifelong Learming Agents (CoLLAs), July 2024.
[13] C. L. Fennema and W. B. Thompson. Velocity determination in scenes containing several moving objects. Computer Graphics and Image Processing, (4):301-335, April 1979. doi:
11

---

[14] K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. Fleet, D. Gnanapragasam, Golemo, C. Herrmann, T. Kipf, A. Kundu, D. Lagun, I. Laradji. H.-T. D. Liu, H. Meyer. Y. Miao, D. Nowrouzezahrai, C. Oztireli, E. Pot, N. Radwan, D. Rebain, S. Sabour, M. S. M. Sajjadi, M. Sela, V. Sizzman, Stone, D. Sun, Vora, Z. Wang. T. Wu, K. M. Yi, Zhong, and A. Tagliasacchi. Kubric: A Scalable Dataset Generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3749-3761, June 2022.
[15] D. J. Heeger, F. P. Simoncelli, and J. A. Movshon. Computational models of cortical visual processing. Proceedings of the National Academy of Sciences, 93(2):623-627, January 1996. doi: 10.1073/pnas. 99.22622......
[16] B. K. P. Horn and B. G. Schunck. Determining optical flow. Artificial Intelligence, 17(1): 185-203, August 1981. doi: 11101660004-37028811900000
[17] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. FlowNet 2.0: Evolution of Optical Flow Estimation With Deep Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2462-2470, July 2017.
[18) S. Jiang, D. Campbell, Y. Lu, H. Li, and R. Hartley. Learning To Estimate Hidden Motions With Global Motion Aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9772-9781, October 2021.
[19] G. Johansson. Visual perception of biological motion and model for its analysis. Perception & Psychophysics, 14(2):201-21 June 1973. doi: 10.3758/BF03212378.
[20] M. Keuper, B. Andres, and T. Brox. Motion Trajectory Segmentation via Minimum Cost Multicuts. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3271-3273 December 2015.
[21] D. Kingma and Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations (ICLR) 2015. arXiv, May 2015. doi: 10.48550/arXiv. 1412.6980.
[22] H. Lamdouar, C. Yang, W. Xie, and A. Zisserman. Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020.
[23] H. Lamdouar, W. Xie, and A. Zisserman. Segmenting Invisible Moving Objects. In 32nd British Machine Vision Conference (BMVC), November 2021.
[24] N. Mayer, E. 1lg. P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A Large Dataset Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Esti- mation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040 4048, June 2016.
[25] M. Menze, C. Heipke, and A. Geiger. JOINT 3D ESTIMATION OF VEHICLES AND SCENE FLOW. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, I1-3/W5:427-44 4334, September 2015. doi: 10.5 00.55994/sprsannal.. 11-3--1-------- 25-42720015
[26] M. Menze, C. Heipke, and A. Geiger. Object Scene Flow. ISPRS Journal of Photogrammetry and Remote Sensing, 140:60-76, June 2018. doi: 10. 00016/..sppspprs.2217......111
[27] S. Nishida, T. Kawabe, M. Sawayama, and T. Fukiage. Motion Perception: From Detection to Interpretation. Annual Review of Vision Science, 4(1):501-523, September 2018. doi: 10.1 (9tp:urreeeiiiiii0011111111
[28) S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, and J. L. Gallant. Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Current Biology, 21(19):
1 1641-1646, October 2011. doi: 10. 20.101 100-220011..00033 108.031.
[29] P. Ochs and T. Brox. Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions. In 201 International Conference on Computer Vision, pages 1583-1590 November 2011. doi: 10.1 10.110//CCC22011111226113
12

---

[30] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, Kopf, Yang, Z. De Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang. J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., December 2019.
[31] S. Robert, L. G. Ungerleider, and M. Vaziri-Pashkan Disentangling Object Category Represen- tations Driven by Dynamic and Static Visual Input. Journal of Neuroscience, 43(4):621-634
January 2023. doi: 111122//NNUUROSS11331-222222
[32] N. C. Rust, V. Mante, E. P. Simoncelli, and J. A. Movshon. How MT cells analyze the motion of visual patterns. Nature Neuroscience, 9(1 1) 14:11111143,, November 2006. doi: 10. 10.1038/ 786.
[33] Schmalfuss, L. Mehl, and A. Bruhn. Attacking Motion Estimation with Adversarial Snow. In ECCV 2022 Workshop Adversarial Robustness in the Real World, October 2022.
[34] J. Schmalfuss, P. Scholze, and A. Bruhn. A Certurbatiion-Consrriinn for Evaluating the Robustness of Optical Flow. In Computer Vision - ECCV 2022, pages 183- 200, Cham, October 2022. Springer Nature Switzerland. doi:
Adversarial Attack
[35] H. H. Schütt, S. Harmeling, J. H. Macke, and F. A. Wichmann. Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data. Vision Research, 122: 105-123, May 2016. doi: 10.1016/j. wwwp:.......ss.........
[36] X. Shi, Z. Huang, D. Li, M. Zhang, K. C. Cheung, S. See, H. Qin, J. Dai, and H. Li. Flow- Former++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1599-1610, June 2023.
[37] E. P. Simoncelli and J. Heeger. A model of neuronal responses in visual area MT. Vision Research, 38(5)7743-761, March 1998. doi: 10. 10.1016/S0042. 6981
(38) F. Solari, M. Chessa, N. V. K. Medathati, and P. Kornprobst. What expect from V1-MT feedforward architecture for optical flow estimation? Signal Processing: Image Communication, 39:342-354, November 2015. doi: /000://.maae.2015....
[39] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9934-8943, June 2018.
[40] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 42(6):1408-1423 June 2020. doi: 1AL100/TAAM..201228945555
[41] Z. Sun, Y.J. Chen, Y.H. Yang, and S. Nishida. Modeling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network. In Advances in Neural Information Processing Systems, volume 36, pages 4333-24434.. Curran Associates, Inc.. December 2023.
[42] M. Tangemann, S. Schneider, J. von Kügelgen, F. Locatello, P. V. Gehler, T. Brox, M. Kümmerer, M. Bethge, and B. Schölkopf. Unsupervised Object Learning via Common Fate. In Proceedings of the Second Conference on Causal Learning and Reasoning, pages 281-327. PMLR, April 2023.
[43] Z. Teed and J. Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In Computer Vision - ECCV 2020, pages 402-419, Cham, August 2020. Springer International Publishing, doi: 11.100079788-3000--5553---
[44] "Tokmakov, C. Schmid, and Alahari. Learning to Segment Moving Objects. International Journal of Computer Vision, 127(3):282 301, March 2019. doi: 10. 1263-018-1181 1-11--2---22-2.
[45] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv preprint IrXiv: 1607.0802,, November 2017. doi: 10.48550/arXiv.11 1607. 08022.
13

---

[46] A. B. Watson and A. J. Ahumada. Model of human visual sensing. Journal of the Optical Society of America A, 2(2):322-342. February 1985. doi: 10.1364J00SAA.200003222
[47] M. Wertheimer. Experimentelle Studien über das Sehen von Bewegung. Zeitschrift für Psy- chologie, 61:161-265, April 1912.
[48] M. Wertheimer. On Perceived Motion and Figural Organization. The MIT Press, Cambridge, MA, July 2012.
[49] A. Wichmann and R. Geirhos. Are Deep Neural Networks Adequate Behavioral Models of Human Visual Perception? Annual Review of Vision Science, 9:501-524, September 2023. doi: 10.1 1 146/annurev-vision Setisionn120222-000 120522-031739.
[50] F. A. Wichmann and J. Hill. The psychometric function: I. Fitting, sampling, and goodness of fit. Perception Psychophysics, 63(8):1293-1311 November 2001. doi: 10.3755/BFF0114544
[51] Xie, W. Xie, and A. Zisserman. Segmenting Moving Objects via an Object-Centric Layered Representation. In Advances in Neural Information Processing Systems, volume 35, pages 28023-28036, Curran Associates, Inc., December 2022.
[52] Xie, W. Xie, and A. Zisserman. Appearance- Baseed Based Refinement for Object-Centric Motion Segmentation. arXiv preprint 1rXiv:2312.11463, August 2024. doi: 10.48550/arXiv.2312 11463.
[53] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao. GMFlow: Learning Optical Flow via Global Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8121-8 1121-8130, 8130, June 2022.
[54] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, and A. Geiger. Unifying Flow, Stereo and Depth Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (11):13941-1395) November 2023, doi: (0.1109/PAAM2202333009
[55] D. L. K. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo. Performance-optimize hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619 (119-86611-822 9-6244 May 2014. doi: 10. 1073/pnas. 1403122111.
[56] Y.-H. Yang. T. Fukiage, Z. Sun, and S. Nishida. Psychophysical measurement of perceived motion flow of naturalistic scenes. iScience, 26(12), December 2023. doi: 10.1016/j iic.iiici.00 2023.. 108307.
[57] J. Yates. Motion Illusions, December 2020. mo potioon on illlsiioss////o (accessed: 2024-10-24)
https://jake.. hwwpio/ioiiiiiiiio//ooooo.
14

---

Supplemental information
Additional defails about the results
B
Additional experiments
15
B.1
Importance of components of the motion energy model
15
B.2
Multi-trame optical flow
16
B.3
Comparison with state of the-art motion segmentation
16
Additional details about the human subject study
18
Comparison of humans and machines by example difficulty
18
Screenshots of the experiment
Additional details about the results
For additional overview, view visualize the segmentation performances random dot stimuli as reported in Table [
0.6
0.4
10U
0.3
0.0
(+P) (Flying Thingggn
GMA (+P) (Mixod)
GMA (Mixcc)
GMA (fyingTTings30 Sintel)
GMA (P-onty) (Flyingg Thingg3D)
GMA (P-only) (Mixed))
RAFT (FlyingChairs)
GMA (Flying Things3D) 3D)
GMA (+P) (Flying
RAFT (KIT))
RAFT (Mixed)
GMA Pn-nnyy) (XITT))
RAFT (FlyingThingsss
PWC Not (XITT))
RAFT RAFTiFyynng (Fyinn Things 3D Sirtel)
GMA (FlyingChairrs
GMA (Ponly (Fhinng
GMA (KITT))
FlowFamer+- (PlyingChairs)
GMA (-P) (ITT))
FlowNee22 (Flying Things3D)
Flow nowfo-eer++ (Mixed)
FlowFome--- (Sintel)
FlowFormrre+ (FijingThingsa))
FlowFormer (XITI))
FlowNat2 c5 (FlyingChairs
Flowhet2 css (FlyingChiirs)
Flowlee22 cs (Flying Things30 3D)
Flowitetz FlyingChhiiss
PWC Net (Pyingghhimi
PWC Net (Flying Thir gr3D)
FlowMer2 (Plying Thin (
Flowee22 SD (FyingChairss
Motion Energy H
Figure 5: Segmentation performances of the evaluated models on the random dot stimuli. Same data as in Table
B Additional experiments
B.1 Importance of components of the motion energy model
We conducted additional ablation study in order to better understand which aspects of the motion energy model are essential for generalization to random dot stimuli. We removed or replaced individual layers described in Table and trained the ablated models from scratch using in the same way as the baseline model.
The results in Table 2 hint at the normalization and pooling layers being important for generalization. When the Gaussian pooling layers are removed completely, performance on original videos slightly improves while the generalization to random dot stimuli is substantially reduced.
15

---

Replacing the squaring- based nonlinear layers with ReLU layers, however, hardly changes the model's performance.
Original
Random Dots
Condition
IoU f
F-Score
IoU 1
F-Score f
Baseline
0.759
0.845
0.600
0.718
Replace RectifiedSquare + ReLU (MT)
0.753
0.838
0.609
0.725
Replace Square - ReLU (V1)
0.770
0.854
0.536
0.663
Remove MT Linear
0.768
0.856
0.481
0.609
Remove MT
0.770
0.854
0.451
0.583
Remove Blur (V1, MT)
0.801
0.872
0.421
0.540
Replace ChannelNorm InstanceNorm (V1, MT)
0.592
0,703
0.230
0.340
Remove Normalization (V1, MT)
0.400
0,516
0.018
0.018
Table 3: Ablation study: Performance of the model on original videos and corresponding random dot
stimuli with various layers of the motion energy model removed or replaced. Results are ordered by IoU on the random dot stimuli.
B.2 Multi-frame optical flow
The motion energy model uses a window of frames as input, while typical optical flow methods estimate correspondences between only two frames. To rule the possibility that the results observed in our paper are mainly explained by the different input window lengths, we perform an ablation study in which apply optical flow methods using the same frame windows. For each window, we compute the optical flow between the central frame, for which the segmentation has to be predicted, to the 8 other frames in the window. The stacked optical flow fields then used the input to the segmentation network.
The results in Table 4 and Figure 6 show some improvement the original videos but wider gap to the motion energy model terms of of generalization to random dots. The differences between the motion energy and optical flow models therefore cannot be explained by the different input lengths.
loU
Optical Flow frames)
Random dots (zero shot)
0.8
Motion Energy
0.6
0.4
0.2
0.2
0.4
0.6
0.8
Original videos
Figure 6: Performance of multi-frame optical flow based models on the original videos and corre- sponding random dot videos.
B.3 Comparison with state-of-the-art motion segmentation
In our study we used a relatively small segmentation network downstream to the respective motion estimator. State-of-the-arr motion segmentation models typically target multi-object segmentation in real world videos and therefore use more complex segmentation networks. order verify that the results in our paper are not caused by using smaller segmentation network, we evaluated the state
16

---

Original
Random Dots
Motion Estimator
Training Dataset
IoU
F-Score
IoU
F-Score
Motion Energy (ours)
0.759
0.845
0.600
0.718
FlowNet2 SD
FlyingChairs
0.878
0.928
0.221
0.325
FlowNet2
FlyingChairs
0.808
0.868
0.209
0.300
FlyinTThing331
0.881
0.929
0.058
0.100
PWC-Net
FlyingChairs
0.816
0.886
0.163
0.250
0.825
0.886
0.137
0.221
KITTI
0.712
0.811
0.038
0.060
RAFT
FlyingThingss + Sintel
0.912
0.948
0.156
0,222
FlyingChairs
0.863
0.914
0.126
0.195
Mixed
0.896
0.934
0.117
0.164
FlyingThings3D
0.894
0.934
0.090
0.132
KITTI
0.714
0.794
0.031
0,053
FlowNet2 CS
FlyingChairs
0.841
0.899
0.137
0.220
Flying Things3D
0.847
0.904
0.075
0.129
GMA (+P)
FlyingChairs
0.856
0.912
0.132
0.212
Mixed
0.900
0.936
0.114
0.179
FlyingThings3ID
0.899
0.936
0.104
0.171
GMA
FlyingChairs
0.864
0.917
0.131
0.212
Mixed
0.900
0.937
0.090
0.139
FlyingThings3DD Sintel
0.909
0.943
0.066
0.100
FlyingThingg3D
0.903
0.943
0.060
0.098
KITTI
0.756
0.834
0.051
0.084
GMA (P-only)
FlyingChairs
0.846
0.901
0.128
0.207
KITTI
0.766
0.847
0.092
0.155
FlyingThings3D
0.903
0.940
0.083
0.139
Mixed
0,912
0.947
0.077
0.117
FlowNet2 CSS
FlyingChairs
0.850
0.908
0.084
0.141
FlyingThings331
0.862
0.918
0.070
0.121
Table Ablation study: We apply the optical flow estimators to window of 9 frames by using the central frame as references and computing optical flow each of the 8 other frames. The stacked optical flow fields are used as inpute for the segmentation network.
of the OCLR model [5] in our setting. The OCLR model optical flow estimated by RAFT [43. which we also included in our experiments. The segmentation network however uses a U-Net architecture with Transformer bottleneck and was trained to segment multiple objects on synthetic dataset. We use the published weights and do not retrain the model on our data.
The results in Table that the model performs very well the original data. OCLR outperforms our motion energy based model and achieves a performance similar to the best optical flow based models considered in this work. At the same time, the model does not generalize the corresponding random dot stimuli. These results provide further evidence that the low generalization to random dots due the architecture of the segmentation network the RGB training data, but property of the motion estimator.
Model OCLR
IoU (original)
IoU (random dots)
0.838
0.026
Motion Energy Segmentation
0.759
0.600
Table Comparison of the state-of-the-arr motion segmentation model OCLR, our segmentation model based on motion energy model.
17

---

C Additional details about the human subject study
C.1
Comparison of humans and machines by example difficulty
As measure of task difficulty, we count the number of informative dots. A dot is informative, if it is contained in either the target and distractor shape but not both (see Figure left). Only these dots allow discriminating between the different shapes.
We fitted psychometric curves for human participants and models function of the number of informative dots, using the psignifit toolbox [35]. The results in cigure that only the motion energy model is able match the performance of human subjects, especially for stimuli with medium number of informative dots.
Motion Energy
Proportion Correct
Number informative dots per frame
Figure 7: (leff) As measure of task difficulty, we count the number of informative dots that allow discriminating betwen the two shape alternatives. (right) Psychometric curves for humans, the motion energy based model and the four best optical flow models for the task as in g
Human
0.9
Optical Flow
Motion Energy
Acoursyy
0.7
Flowformere (hyingrharr)
GAV +P (lyingtheins)
PWCce (CTT)
GMA GMA (iinahinng33))
PATT [flyingCheer))
FlawFormere (FyinaThing:30)
Finearmers (Sinte)-
CHA (FiiinCraairs)
Faooturz Cs Ryyingaairs)
CHA i-any (Fying Iningst))
GIM hanryy Pyyingssiie
ONA (flynng Thingsl + Sintel)
CMMA P only (00 TD)
Golioow scale) (Mtxed)
GMA +p (Ntod))
RAFT (CTT))
gNia TT)
GMA +P (GTT)
GMA P wonniooaae
mnyiino (inee)
refnermen tu (noodo
(oxzt)
CMMFOW 12 scales, retter p Eeyinthha (ocoo CONFRo (2 scales, e infommmet (Sinee)
CM 12 scales) (site)
Foowteer css dnyingoraii poair))
Soveete so ggggann
CMROw scalis, rufirementt a (eTT))
coNtioN 12 scates) sostes))aa (Mized)
PWC Net (ftiigg tningggnngsss thingslo
Things33) (D)-
Hummm
Motion Energy
Reuder CSS (Pying Thingg
Mar (Pyinghn ond
momeiooon to a
amitron (2 scalesi (mying
CoM mon it soa nhwar
Floweee cs (1imm
Flowwed
Goffow scates a remi
RAFT sawiine
Figure 8: Comparison of the human and model performances for the random dot shape matching task.
18

---

C.2 Screenshots of the experiment
Figure 9: Screenshots from the human subject study on random dot shape identification. (top left) Instructions that were shown prior to the experiment. (top right) We showed 20 training trials during which subjects could familiarize themselves with the task. (bottom left) The training was followed by 500 test trials. A video with the random dot stimuli was shown first. (bottom right) Once the video finished playing, the two shape options were shown below.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the main contributions of our work in both the abstract and the introduction. All mentioned results supported by the experimental data presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper includes separate section that discusses the limitations of our work in detail, including limitations due to computational constraints.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the models, data and evaluation protocol used in the paper in detail. Additionally, the code, pretrained models and the contributed dataset are publicly released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
All code and data for the paper is publicly released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
/ //nips.co.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Hyperparameters and training details are explicitly reported with the descrip- tion of the models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Due to space constraints we did not include further statistical information in the main table. However we included Figure showing the same data with error bars in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We reported the computational resources of our models with the description of the training details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p////errrpssc.ppp/////cccco
Answer: [Yes]
Justification:
We have read the code of ethics and conform in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We are discussing potential broader impacts of our work in dedicated section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used the Kubric generator with the built-in asset library and range of pretrained models. We cited the sources of all data and implementations that we used.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
batasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The synthetic data generated for the paper is described in the paper, and published with the code used generate
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yss]
Justification: Screenshots of the experiment are included in the supplemental material.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The study in this paper does not pose any particular risk on participants. IRB approval exists.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Ra2j4iXzwd", "reviewer": "Reviewer_ezYE", "summary": "The paper measures the zero-shot generalisation of motion energy (as described and estimated by Simoncelli & Heeger (1998)) when applied to the moving object segmentation task on the random-dot moving patterns. The comparison is performed with optical flow representation. Given a synthetic dataset of single-moving object scenes, either motion energy or optical flow is estimated using the Simoncelli & Heeger method or of-the-shelf pretrained network, respectively. This is fed into the downstream segmenter, which is trained in a supervised manner. This is then applied to moving random dot images. While optical flow achieves better performance in the simulated 3D object videos, the performance degrades drastically on moving dots. Motion energy modality performs worse than many recent optical flow estimators in the original setting but retains much of the performance when transferred to moving dots.", "questions": "The rebuttal could concentrate on answering the critical question presented above in the weaknesses section. \n\n#### Additional Remarks\n - The size of the datasets and evaluation protocol for \"Original\" and \"Moving dot\" settings are not described. As the datasets are new, details are required to reproduce and verify the results.\n - How are the inputs to the segmentation network processed? Are the motion energy/optical flow normalised and/or transformed in some way?\n - On L78, it is said that the study of moving object segmentation is conducted, while the focus of the study is really the moving dot segmentation setting.\n - The presented motion energy model outperforms all human subjects in the study. While possible, this is quite strange and could indicate that the small sample size and/or task set up are inappropriate.", "limitations": "The key limitations are addressed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1) A multitude of optical flow estimators are tested, offering a wide coverage and interesting results that older, arguably, less data-driven approaches retained some performance on moving dots.", "weaknesses": "#### Presentation\n1) The paper lacks a description that explains precisely what motion energy is (what form it takes and what the inputs are to estimate it). Similarly, optical flow is not described either. This is problematic as a reader unfamiliar with either of these two concepts will have a hard time making sense of the study presented.\n\n#### Soundness\n\n2) It is also not clear if the presented comparison is meaningful and sound. In the discussion (L290-292), the paper states that optical flow might not be optimal for humans, and that the truthfulness of flow is not critical for the task of segmentation. Why compare to it?\n\n3) Similarly, as stated in the L302, motion energy is estimated from far more frames than flow. For moving dot patterns, the \"true\" flow is non-zero only for dots that have moved. This paper does not establish that this is enough to segment the shape. This is even before considering pretrained flow estimators operating outside the learned distribution. Similarly, the statement on L64 seems to misleading: optical flow and/or motion energy _are_ the motion information. It is just that optical flow does not work on moving dots whereas motion energy seems to still extract signal (perhaps due to many-frame observation). \n\n4) Does running optical flow on dense moving dot patterns even make sense? It is worth considering whether obtaining \"true\" optical flow between two frames is ill posed in this situation. Over a larger number of frames, it perhaps is the simplest explanation of the motion consistent with the overall trend. Given just two frames with indistinguishable dots, there could be many possible \"simple\" motions that rearrange the dots. It is perhaps too much to expect that the appropriate optical flow could be estimated with only two frames, and that the estimate will have enough information to describe the shape (which is essentially the overall trend of motion in these simple cases).\n\n5) There seems to be a premise in the paper that generalising to the moving dot patterns is desirable and important. This is not motivated. Is it even true? One would usually expect to apply computer vision systems more to the real-world setting than to moving dot setting. Based on the results in Table 1, recent optical flow estimators are preferable in this situation. Based on further study in Table 2, optimising the motion energy model weights to perform better in a more \"real\" setting reduces performance on moving dots. Why is the generalisation to the moving dot settings relevant? \n\n#### Limited Contribution\n7) While the presented study is novel, its usefulness is not argued or clear. It is not obvious how the findings could be used, which essentially are the measurements performed. The motion energy model is taken from prior works, does not improve with training in moving dot patterns and under-performs in the real setting. Overall, the advancement of the knowledge presented in the paper is minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful review. As argued in our general response, we see the main contribution of our work not to computer vision but to cognitive neuroscience; by establishing a compelling link between a biologically motivated model and high-level inductive biases of human perception.\n\n**Defining motion energy and optical flow.** Thank you for the suggestion. We will update our paper to review the concepts of motion energy and feature-matching based optical flow. In a nutshell, we’re comparing two approaches for motion estimation: Matching features between two frames and using spatio-temporally oriented filters (3d convolution). This is also the difference we ment in our statement in L.64, which we will clarify accordingly.\n\n**Comparison to optical flow estimators.** Deep learning based optical flow estimators have achieved remarkable accuracy during recent years. Due to this success, we find it natural to consider this class of models for our goal of building a model for motion perception in humans.\nWhile the goals of computer vision and modeling human vision differ in some aspects (for example regarding motion illusions) we believe that both lines of work are sufficiently close to be mutually beneficial.\n\n**Number of input frames.** The different numbers of frames used by motion energy and optical flow methods is a valid concern in our view. We therefore conducted an additional ablation study in which we applied optical flow using the same number of frames as motion energy. As is common for motion segmentation, we are using multiple gaps of [-4, -3, …, +3, +4] to the central frame and using the concatenated optical flow as input to the segmentation network. The results in the additional PDF show that optical flow models are unable to achieve the same generalization to random dot stimuli as motion energy even when using the same number of frames.\n\n**Relevance of random dot stimuli.** We agree that the “true” optical flow for random dots is ambiguous and believe that this makes random dot stimuli particularly interesting. The consistent perception of humans demonstrates that the sparse information of these stimuli is enough to perceive shapes, which reflects inductive biases of the underlying motion perception system. While the physically correct prediction might not be well defined, a good model for human perception clearly should reflect these biases. Moreover, random dot stimuli are well established in the literature on human motion perception. If models handle these stimuli similar to humans, this opens many opportunities for comparison with previous data.\n\n**Relevance for computer vision.** The main goal of our work is to improve models for human motion perception and interpretation. While clean videos are more relevant for practical applications of computer vision, random dot stimuli are a highly important stimulus class in human vision. Nevertheless, we think that this study also provides some directions for building better computer vision models. Given that the motion energy model has several orders of magnitude fewer parameters compared to the optical flow models (<2.5K), the model performs remarkably effectively. We hypothesize that the spatiotemporal filtering efficiently aggregates information from multiple frames, which is an operation that could well be included with computer vision models and help to identify more efficient architectures.\n\n**Contribution.** We argue that our study makes several valuable contributions for modeling human perception and fits the “neuroscience and cognitive science” subject area of NeurIPS very well.\n- The motion energy model was not developed as an image-computable model for end-to-end machine learning, but to explain the behavior of individual neurons. We show that this model can nevertheless be used as a DNN component for the computer vision task of motion segmentation.\n- Our knowledge of how humans organize motion patterns into a higher-level scene representation is still limited (e.g., Nishida et al. 2018). Our work makes a valuable contribution to this end by showing how a well understood, low-level mechanistic model of neural activity leads to inductive biases that agree with human scene perception.\n- Random dot stimuli have a long tradition in vision science. Nevertheless we are the first to measure segmentation of random dot stimuli in humans. We believe that this experimental data is a valuable contribution to the psychophysics of human perception.\nThe other reviewers seem to agree with us on these points and write that “this paper provides a valuable set of experiments that can guide future work in this interdisciplinary area” (834V), denote our work as a “Good contribution and novelty” (crYq) and posit that it “can open a new line of investigation to model human motion perception” (DKrM).\n\n**Additional Remarks.** Thank you for your feedback, we’re happy to improve the clarity of our paper in these regards.\n- Our new dataset consist of 901 training and 100 test videos, each having 90 frames at 30Hz. We use distinct backgrounds and foreground objects for both subsets. Models predict a binary segmentation with distinct foreground and background channels. So the IoU and F-Score metrics are computed as in for semantic segmentation, but using 2 classes.\n- Motion energy and optical flow are not normalized or transformed.\n- We argue that moving object segmentation is exactly the task which is targeted in our work, but are happy to clarify that we’re focussing on the generalization to random dot stimuli.\n- Compared to humans, our model does not suffer from accidental mistakes and slipping attention. Please also see our response to reviewer DKrM where we discussed this aspect in greater detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RtSqrPS0wr", "reviewer": "Reviewer_8R4V", "summary": "The authors attempt to use a classical evaluation of human gestalt perception, namely shape identification in moving random dot stimuli, to evaluate both a traditional model of motion perception (motion-energy model from Simoncelli and Heeger) against state-of-the-art deep learning-based optical flow methods. They find that while SoTA methods outperform the MT motion-energy model on natural video optical flow estimation, the motion-energy model in fact outerperforms these more complex methods in correctly estimating motion patterns in random dot stimuli. The poor estimation of motion in random-dot stimuli thus correlates with poor figure-ground segmentation using the SoTA deep network approaches. Due to the fact that human performance on random-dot stimuli is far greater, this suggests that curren optical flow models fall short of human performance and might be computing motion information in a way that is misaligned with human vision. On the other hand, the fact that the motion-energy model can fill this gap suggests that this more biologically-inspired model (that fits responses of MT cells in visual cortex) is sufficient to connect cortical motion processing with some aspects of human Gestalt perception.", "questions": "See above weaknesses. In addition: \n1) clearly based on the results in Figure 6 (appendix) random-dot shape matching is in fact significantly better for the motion-energy model than humans (while FlowNet etc. underperform). This seems to suggest that there is still a misalignment between even the motion-energy model and human perception. Do you have any thoughts on what this missing piece is and how we might be able to arrive at a better model of human perception using the motion-energy model? Perhaps because the motion-energy model doesn't really model appearance at all there is something missing? From the perspective of human vision science, I think it would be helpful to add more text discussing how this work can inform our understanding of human figure-ground segmentation and where the additional gaps may be. \n\n2) Recent work (see below) has shown that there is a perception of opposite-direction motion in random dot kinematograms that can be verified in human vision and also seen in an application of the motion-energy model. This brings up the larger question that perhaps aligning with aspects of human vision may not be totally beneficial to computer vision systems that detect optical flow etc. Could you comment on this?\n\nBae, Gi-Yeul, and Steven J. Luck. \"Perception of opposite-direction motion in random dot kinematograms.\" Visual cognition 30.4 (2022): 289-303.", "limitations": "The authors do address many limitations and in the text. Other than the weakenesses/questions above, I think the authors do a reasonable job of this.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are many strengths of this work and I see it as a careful and nice study of where current optical-flow methods stand relative to much simpler interpretable methods for estimating motion. The additional comparisons with human segmentation performance given random-dot stimuli are compelling and overall I think this paper provides a valuable set of experiments that can guide future work in this interdisciplinary area of human vision science and computer vision. \n\nTo be more specific, I think the work is presented quite clearly and written well. I also think that the problem framing is clear and the authors successfully verify many of their hypotheses. I encourage the authors to release all of their code and data as I think a public pytorch version of the motion energy model and this human data could be valuable to the field, and could encourage the deep learning community to consider testing motion-estimation setups with the motion energy model. \n\nFinally,  the idea of closing the loop between motion estimation and figure-ground segmentation provides a clean test of how well models can perform in comparison to humans on this core aspect of human perception. The experiments seem relatively complete (see below for some concerns), and I find the conclusions to be relatively well supported. Overall I see this work as quite similar to the Dapello et al. 2020 paper on simulating a V1 front-end for spatial visual models, and this work provides a complement to this in the motion processing domain.", "weaknesses": "I have a few concerns regarding the experiments and some comparisons to prior work:\n\n1) It seems that Yang et al. 2023 is a very related work that has not been discussed in the paper. Specifically Yang explore multiple models of optical flow and compare these to human performance in motion flow perception including natural scenes and random-dot kinematograms. While the conclusions and framing are quite different, I think this work deserves much more careful comparison. For example, in this work, the authors find that SoTA methods  (RAFT etc) do in fact correlated better with human perception than local-pooling based methods (spatial and spatio-temporal pooling). I believe the ablation of these pooling methods would be useful in this work as it could provide additional bio-inspired baselines to compare the motion-energy model to. In this vain, I also believe more work can be done to ablate aspects of the motion-energy model (as in Section 4.2) to understand what is the key component to this improved generalization. I think isolating exactly what features of the motion-energy model lead to this generalization could enhance the work greatly. \n\nYang, Yung-Hao, et al. \"Psychophysical measurement of perceived motion flow of naturalistic scenes.\" Iscience 26.12 (2023).\n\n2) While RAFT and the other methods used are strong optical flow estimators, these are still significantly older methods and perhaps comparing to a few more recent methods such as GMFlow, and FlowFormer++ (which outperform RAFT) would increase the impact of the claims \n\nXu, Haofei, et al. \"Gmflow: Learning optical flow via global matching.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\nShi, Xiaoyu, et al. \"Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n3) I think one of the main weaknesses with the work overall is that the main results seem to suggest that the motion-energy model is much better at generalization, but significantly underperforms on the natural videos (albeit this can be improved with finetuning). I think there should be some more extended analysis of study of why this is and some more concrete proposals of how this difference can be fixed. For example, is the issue simply that the SoTA methods are very domain sensitive? For example, could you somehow finetune those methods with random-dot videos and fix the generalization gap? An analysis; however preliminary, about this would greatly increase the impact to the computer vision field in my opinion. \n\n4) While Section 4.2 ablating finetuning aspects of the motion-energy model is quite interesting and potentially useful for the community, I believe the results are actually a bit concerning. They seem to suggest that finetuning the model does in fact overfit to the training data and then causes a large decrease in generalization to the random dot task. The authors say it is surprising that the motion-energy model (without any tuning) performs so well on the random-dot task, but I see this as perhaps a byproduct of the fact that most research on MT had been done using simple stimuli like random-dot motion and so the parameters may have initially been set to fit this data well. If this is the case, this potentially limits the scope of application of this method from the computer vision perspective as it seems like even using the motion-energy model cannot really create a \"generalist model\" that performs very well on multiple domains. Please correct me if I am mis-understanding, but I think the authors should comment more generally on the practical steps going forward on how this can inform creation of more human-aligned, better optical flow methods, rather than simply using this as a study to point out one flaw with current methods. At the end of the day, detecting motion in random-dot kinematograms is a great tool for understanding human and machine vision, but from a practical perspective most applications will be to natural videos or many domains. If this is the case, then there is little reason to use the motion-energy model unless you can show that there is a tangible way to use these results to improve the generality of current methods. \n\n\n\nIf the above concerns are addressed I am definitely willing to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We are happy that you see “many strengths” in our work.\n\n**Comparison to Yang et al. (2023).** Indeed, this work is very related since it also compares various motion estimation models to human perception. Thank you for making us aware of this paper. The focus of Yang et al. is however on naturalistic stimuli and does not consider segmentation, which limits direct comparisons. Interestingly however, Yang et al. find that FlowNet 2.0 agrees better with human perception than the more recent RAFT model which parallels the results in our work. We will extend the related work and discussion sections in our paper to include a comparison to the results of Yang et al.\nWe’re not entirely certain, however, whether we correctly understood the suggested comparison to the pooling methods. The pooling baselines from Yang et al. use ground truth optical flow as input, so in our setting the performance is equal for the original and random dot videos by design—which is much closer to human perception than RAFT. Please correct us if we misunderstood your suggestion, we’re happy to include additional experiments in this direction.\n\n**Additional ablations.** Following your suggestion, we performed additional ablation studies by removing or replacing components of the motion energy. We provide detailed results in our response to reviewer DKrM. The results suggest that among the ablated components, pooling and normalization are particularly important for generalization to random dots—which confirms your suggestion that the pooling is an important component which deserves further focus. Interestingly, removing the pooling slightly improved the results on the original videos, at the expense of much worse generalization. Gaussian pooling is a relatively simple method which does not take into account object boundaries. It has been suggested that the brain uses a more sophisticated approach (e.g., Braddick et al. 1993), so we think that improving the pooling is promising for further improving the model in the future. We will include the additional results and discussion in our paper.\n\n\n**More recent optical flow models.** We included FlowFormer++ and GMFlow in our analysis and provide results in the additional PDF page. Both models support the claim of our paper: FlowFormer++ performs extremely well on clean stimuli but does not generalize to the random dot stimuli at all. GMFlow is among the best optical flow models that we tested, but like FlowNet 2.0 still falls substantially short of the motion energy model on random dot stimuli. Nevertheless, we believe this is a very interesting result since it is an exception to the trend that more recent models generalize worse which was observed so far. We are thankful for the suggestion and will include all results in our paper.\n\n**Finetuning on random dots.** We expect that training optical flow estimators on random dot stimuli would most likely improve their performance. However, as argued in the general response, the inductive biases of human motion perception allows for zero-shot generalization to random dot stimuli which recent optical flow models do not exhibit—and which is not resolved by finetuning.\nAt the same time, we think it is unlikely that the differences between the motion energy model and the optical flow methods can be fully explained by manual overfitting to random dots. In particular, the model by Simoncelli & Heeger has not been optimized on random dot stimuli according to the original paper. As discussed in our response to reviewer DKrM, we think that the result that training the Simoncelli & Heeger model does not lead to a generalizable model stems from missing inductive biases or too narrow tasks. For example, we expect that including an explicit loss term for the motion predicted by the model will improve the results.\n\n**Scaling the motion energy model.** We see practical steps for improvement in combining the strengths of both approaches. Optical flow estimators exploit the rich information of deep feature representations, which are lacking in the motion energy model. At the same time, the spatio-temporal approach of the motion energy model which goes beyond the dominant two-frame paradigm turned out to be highly effective. We believe that combining these approaches is possible and will allow for both high-performing and human-aligned motion perception models. We agree that a discussion of this aspect is helpful, and will update our paper in this regard.\n\n**Differences between human performances and the motion energy model.** As argued in our response to reviewer DKrM, we see our experiment as an ideal observer analysis. Machine algorithms do not suffer from accidental mistakes, lack of concentration or noise which is inherent to biological systems. These effects could be modeled by introducing corresponding noise in the model. Moreover, as suggested in your review, we think that the lack of appearance cues in our model also could also explain some of the differences: While appearance cues are not informative for the random dot stimuli, the brain has to explicitly ignore these cues which might make this task more complex for humans.\n\n**Alignment of computer vision and perceptual models.** We agree that the goals of modeling human perception and building computer vision models of motion estimation are not completely aligned. Motion illusions, like the one you mentioned, are an important feature for models of human perception but might be considered errors in a computer vision setting. However, we think that both goals are sufficiently close to be mutually beneficial. Moreover, motion estimation is usually only a means to an end for downstream tasks. The types of illusions that humans are sensitive to don’t seem to be harmful for overall perceptual performance. So research on motion illusions might reveal “acceptable errors” for real-time and energy efficient systems that behave successfully in the real world.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rs9uDSmxvx", "reviewer": "Reviewer_DKrM", "summary": "The manuscript investigates whether artificial neural networks perceive moving random dots similar to humans. To this end,  they propose a foreground-background segmentation framework to objectively measure several models. Two groups of models are explored: (1) driven by optical flow that finds matching points between two frames, and (2) driven by motion energy inspired by known mechanisms of the biological brain. The conclusion is that the neuroscience-inspired motion energy model matches better human perception.", "questions": "1. Why does neither of the entries in Table 2 match the Motion Energy results in Table 1?\n\n2. Results of Table 2 suggest that the original weights of the proposed motion energy model from Simoncelli & Heeger [35] result in the best prediction of Random Dots, and any deviation to that deteriorates the performance rather significantly. This is rather puzzling to me. It would be interesting to read the thoughts of the authors on what kind of dataset one can obtain similar weights from Simoncelli & Heeger. Essentially, how has the human brain obtained those parameters in the first place?", "limitations": "The limitations of the current manuscript are well explained.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Evaluation of artificial neural networks with random dot stimuli offers a more controlled environment to compare networks to human psychophysics.\n\n2. A simple energy model, inspired by visual area MT in the human cortex, can open a new line of investigation to model human motion perception.", "weaknesses": "1. The comparison of accuracy between networks and humans is ambiguous. The performance difference between the human participants and (a) the best optical flow and (b) the motion energy model, is roughly the same but in different directions. This might suggest that both models that equally far away from human responses (the difference is that motion energy is better than humans while FlowNet2 is worse). Nevertheless, based on these results one cannot conclude that the motion energy model better captures human responses.\n\n2. It is unclear why the online experiment has resulted in a \"suspiciously low\" quality outcome. Does this indicate large individual differences or suggest the experiment is demanding and can result in different interpretations by participants? If such concerns or similar arguments are valid, one must question their validity to test networks as well.\n\n3. I would have liked to see more results from the Motion Energy model perhaps with a different set of conclusions that are illustrated in Figure 2. Perhaps some ablation studies to better investigate the role of area/component (i.e., V1/MT Linear/Blur).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive review. We are happy that you think that our work “can open a new line of investigation to model human motion perception”.\n\n**Human-machine comparison.** We see our study as an ideal observer analysis and argue that humans are not expected to reach the model performance. Several participants of our study reported accidental mistakes and missed stimuli due to slipping attention during the experiment. Both are no concerns for the model. Moreover, the model computation is not affected by noise in the same way as biological systems. Finally, while the motion energy model is motivated from neuroscience the segmentation model is not and might use information more efficiently than humans. Our analysis therefore is an ideal observer analysis (Burge, Annu. Rev. Vis. Sci. 2020). This analysis shows that optical flow methods do not extract sufficient information to explain human performance while the motion energy model does. In our view this is clear evidence in favor of the motion energy approach. We agree that this point deserves more attention and will include it in the discussion of our paper.\n\n**Performance of online study participants.** Several factors probably influenced the low performance of the participants in the online study. First of all, the viewing conditions (contrast, stimulus size, brightness of the surrounding, …) were not controlled so that participants might have taken the study in a difficult setting. Moreover, we did pay subjects independently of their performance. Since the performance depends on concentration and motivation, the participants might not have entirely fulfilled their potential. As promised in the paper, we repeated the study with more people in a controlled lab setting, where participants reached consistent performances (results in the PDF). These results provide evidence against the possibility of large subjective differences.\n\n**Additional ablations.** Following your suggestions, we performed several additional ablations on the Motion Energy model: We removed the entire MT block, the MT linear layer, the pooling layers, and replaced the types of normalization and nonlinearities used in the model. The results below suggest that among the ablated components, pooling and normalization are particularly important for generalization to random dots. As discussed in our response to reviewer 8R4V, we think that the role of the pooling layers is particularly interesting regarding future improvements of the model. We will include the additional ablations in our paper.\n\n|                                   \t| Original  |       \t| Random Dots |       \t|\n|---------------------------------------|-----------|-----------|-------------|-----------|\n| Condition                         \t| IoU ↑ \t| F-Score ↑ | IoU ↑   \t| F-Score ↑ |\n| RectifiedSquare -\\> ReLU (MT)     \t| 0.753 \t| 0.838 \t| **0.609**   | **0.725** |\n| Baseline                          \t| 0.759 \t| 0.845 \t| 0.600   \t| 0.718 \t|\n| Square -\\> ReLU (V1)              \t| 0.770 \t| 0.854 \t| 0.536   \t| 0.663 \t|\n| \\- MT Linear                      \t| 0.768 \t| 0.856 \t| 0.481   \t| 0.609 \t|\n| \\- MT                             \t| 0.770 \t| 0.854 \t| 0.451   \t| 0.583 \t|\n| \\- Blur (V1, MT)                  \t| **0.801** | **0.872** | 0.421   \t| 0.540 \t|\n| ChannelNorm -\\> InstanceNorm (V1, MT) | 0.592 \t| 0.703 \t| 0.230   \t| 0.340 \t|\n| \\- Normalization (V1, MT)         \t| 0.400 \t| 0.516 \t| 0.018   \t| 0.018 \t|\n\n\n\n**Inconsistency between tables 1 and 2.** Thank you for making us aware of this inconsistency. We used a slightly different architecture of the segmentation network for the ablation study in Table 2. We agree that this is suboptimal, and repeated the ablation study using the very same setup as for Table 1 for which we obtain consistent results.\n\n\n**Learning of motion perception in the brain.** We agree that this is a highly interesting question and are happy to extend the discussion of this result in the paper. While we cannot offer definitive truth, we have hypotheses that might inform future work in this direction:\n- We only use motion information for segmentation, while the motion based tasks performed by the brain are much more diverse (e.g., Nishida et al., Annu. Rev. Vis. Sci. 2018). Training on a more exhaustive set of tasks might be important for obtaining the parameters in the first place.\n- There might be additional inductive biases, e.g. due to the biology of the brain. In our ablation study we did not restrict the trained parameters in any way. So there is the possibility to substantially deviate from the mechanisms of the original model, e.g. beyond the spatio-temporal energy approach. Inductive biases are assumed to be central for learning in the human brain (Goyal and Bengio, 2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8x101MhhRw", "reviewer": "Reviewer_cRYq", "summary": "Authors propose a study on the generalization of motion segmentation models to random dot kinematograms, where they study various learning based (optical flow) motion estimation models vs. a classical motion energy model. Their work is the first to explore such a motion energy model for the motion segmentation task, especially with random dot scenes. They also provide human performance upper bound.", "questions": "How is there proposed model different from SOENet cited above in the weaknesses? beyond the fact that SOENet was evaluated for dynamic texture recognition not motion segmentation.", "limitations": "Mainly the used training data might not be the best to conduct such study, what if their generated dataset played a role in how optical flow models generalized to such random dot scenes. A second evaluation training on large-scale VOS benchmark such as YouTube-VOS+DAVIS can help ensure the consistency of the conclusions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good contribution and novelty to study the generalization of motion segmentation methods to random dot kinematograms.\n- Good experimental analysis with supportive psychophysical experiments on humans serving as an upper bound\n- Comprehensive analysis of 28 Optical Flow models", "weaknesses": "- Missing quite relevant related work that actually utilized similar models that are not learnable but based on classical theory on these spatiotemporal oriented energy models and was evaluated on dynamic texture recognition w.r.t SOA deep learning based ones\nHadji, Isma, and Richard P. Wildes. \"A spatiotemporal oriented energy network for dynamic texture recognition.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n- Weak motion segmentation training data. I am wondering how the same study will conclude if the training data used included diversified and large-scale video object segmentation or motion segmentation datasets, e.g.:\n              1. DAVIS: Pont-Tuset, Jordi, et al. \"The 2017 davis challenge on video object segmentation.\" arXiv preprint arXiv:1704.00675 (2017).\n              2. YouTube-VOS: Xu, Ning, et al. \"Youtube-vos: A large-scale video object segmentation benchmark.\" arXiv preprint arXiv:1809.03327 (2018).\n              3. TAO-VOS: Voigtlaender, Paul, et al. \"Reducing the annotation effort for video object segmentation datasets.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n\nWhile they aren't motion segmentation datasets per se, i.e. they focus on the most appearance and motion salient objects (VOS) they still can be used to train such models as random dot kinematograms do exhibit the motion salient objects as well.\n\nNonetheless, still quite interesting research question, contribution, novelty and study so I am still inclined towards an accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We’re happy you find our paper to have “good contribution and novelty” and to provide a “good experimental analysis”. In the following we would like to address your concerns.\n\n**Comparison to SOE-Net.** Thank you for making us aware of the work by Hadji and Wildes. We agree that this work should be discussed, and are happy to include a comparison to SOE-Net in our paper. Regarding the architecture, we see the following similarities and differences compared to the Simoncelli & Heeger model:\n- Both models use a separable Gaussian derivative filter, divisive normalization, a variant of quadratic nonlinearity, and Gaussian spatial pooling. Implementation details for all components differ, nevertheless we expect these components to behave very similarly.\n- Larger differences exist in the overall architecture: The Simoncelli & Heeger model is applied in parallel at different spatial scales, while spatio-temporal processing stages are applied sequentially in SOE-Net. Moreover, SOE-Net does not include a stage corresponding to MT, which is used by the Simoncelli & Heeger model to resolve ambiguities in the local motion signals.\n- The processing stages of the Simoncelli & Heeger model have been experimentally compared to neural activity. This allows us to establish a compelling link between neural mechanisms and high-level behavior. To our knowledge, it has not been established how well SOE-Net matches computations in the brain.\n- The memory and computational footprint is much higher for SOE-Net. We see two main reasons for these differences: (1) Each spatial scale in the Simoncelli & Heeger model reduces the input size by a factor of two, while the SOE-Net with default parameters uses larger feature maps (only a single reduction by a factor of 2 in the 3rd layer). (2) The layers in the Simoncelli & Heeger model use at most 28 feature maps, while SOE-Net uses 20, 400, 800 and 1600 feature maps, respectively. Taken together, these differences make it challenging to use the model for dense prediction tasks from an engineering perspective.\n\nWe reimplemented SOE-Net using PyTorch, but needed to make small adjustments to reduce the memory footprint for our dense prediction setting:\n- We used a stride of 2 in the spatial pooling for each layer. The original model only decreased the resolution after the third application of the spatio-temporal model. This change also means that the model predicts the segmentation mask in only half of the input resolution. We use bilinear interpolation to predict the full resolution segmentation.\n- We replaced the global pooling in the last layer with spatial and cross-channel pooling as used in the previous layers.\n\nWe use the representations of SOE-Net after each layer as multi-scale input to our coarse-to-fine segmentation network. We trained segmentation models using two variants of SOE-Net, using a square nonlinearity and the two-path square nonlinearity, respectively. As the results below show, the model performs similarly as the Simoncelli & Heeger model on the original videos. The performance gap might be explained by the reduced resolution of SOE-Net. However, SOE-Net does not generalize to random dot stimuli. We didn’t have enough time to perform further analyses, so we can only speculate on what causes this difference.\n\n| Model  \t| IoU (original) | IoU (random dots) |\n| ------------- | ------------- |--------|\n| Motion Energy | 0.759 | 0.600 |\n| SOE-Net (Square) | 0.710 | 0.053 |\n| SOE-Net (Two-Path) | 0.723 | 0.052 |\n\n**Standard computer vision benchmarks.** We agree that scaling our approach to standard computer vision datasets would have been interesting and continue to explore this direction. Optical flow estimators have been heavily improved over the recent years while motion energy approaches have received hardly any attention in comparison. While we think that scaling motion energy to standard computer vision benchmarks is possible, we think the engineering effort required for that goal is beyond the scope of this work.\nAs discussed in our general response, we see the main focus of our work in showing that a classical motion energy approach has more similar inductive biases compared to humans than state-of-the-art optical flow estimators. We believe that our dataset is adequate to support this claim.\n\n**Influence of training data.** To rule out a strong influence of the training data we additionally evaluated the OCLR model (Xie et al. 2022) on the random dot stimuli. To our knowledge, this is the state-of-the-art motion appearance free (i.e. purely optical flow based) motion segmentation model. This model uses optical flow predicted by RAFT using several displacements and was trained for strong performance on standard computer vision benchmarks. As the results below show, the model transfers well to our clean stimuli but fails to generalize to the random dot stimuli. This result suggests that the choice of training data is not critical.\n\n| Model  \t| IoU (original) | IoU (random dots) |\n| ------------- | ------------- |--------|\n| OCLR | 0.838 | 0.026 |\n| Motion Energy | 0.759 | 0.600 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful reviews and are happy that the reviewers see “many strengths” (8R4V) and that our work “can open a new line of investigation to model human motion perception” (DKrM). The reviews also contain concerns and valuable suggestions that we are happy to address. In this general reply we will respond to concerns raised by several reviewers, and we respond to more detailed points in the responses to the respective reviews.

The primary goal of our work is finding good models for *human* motion perception. We focus on the fact that humans generalize in a zero-shot manner to random dot stimuli. Since generalization is zero-shot and since motion is not uniquely defined for these sparse stimuli, this setting reveals distinct inductive biases that underlie human motion perception. A good model of human motion perception should exhibit the same inductive biases. In our work, we show that a broad range of high performing optical flow estimation networks are substantially outperformed by a simple, classical motion energy model in this regard. Despite inferior performance on clean data, the motion energy model therefore captures an important aspect of human vision that is missing in state-of-the-art optical flow models. We believe that this is a valuable result towards understanding human motion perception.

Improving computer vision approaches to motion estimation on standard benchmarks is only a secondary goal of this work. Optical flow estimators have been steadily improved during the last decades while the motion energy model was published almost 30 years ago and was designed for explaining responses of individual neurons, not as an image-computable motion estimator. Inferior results on clean data and standard computer vision benchmarks is therefore not surprising in our view. Nevertheless, we think the relatively simple motion energy model with several orders of magnitude fewer parameters (<2.5K) is remarkably effective and provides valuable insights also for improving computer vision. Recent optical flow methods focus on matching deep feature representations between two frames while motion energy models are based on spatiotemporal filtering using a longer temporal context. We hypothesize that combining both approaches enables models that both perform well on standard benchmarks and behave similar to human perception. While we are convinced of the feasibility, we think that scaling classical motion energy models to performance levels that are competitive with state-of-the-art approaches on computer vision benchmarks is beyond the scope of this work.

We’re happy to more clearly describe the motivation and contribution of our work and to extend the discussion to provide an outlook on how to scale motion energy models on practical computer vision tasks, as outlined above. Moreover, we have conducted several additional ablation studies and comparisons as suggested by the reviewers, which further support the central claims of our paper:

- We additionally included the more recent FlowFormer++ and GMFlow models in our work. The results confirm the claims made in our submission.
- We additionally included SOE-Net as a motion representation. This model is based on similar components as the Simoncelli & Heeger model, but applies the spatio-temporal filter repeatedly in order to obtain a feature hierarchy. While reaching comparable performance on the original videos, this model does not generalize to random dots. We hypothesize that the repeated application of the spatio-temporal filter and the cross-channel pooling substantially changes the inductive bias of the model.
- (*) We performed an additional ablation study in which we removed components (MT / MT Linear / Pooling / Normalization) or replaced them with other variants (InstanceNorm, ReLU). The results suggest that the normalization and pooling operations are particularly important for generalization.
- (*) We additionally evaluated the optical flow models using 9 frames as input, matching the input length of the motion energy model. This change did not improve generalization to random dots.
- (*) We evaluated the OCLR model (Xie et al. 2022) on our data. This is a state of the art motion segmentation model based on RAFT and optimized for performance on computer vision benchmarks. Confirming the claims in our paper, this model performs well on the original videos but does not transfer to random dots.
- We repeated the human subject study in a controlled vision lab using more participants (13). We obtained similar results as in our preliminary, less controlled study.

We share more detailed results in the attached PDF, and in the responses to reviewers DKrM and cRYq. We will include all additional results in the paper. To match the space constraints we plan to move results marked with (*) to the supplement. Please let us know if you find a different organization to be more adequate.

Meta-review: Reviews were initially balanced about acceptance. Though the discussion period has resolved several concerns and all reviewers now agree that the paper is a valuable contribution.

The reviewers reported several strengths :
- "A multitude of optical flow estimators are tested"
- " novelty to study the generalization of motion segmentation methods to random dot kinematograms"
- "Good experimental analysis with supportive psychophysical experiments on humans"
- "Evaluation of artificial neural networks with random dot stimuli offers a more controlled environment to compare networks to human psychophysics."
- "the authors successfully verify many of their hypotheses"
- "provides a clean test of how well models can perform in comparison to humans on this core aspect of human perception"

The authors are expected to update their paper for the camera-ready version by including :
- comparison to the results of Yang et al,
- the extra ablation studies,
- the results with FlowFormer++ and GMFlow,
- other reviewers requests and promises that were made during the rebuttal that might have missed...
Decision: Accept (poster)