Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark
Authors: Bing Cao
Quanhao Lu
Jiekang Feng
Qilong Wang
Pengfei Zhu
Qinghua Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Video object counting
masked autoencoder
multimodal self-representation learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Density-Embedded Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark

Abstract: The dynamic imbalance of the fore-background is a major challenge in video object counting, which is usually caused by the sparsity of target objects. This remains understudied in existing works and often leads to severe under-/over-prediction errors. To tackle this issue in video object counting, we propose a density-embedded Efficient Masked Autoencoder Counting (E-MAC) framework in this paper. To empower the model’s representation ability on density regression, we develop a new Density-Embedded Masked mOdeling (DEMO) method, which first takes the density map as an auxiliary modality to perform multimodal self-representation learning for image and density map. Although DEMO contributes to effective cross-modal regression guidance, it also brings in redundant background information, making it difficult to focus on the foreground regions. To handle this dilemma, we propose an efficient spatial adaptive masking derived from density maps to boost efficiency. Meanwhile, we employ an optical flow-based temporal collaborative fusion strategy to effectively capture the dynamic variations across frames, aligning features to derive multi-frame density residuals. The counting
accuracy of the current frame is boosted by harnessing the information from adjacent frames. In addition, considering that most existing datasets are limited to human-centric scenarios, we propose a large video bird counting dataset, DroneBird, in natural scenarios for migratory bird protection. Extensive experiments on three crowd datasets and our DroneBird validate our superiority against
the counterparts. The code and dataset are available.

Published as a conference paper at ICLR 2025
EFFICIENT MASKED AUTOENCODER FOR VIDEO OB- JECT COUNTING AND A LARGE-SCALE BENCHMARK
Bing Cao, Quanhao Lu, Jiekang Feng, Qilong Wang, Qinghua Hu, Pengfei Zhu" Tianjin University
(caobi ng, luquanhao, fengjiekang, qlwang, huqi nghua, zhupengfei) et etjuu.ed wdw.edu.cco cn
ABSTRACT
The dynamic imbalance of the fore- bore-backgroun is major challenge in video object counting, which usually caused by the sparsity of target objects. This remains understudied in existing works and often leads to severe under-/over- prediction errors. To tackle this issue video object counting, we propose a demsity-embedded Efficient Masked Autoencoder Counting (E-MAC) framework in this paper. To empower the model's s representation ability on density regression, we develop first takes the density map as an auxiliary modality to perform multimodal self- representation learning for image and density map. Although DEMO contributes effective cross-modal regression guidance, also brings in redundant background information, making it difficult to focus on the foreground regions. To handle this dilemma, we propose an efficient spatial adaptive masking derived from density maps to boost efficiency. Meanwhile, we employ an optical flow-based temporal collaborative fusion strategy to effectively capture the dynamic variations across frames, aligning features to derive multi-frame density residuals. The counting accuracy of the current frame is boosted by harnessing the information from ad- jacent frames. addition, considering that most existing datasets are limited to human-centric scenarios, first propose large video bird counting dataset, Brrd, in natural scenarios for migratory bird protection. Extensive experi- ments three crowd datasets and DroneBird validate our superiority against the counterparts. The code and dataset are available
Density-Ennn Embedded Masked mOdeling (DEMO) method, which
INTRODUCTION
Video object counting aims to estimate the number of objects in video scenes and has been used in various practical applications, from traffic management to public security. It has the potential to be used in decreasing the workload public management and protecting migratory birds. Due the crucial role of object counting in multiple application scenarios, it has attracted broad attention in recent years with the development of computer vision.
Despite many excellent works that have been proposed over the past decades, most of these methods are based on static single-frame images (Zhang et al. 2016 Li et al. 2018) extracted from video, leading to significant loss of dynamic inter-frame information, especially for the swiftly moving tar- gets. In practice, such as crowd or animal activity analysis, the source data is often captured in video form by surveillance cameras drones. Unlike static single-frame images, video data is signifi- cantly dynamic in the spatial motion variations of foreground objects across adjacent time instances, thereby providing richer contextual information. Therefore, by capturing the inter-frame information between video frames, the model is qualified to better perceive dynamical targets, thereby improv- ing the accuracy and stability of the counting performance. To this concern, some video counting methods appeared recently (Zou et al. [2019 Bai & Chan 2021 Hossain et al. (020). which aim to capture the dynamism between frames by employing techniques such as 3D convolutions incor- porating additional information. However, since the video data inherently suffers from the problem
"Corresponding author https://gthha.commmsstrroo/f

---

Published as a conference paper at ICLR 2025
of redundant background information Zhou et al. (2022). extracting features of dynamic targets in multiple frames may lead to an imbalance between foreground and background information, posing challenges for the model's optimization and inference.
More recently, inspired by the unprecedented strong self-representation ability of pre-trained vision foundational models (He al. 2021; Tong et al. 2022). researchers have injected these founda- tion models into downstream vision tasks to fully exploit their representational potential. Inspired by this, we present an Efficient Masked Autoencoder Counting (E-MAC) framework for video ob- ject counting. Our E-MAC introduced optical flow-based Temporal Collaborative Fusion (TCF) to establish inter-frame relationships, constructing a pre-trained visual foundation model-based video counting framework. The optical flow between frames is used to warp the predicted density map of the adjacent frame to the current frame. Then, we perform cross-attention between the warped density map and the predicted current density map to get the final results.
However, the high dynamic of the video data often leads to imbalanced optimization of the sparse foreground. Different from most existing techniques, we take the density map as additional aux- iliary modality of images and transfer the self-representaation foundation model to object counting for the first time. We constructed a bensity-Embedded Masked mOdeling (DEMO) that takes inputs from both the image and the density map, which performs feature interaction through the encoder and reconstructs the density map from masked image and density map. To this end, the density self-representationn learning drives the regression implicitly by reconstructing the masked density maps. In addition, while the self-representationn learning of density maps facilitates efficient density regression, the dynamic nature of foreground objects in video data still brings significant imbal- anced challenges optimization. Stochastic masked image modeling struggles focus the model on extracting features from dynamic moving targets, leading to redundant background reconstruc- tion that hinders model optimization. To handle this dilemma, we further develop Spatial Adaptive Masking (SAM) to generate dynamic efficient masks. During training, SAM dynamically generates masks based on the correlated density map of each sample, providing valid information while filter- ing out redundant background details. Our framework employs post-fusion strategy and develops a simple cross-attention module to compute the residuals between adjacent predicted density maps, and design skip connect to add the residuals to the predicted density map of the current frame, which ultimately filters the non-dynamic objects in the background.
In this paper, we validate our E-MAC not only in human- scenarios but also in natural scenar- ios. A large-scale video bird counting dataset DroneBird is collected for migratory bird protection. To the best of our knowledge, DroneBird is the first video bird counting dataset that is captured from a drone's viewpoint and provides abundant annotations and rich attributions. Experimental re- sults three human-centric datasets and our DroneBird dataset demonstrate the superiority of our method over the competing methods. Our main contributions are summarized as follows:
We propose a video object counting, which integrates the foundational model and takes the density map as an auxiliary modality to perform self-repressenattion learning, effectively driving density map regression implicitly.
efficient masked autoencoder counting framework for
We propose an efficient spatial adaptive masking method to overcome the dynamic density distribution and make the model focus on the foreground regions. It adaptively generates image masks according to the corresponding density maps, effectively addressing the prob- lem of imbalanced fore-background.
We propose a large-scale bird counting dataset DroneBird for bird activities analysis. To our knowledge, DroneBird is the first video bird counting dataset. Extensive experiments on three human-centric scenarios and our BroneBird dataset validate our superiority com- pared to the competing methods.
2 RELATED WORK
Object Counting. The vast majority of proposed object counting methods were commonly based on single image. Existing counting methods (Li et al.) 2018 Liu et al.] 2019 Liang et al. 2022) were mainly based on density map estimation, which generated the density map from point anno- tations and took it as the ground truth. Most current counting methods tend use density map
2

---

Published as a conference paper at ICLR 2025
Croaded
Sparse
Crowelad
Spans
2004
30.05
Side
Sak Vine
Backlighr Crowdad, For
Top rios
Sung
Top New Hondiag
Chass Chaps
Far ringe
19055
Sunny, Crowded: Top view: For nage: Flying
Figure I: The chord diagram illustrates the associations between various attributes of our proposed dataset. Each attribute showcases portion of the dataset's examples references. We provide two zoomed-in examples for better visualization. The right part represents the experimental result of our proposed method and previous video counting method each attribute of our DroneBird dataset.
regression as the pretext task of object counting since it provides more low-level supervision sig- nals and is easier to optimize. Earlier researchers (Zhang et al.) 2016 Li et al. improving convolutional neural network structures to enhance density regression performance by extracting multi-scale features from images. Recent methods (Ma et al. 2019: Lin et al. 2022) utilized Bayesian loss for density contribution models from point labeling. improving upon density map supervision. Additionally, researchers have integrated CNNs and Transformers to leverage the attention mechanism (Tian et al. 2021] Liang et al., 2022). More recently, some methods intro- duced pre-trained foundational models to build object counting methods (Jiang et al.
2018) explored
2023 Kang
et al. 2024). thereby counting the number of any examples. This inspired us to explore the visual foundation model based video object counting framework.
Video Object Counting, The single-frame image methods focus on spatial information from static images and neglect temporal processing, making difficult address the dynamic nature of video object counting tasks. The target of video object counting is to predict the number of objects in each frame of the video. For the evaluation of counting results, video counting calculates the difference between the predicted results and the ground truth for each frame, and then computes the mean absolute error (MAE) and mean squared error (RMSE) across all frames. Video object counting methods aim to leverage information from neighboring frames to enhance the estimation of the current frame. LSTM and 3D convolutions are commonly used methods for modeling temporal dependencies between frames (Zou et al. 2019 Shi al. 2015). Unlike these implicit methods of establishing frame associations, leveraging object movement direction and optical flow information further enhance counting accuracy (Zhu et al., 2021 Hou al. 2023). However, existing video counting methods (Liu et al. 2020) Hou et al. 2023) mainly address temporal relationships but often neglect intra-frame dynamics of foreground regions. Additionally, the high of dot annotations restricts the availability of large video counting datasets, complicating effective learning of dynamic regions. Our proposed method treats counting as a density reconstruction task, incorporating self- representation learning of density maps with a dynamic spatial adaptive masking module, which significantly enhances the counting performance.
Masked Image Modeling, Masked image modeling refers to the reconstruction of the masked por- tion of masked image by learning representation. With the application of Transformer (Vaswani et al. 2017) in vision and the success of the BERT (Devlin et al., 2019) pre-training paradigm in natural language processing in recent years, masked image modeling has achieved great progress. After some enlightening work (Vincent et al. 2008) Chen et al. 2020 Bao al. 2022), MAE (He) ctt al., 202I) chunks the image, randomly masks out the majority the image patches and then reconstructs them, which has achieved great success on downstream tasks. Inspired by MAE, many works TTong et al.) 2022 Bachmann et al.] 2022) have begun to apply masked image prediction diverse scenarios. Considering the strong representation ability of visual foundation models, we attempt to embed the density map to guide the masked prediction for intra-frame, performing density-driven regression from image to density map and forming an efficient self-representatioon learning framework for video object counting.
3 DRONEBIRD DATASET
Video object counting methods not only hold promising application prospects in human-centric ac- tivity analysis, but also possess invaluable potential in natural scenarios, such migratory bird
3

---

Published as a conference paper at ICLR 2025
Tempural Cillaburativ
Demitt-rmbeooed Masked Modeling
Embeddiin
Patch
Transformer
Transformer
Don
IN
Encoder
Decoder
G
D
D
Maleii Maried
Laaro Toknn
Rarde
Spatial Adaptive Mask
Imngee Token
Optical Flow
Dersiy Token
Mlack Tikke
Peary Warp
Chanvattzation
Adaeti
D
Acdapio
blemene-ntse Add
Figure 2: An overview of our E-MAC. For the temporal collaborative fusion, we use optical flow to fuse multi-frame density maps. For the density-embedee density map are treated as multi-modal data and are fed into the transformer autoencoder for self- representation masked modeling simultancously. The spatial adaptive masking uses the density map to balance the dynamic fore-backgroun. During inference, the density map is fully masked.
masked modeling, the image and
protection. In the scenario of counting volant species like birds, the best of knowledge, the existing open-source data is largely limited to discrete image data (Arteta et al. 2016 Wang et al.) 2023), which makes it challenging to apply these methods dynamie bird activity analysis scenar- 1os. To alleviate the issue of data scarcity as well as to assist in migratory bird activity analysis, we collected new large-scale video bird dataset called DroneBird. DroneBird provides point annota- tions for bird counting, and also provides additional trajectory annotations for further bird tracking. To the best of our knowledge, DroneBird the first bird dataset captured in video from a drone's viewpoint and provides both point annotations and trajectory annotations.
We have collected statistics on various aspects of our DroneBird dataset and compared them with some existing datasets in Table All the videos in DroneBird are recorded at 30 frames per second with resolutions of 2160 4096 or 2160 x 3840. Each frame contains between and 673 annotated objects, averaging 171.5 per frame. The dataset includes 3, 686, 409 bird annotations and 9, 389 bird trajectories, ranging from to 500 frames in length. To further investigate DroneBird, we have analyzed five main attributes of each sample, i.e., Illumination, Density, Perspective, Distance, and Posture. We present the distribution of these attributes and their correlation in Fig. [ Each are represents an attribute, and each chord connects between that possess both attributes represented by the two arcs. For each attribute, we provide two example images in the DroneBird dataset for reference. Detailed descriptions of these attributes and clearer visualization are presented in Appendix A.]
arcs, indicating that there images
METHOD
this paper, we introduce an Efficient Masked Autoencoder Counting (E-MAC) framework based on self-representation foundation model for video object counting. The framework of our E- MAC is depicted in Fig. 2 which consists of temporal collaborative fusion (TCF), demsity-embeddded masked modeling (DEMO), and spatial adaptive masking (SAM). We utilize optical flow to establish connections across multiple frames to capture inter-frame information. A temporal residual map is constructed by leveraging optical flow information between frames, which utilizes historical data to enhance the counting performance of the current frame. For intra-frame information, we employ density-embedded masked modeling (DEMO) and spatial adaptive masking (SAM) based on the self- representation foundation model to effectively balance the learning on foreground and background for more accurate density map estimation.
4.1 TEMPORAL COLLABORATIVE FUSION
The temporal collaborative fusion aims to integrate multiple frames for more accurate estimation. Given the frames time and t 1, each sample consists of frame image and density map. The
4

---

Published as a conference paper at ICLR 2025
samples of two frames can be described as S, (4, Dr) and St-1 masked modeling. Different from most existing methods,
Dt-1), which are then
fed into the DEMO for take the density map auxiliary modality corresponding to the image modality.
Specifically, for a sample S, = {4,, Dr). the patch embedding module patchifies and embeds both the image modality and density map modality into multi-modal tokens. The SAM removes specific patches from these multi-modal tokens before the transformer encoder. After passing through the encoder, the masked positions of the density map are filled with random mask tokens. The decoder then reconstructs the complete original density map based on the incomplete input information. In our framework. two temporally adjacent samples (S,. S are simultaneously fed into the DEMO, where the aforementioned process is used to complete the reconstruction and generation of the predicted density maps (D., D.
The reconstructed density maps {D,-1, D.} obtained by the output of DEMO. To align their spatial distributions, a pre-trained optical flow network (Sun et al. 2018) estimates the motion dis- placement, followed by a warp operation on Di-1, resulting in D app The cross-attention between D and D) then produces Dis representing the temporal density residuals of adjacent frames. D and D, are combined via element wise addition to output the final fused prediction Disee The TCF can be formally described by Equation [
The fusion effect is improved by utilizing opti- cal flow to align information hetween adjacent frames. We present the whole training process in Appendix A.2 to make easy to understand.
Druse (Ccal @Wapp (OppiccllFoo (I, D())) DD.
Temporal residual density of adjacent frames
4.2 DENSITY-EMBEDDED MASKED MODELING
As depicted in Fig. 2 the density-embedded masked modeling (DEMO) is a Transformer-based au- toencoder. The input sample S, is first divided into patches ppnchh D patch). which are then converted into a token sequence and C' denotes the feature channels. The image 4, and density map D) are tokenized simultancously, then concatenated along the L. dimension, where I, = .N + N.D. .N, and ND represent the number of tokens from the image and density map modalities. SAM is density-guided masking strategy that uses human annotations as priors. Further details are provided in S.c retains Wfet foreground tokens from image Ie and randomly keeps Afpet tokens from D,, generating new token sequence T I (NT (NN7))N
IR BxCC where B represents the batch size, L is the number of tokens,
The retained tokens are sent to the transformer encoder, while the remaining tokens are discarded and passed into the Transformer. The output token dimension of the encoder B x (N7* +N')') )> In the decoder, the retained density map tokens Te are separated from the retained token sequence T where Tret B +NB)xC and T E R Bx.N xc The learnalle random mask tokens are filled at the masked positions in the retained density map tokens T as placeholders, and we use To to represent the filled density map tokens. Cross-attention is then applied, with Tn as the query and T as the key and value. Then, the reconstructed density map D) is generated by the two-layer transformer, as the end of the self-representation masked modeling.
4.3 SPATIAL ADAPTIVE MASKING
The masked modeling approach discards a subset of tokens prior to the transformer encoder, utiliz- ing the decoder to reconstruct the missing information. This process allows the model to capture the relationships between tokens. In the context of multi-modal masked modeling, further enables the model to learn associations and interaction mechanisms across different modalities. A substan- tial body of research indicates that random masking strategies may introduce excessive redundant information due to the imbalanced fore-background, which is detrimental to the model's learning process. To this concern, we developed spatial adaptive masking (SAM) for efficient learning of the dynamic changing targets in videos. This strategy reduces redundant background optimization and focuses the model's 's attention on the image foreground, thereby improving the efficiency of seff-representation learning.
5

---

Published as a conference paper at ICLR 2025
For a video frame I, its density distribution D serves the standard for delimiting the fore- ground and the background. The lower-left part Fig. 2 provides a detailed illustration of the SAM. The symmetric Dirichlet distribution (Bachmann et al. 2022) is used to determine the number of retained tokens for the image modality and density map modality when generating multi-modal masks, denoted No and NB respectively. We calculate the number of targets V in the i- th density map patch D patch corresponding to each token, and Vn (V): VD. V y where v Osum D patch). Osum represents the pixel-wise sum operation in each density patch (Zhang jet al. 2016; Li et al. [018)) and the results represent the number targets in the corresponding sort the image tokens according the number of targets V in the corresponding density map modality. While the foreground provides more valid informa- tion, the background should not be completely ignored. Therefore, we set background retention probability (BRP) P to introduce the background information, where BRP determines the sorting manner, in ascending order with a probability of P (focus on the background) or in descending or- der with probability of P (focus on the foreground). Detailed experiments of P are presented in the Sec. 5.4 The first No tokens are retained to guide the masking of image I, preserving the foreground while discarding the background. Here, we denote IC as the set of positions that should be kept, and N a random variable that follows a Uniform distribution between and 1, which is produced by a random number generator.
patch. To focus the foreground,
argsort des Sam patch M1 N77), if NSI-P, K
(2)
argsort ase (msmm Dpateh) {1 ' otherwise.
Based on this, we can obtain the spatial adaptive mask Maduptive = {M adupptive | S N1} for image For each token in position and its corresponding mask Maluptive we have
{
0, if i
M M adaptive
otherwise,
where O represent keeping and represent masking. We denote the retained tokens from the image I as T E Bx where TTt = Tr & (1- Madaptive) and Tr denotes all the tokens of image For the density token TD corresponding density map patch D Dpaich, we generate a random mask Mrandorn to retain ND tokens as T E BB xC These retained tokens are then concatenated to and fed into the decoder for prediction.
T E Bx
During inference, the density maps tokens are fully masked and removed. Only the image tokens are fed into the trained network, which is then required to fully reconstruct the density maps. In other words, we set A = and Appl NN.
4.4 LOSS FUNCTION
In this work, we minimize the Mean Square Error (MSE) to ensure that both multi-frame fused density map Djus and the single-frame predicted result D, approach the ground-truth density map Dr. To simplify the optimization of the optical flow network, we apply an MSE loss between the warped prap and the original image T;:
h
w [Ba,
Gt.))²
LMSE
(4)
2hw
j=l
where É and G represent the estimated vector and its ground truth. Specifically, É in Liuses Lears Lopt represents D D, and pwarp respectively, and the corresponding G represents D,, Dr and Ie. A Total Variations (TV) loss (Rudin et al.) 1992) is introduced as a regular term to encourage spatial smoothness in Drss. TV loss can be expressed as:
(Did fuse
(Diu fusc Di 27
)
Lrv hw i=1
(5)
j=1
The objective loss function can be expressed as follows, where A da are hyperparameters. Xt + >p + Aa + A1LTV.
(6)

---

Published as a conference paper at ICLR 2025
Table 1:
Quantitative comparison between our proposed method and existing methods with metrics
MAE and RMSE, lower metrics better. Further comparative results can be found in Sec. A.3
FDST
VSCrowd
DroneBird
Method
Type
MARJ
RMSE
MARJ
RMSE
MAEJ
RMSEJ
MAEJ
RMSEJ
MCNN Zhang et al. 2016
Image
3.77
4.88
27.1
46.9
122.35
149.07
CSRNet (Lt et a. 2018
Image
2.46
4.70
2.56
3.12
13.8
21.1
66.11
79.33
CAN
al 2019
Image
70.21
92.15
MAN
2022
Image
2.79
4.21
8.3
10.4
39.11
50.08
HMoDE: Du m 2023
Image
2.82
33.41
2.49
3.51
19.8
39.5
67.47
81.40
PET (Liu al. 2023
Image
1.89
2.46
1.73
2.27
6.6
11.0
45.10
52.35
Grumforntrr Lam
2024
Image
1.69
2.14
5.15
6.32
8.09
15.65
49.11
65.50
EPF cL 2020
Video
2.17
2.62
10.4
14.6
97.22
133.01
PFTF Avyenul et
2022
Video
2.99
33.72
2.07
2.69
89.76
101.02
GLANeT 2022
Video
2.10
2.90
8.2
10.2
FRVOC (Hou d.
Video
1.79
1.88
STGN Wu 2025)
Video
1.53
1.97
1.38
1.82
9.6
12.5
92.38
124.67
Ours
Video
1.35
1.76
1.29
1.69
6.0
10.3
38.72
42.92
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets and Metrics. We conduct experiments on our DroneBird dataset and three video ob- ject counting datasets: Fudan- Sotta--SannghiTTech (FDST) (Fang et al. 2019), Mall (Loy et al. 2013) and VSCrowd (Li et al., (022)) datasets. We use a fixed Gaussian kernel ground-truth density map on these datasets. Following previous methods, we evaluate the counting performance by using Mean Absolute Error (MAF) and Root Mean Square Error (RMSE) for each frame in datasets. MAE measures accuracy as MAE = Σ²_ D; D.(. and RMSE measures ro-
to generate the
bustness RMSE = / + Σ: (p: 6.)²² where n is the number of samples in datasets, D; and D) represent the ground truth and predicted density maps of the i-th sample, respectively.
Implementation Details. For the backbone network, the optical flow network leverages the pre- trained PWCNet (Sun et al., 2018), while the pre-trained ViT-B from MultiMAE (Bachmann al,, 2022) is used as the encoder in E-MAC. During inference, the density maps {Dz-1, Dr} are fully masked, leaving the video frames (4-1, 4+} intact, enabling the model reconstruct the complete Huawei Atlas 800 density map Das from the input video alone. Our experiments are conducted Training Server with CANN and NVIDIA RTX 3090 GPU. We adopt random horizontal flipping to perform data augmentation. Density maps are standardized by mean and standard deviation for better optimization. For hyperparameter settings, the model employs linear learning rate warm-up for the first 15 epochs, followed by cosine decay learning rate. The weight decay of AdamW is set to 0.05, and layer decay is set to 0.75 for the encoder. The mask ratio is 0.72. Empirically, to maintain a balance between foreground and background tokens, setting a small probability of retaining only the background improves the model's performance for SAM. The probability P for spatial adaptive masking set to 0.2. The trade-off parameters Å1, A2, X3, da are set to 10, 10, 1, and 20, respectively.
5.2 COMPARISONS
We compare our method with several state-of-theear methods on our DroneBird dataset, the FDST dataset, the Mall dataset, and the VSCrowd dataset.
Mall. Mall provides video data from fixed viewpoint in shopping mall, where factors such as lighting are relatively controllable. On the Mall dataset, we follow the previous works (Bai & Chan 2021; Hossain et al. (2020) for fair comparison. The model is trained with the first 800 frames of the Mall dataset, and the rest 200 frames are used as the test set. The input images are set to the size of 448 x 640 and the batch size is set to 3. The quantitative comparisons are reported in T Our method achieved significant advantages on MAE and RMSE metrics, which improves 4% of MAE and 2% of RMSE compared to the runner-up method FRVCC (Hou et al.) 2023) based on CSRNet (Li et al. 2018). Compared to the PFTF, our method achieves significant reductions in MAE and RMSH of 55% and 53%. We compared method to video counting

---

Published as a conference paper at ICLR 2025
Mall
FOST
Image
Ground Truth
34.00
29.00
25.00
:35.00
43,00
39.00
Ours
34.98
30,85
29.29
33.42
45,74
38.98
PFTF
44.57
35,54
42.61
50.57
Figure 3: Visualized comparisons on the FDST dataset and the Mall dataset.
VSerowd
DroneBird
Image
Ground
Trutte
$1.00
91.00
40.00
314,00
220.00
313.00
Ours
75.78
83.41
43.46
275.42
179.61
240.14
MAN
22.30
5.47
25.24
129.35
139.93
88.06
Figure 4: Visualized comparisons on the VSSrowd dataset and our DroneBird dataset.
method PFTF (Avvenuti al. 2022) and visualized the results on the Mall dataset in Fig. 3 Our method produces more clear and accurate density distributions of distant low-pixel targets, resulting in superior visualization performance. The quantitative and qualitative experimental results proved the superiority of our framework in the indoor scenarios.
FDST. The FDST dataset provides a wider range of scenarios, including various outdoor scenes, with more diverse variables compared to the Mall dataset, thus posing greater challenges. For quan- titative comparison, we reported the MAE and RMSE metrics of our model and competing methods on the FDST dataset in Table[]] The result shows that our method achieves the best MAE and RMSE, decreasing the two metrics 7% compared to the runner-up method STGN (Wu et al. 2023), and 31% compared to FRVCC (Hou al., 2023). respectively. For qualitative comparison, the predicted results of our method and the competing video counting method PFTF (Avvenuti et al.) 2022) on several scenarios in Fig. Our method offers better visualization effects and delivers more accurate quantitative predictions. These experimental results validate our method maintains superior performance in more complex scenarios, such as outdoor environments.
visualize
VSCrowd collected more videos by using surveillance cameras or the Internet. Com- pared FDST, the VSCrowd dataset provides more diverse and complex set of outdoor scenes and presents greater challenges for video crowd counting. The evaluation results of our method and the competing method on the VSCrowd dataset are presented Table Compared to existing methods, our approach achieves superior performance on MAE and runner-up performance on RMSE met- rics. Compared to the recent video counting method STGN (Wu et al. (023), our method improves the performance by 38% in MAE and 18% RMSE, respectively. Compared to the runner-up method GNANet (Li et al. 2022), our method beat GANNet on the MAE metric and achieved competitive performance on the RMSE metric. We present detailed visualizations
VSCrowd.
the VSCrowd datasets in Fig. Our method achieves accurate counting results under low-light or long-distance dense con- ditions compared to the previous method (Lin et al. 2022), showing the priority of our framework. These quantitative and qualitative comparison results demonstrate that our framework still possesses competitive performance in more diverse and complex outdoor scenarios.
8

---

Published as a conference paper at ICLR 2025
Table
Ablation studies on three components.
Table Effect of each loss function.
Exp.
DEMO
SAM
TCF
MAE,
RMSE
Exp.
Lise
Leir
Lupt
LTv
MAE 1
RMSE 1
2.45
3.22
VI
1.87
2.50
II III IV
2.32
2.69
VII
v
1.80
2.37
1.57
1.99
VIII
v
1.60
2.03
2.20
IX
V
1.39
1.77
V
1.29
1.69
X
1.29
1.69
DroneBird. Different from the previous three datasets, our DroneBird provides bird flock data from drone's perspective, with scenes mostly consisting of open outdoor areas and exhibiting higher dynamics which pose significant challenges for video object counting. We assessed several existing methods on our dataset, detailed in Table I Our method outperforms both recent video and im- age counting techniques, achieving a 58% improvement in MAE and a 66% improvement in RMSE compared to the STGN (Wuet al. 2023) method. Additionally, our approach shows enhancements in MAE and RMSE over the previous optimal method, MAN (Lin et al. (022). For qualitative compar- ison, we compared the visualization results in multiple challenging scenarios. Our method achieves more accurate counting results of birds, even in complex areas like water reflections. The quantita- tive experimental results across different attributes and the visualization effects demonstrate that our framework still exhibits superior counting performance in even more complex and variable outdoor scenes from a drone's 's perspective, thereby highlighting the superiority of our framework. Further- more, we conduct attribute comparisons with three competing methods (Avvenuti al. 2022 Liu eet al.] (2000; Wu et al., (023) on various attributes of the DroneBird dataset, as illustrated in Fig. These experiments fully demonstrate our superiority in various complex scenarios.
5.3 ABLATION STUDY
We perform the ablation study the FDST dataset to investigate the effectiveness of Density- embedded masked modeling (DEMO), spatial adaptive masking (SAM), and temporal collaborative fusion (TCF). We construct the same architecture as that in comparison experiments and trained for 200 epochs. The hyperparameters are set to the same as the previous experiments on the FDST dataset unless otherwise noted.
We test five variants of our method to assess the impact of DEMO, SAM, and TCF experiments I-V, with results in Table Exp.I is the E-MAC baseline, using pure transformer for density map regression. Exp.II adds optimal flow and fusion module for inter-frame relationships Exp.III incorporates SAM into Exp.II for adaptive masking, Exp.IV introduces DEMO Exp.II, masking both images and density maps, unlike Exp.III which only masks images with SAM. Exp. V combines DEMO and SAM to evaluate overall performance.
Effeet of TCF. We incorporated the optical flow module and fusion module into Exp.II and compared its performance with Exp.I. The results indicate that the construction of inter-frame relationships brought performance improvement of 5% to 16% in terms of MAE and RMSE. By employing optical flow mapping, we were able to effectively leverage the inherent temporal information present in video data, enhancing the information of the current frame and improving the overall performance. Further study and visualization on TCF are presented in Appendix A.5
Effect of DEMO. As shown in Exp.Il and Exp. IV in Table 2 DEMO brings in 27% and 18% improve- ment on MAE and RMSE metrics. In Exp. V, the introduction the self-representattoon learning of density maps resulted in 17% and 15% improvement in MAE and RMSE compared Exp.III The self-representation learning of density maps implicitly drives the regression of density maps and effectively boosts the counting performance.
Effect of SAM. In Exp.III, foreground tokens from images are selected while all image tokens are selected in Exp.II. As shown in Exp.II II and Exp.III, our proposed spatial adaptive masking brings an improvement of 32% in MAE and 26% in RMSE. Exp.Il and Exp.III show the effect of our proposed SAM. Additionally. SAM brought 23% performance improvement to DEMO in Exp. V. Hyperparameter P is set to 0.2 in IxI.III and EXp.IV.
5.4 DISCUSSION
Loss Analysis. We evaluate different loss functions in our framework (Table 3 training for 200 epochs. Starting with Exp.VI VI (baseline using Liuse), incrementally add loss terms. Adding Lrv

---

Published as a conference paper at ICLR 2025
3.2
22
kg
22
A
35
2.7
13
1.6
14
15
1.4
14
14
12
1.2
12
122
022
04
0.22 0.87
40
5
(1)
(b)
(c)
(d)
(c)
Figure 5: Hyperparameter analysis of background retention probability, mask ratio, and loss weights.
(EIp.VII) improves MAE and RMSE by 4% and 5%. Introducing Lept (Exp. VIII) brings further gains of 11% and 14% over Exp.VI VII. Exp.IX adds Lour. achieving 23% and 25% improvement in MAE and RMSE compared to VXI. VII. Exp.X combines fop and Leurs yielding 19% and 17% improvements over Exp. VII. The stronger impact of Ceur stems from its direct supervision for density map generation, outperforming other loss terms.
Impact of Background Retention Probability. We have conducted an in-depth analysis for our SAM. Considering that discarding background redundant information altogether leads imbal- anced learning towards foreground, which in turn exhibits a decrease in performance. On the other hand, omissions during manual annotation could result in some counting information being present in the background. Therefore, we retain only the background of I, during SAM with a certain prob- ability P. Based on Exp.V. V. we conduct further experiments on the choices of P. We choose five sampling points: 0, 0.1, 0.2, 0.4, and 1, in our experiments and compared them with the Exp. V in the ablation study. The subfigure (a) of Fig. 3
provides a more intuitive view of the final performance with respect to the probability P. The horizontal axis indicates the probability of sorting the tokens in ascending order. We notice that the curve shows a clear downward rebound trend, and the quanti- tative metrics show a decline of different degrees in both four experiments compared to Exp. V. We finally choose 0.2 as the default probability in our experiments.
Impact of Mask Ratio. We conducted experiments to evaluate the impact of the mask ratio on DEMO by varying it from 0.67 to 0.83. To more clearly evaluate the impact of the mask ratio, these experiments were specifically performed on our E-MAC without considering temporal information. Fig. shows the results. As the mask ratio decreases, more tokens are input, and MAE decreases. However, at a mask ratio of 0.67, performance drops by 7% compared to 0.72. We conclude that a lower mask ratio provides sufficient information for reconstruction, but too low a ratio introduces redundant information, harming performance.
Hyperparameter Analysis. We conducted experiments on the setting of the hyperparameters (X1, A2 A3, }1} of each loss function, shown in subfigures (c-f) of 5 We vary the weights of the remaining loss terms while fixing the weights of the other three loss terms, and the experimental results correspond to the four subfigures of Fig We finally fixed the weights A2, λa b of each loss term to 10, 10, 1, 20 in our experiments.
6 CONCLUSION
This paper aims to address the dynamic imbalance of the fore-background in video object counting. Considering the dynamic sparsity of foreground objects, we proposed a density-embeddded efficient masked autoencoder counting framework. We introduced the self-representation foundation model to video object counting, which first takes the density map as an auxiliary modality and devel- ops density-embedder masked modeling (DEMO) to drive the regression of density map estimation. To handle the farfa- dynamic density distribution and make the model focus more on the fore- ground region in the self-reprrsentation learning, we proposed a simple but efficient Spatial Adaptive Masking (SAM), which dynamically generates masks depending on density maps to eliminate the ef- feet redundant background information and boost the performance. Furthermore, accounting for the inter-frame dynamism and utilizing the inherent temporal information in the video, we introduce the optical flow and propose a temporal collaborative fusion that learns to hamness the inter- inter-frame differences. Besides, we first proposed named DroneBird. Our DrodeBird provides point and trajectory annotations in different scenes for counting and further localization and tracking tasks. Experimental results verify our superiority.
large-scale video bird dataset in the drone perspective,
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was sponsored by the National Science and Technology Major Project (No. 2022ZD01 16500), the National Natural Science Foundation of China (No.s 62476198, 62436002, U23B2049, 62222608, 62276186, 61925602), the Tiajiin Natural Science Funds for Distinguished Young Scholar (No. 23JCIQIC00270,, the Zhejiang Provincial Natural Science Foundation of China (No. LD24F020004). This work was also sponsored by CAAI-CANN OpenFund, developed on Openl Community. The authors appreciate the suggestions from ICLR anonymous peer reviewers.
REFERENCES
C. Arteta, V. Lempitsky, and A. Zisserman. Counting in the wild. In European Conference Computer Vision, 2016.
Marco Avvenuti, Marco Bongiovanni, Luca Ciampi, Fabrizio Falchi, Claudio Gennaro, and Nicola Messina. spatio-temporal attentive network for rideo-based crowd counting. In 2022 IEEE Symposium on Computers and Communications (ISCC), pp. 1-6, 2022. doi: 10. 1091//S/CC555.8 2022.9913019.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi multi- task masked autoencoders. European Conference Computer Vision, 2022.
Haoyue Bai and S. H. Gary Chan. Motion- non-local spatial-temporal network for video crowd counting, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans-
formers. In International Conference on Learning Represematiins, 2022. URL https:
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Reseurch, pp. 1691-1703. PMLR, 13-18 Jul 2020. URL https: / http://pooeed//eeeeee/eo ngs.
m1 r. wwwsss/vl9//ct/ttee.
html
Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-41866 Minncapolis. Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 11.1885//1/N11-1446 1423. URL https::/aclantthlleeee
N19-142.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19999, 2020.
Zhipeng Du Miaojing Shi. Jiankang Deng, and Stefanos Zafeiriou. Redesigning multi-scale neural network for crowd counting. IEEE Transactions on Image Processing, 2023.
Yanyan Fang, Biyun Zhan, Wandi Cai, Shenghua Gao, and Bo Hu. Locality constrained spatial transformer network for video crowd counting. arXiv preprint 1xXiv:1907.079111
Yanyan Fang, Shenghua Gao, Jing Li, Weixin Luo, Linfang He, and Bo Hu. Multi-level fea- ture fusion based locality-constrained spatial transformer network for video crowd counting. Neurocomputing, 392:98-107. 2020. ISSN 0925-2312. doi: http:://bb.oog///..0006..eeem
2020.01.087. URL. https: wwww///ww/////////////////ww sci encedi rect. wtt. com./sclence/article//// 50925231 220301154
www.sciincediiiii
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv:2111.06377 2021.
11

---

Published as a conference paper at ICLR 2025
Mohammad Asiful Hossain, Kevin Cannons, Daesik Jang, Fabio Cuzzolin, and Zhan Xu. Video- based crowd counting using a multi-scale optical flow pyramid network. In Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020.
Yi Hou, Shanghang Zhang, Rui Ma, Huizhu Jia, and Xiaodong Xie. Prame-reeurrent video crowd counting. IEEE Transactions on Circuits and Systems for Video Technology, 53(9):518--5199 2023. doi: 10. 1..11..00000 1..1109TTSVVT.22232330000
Ruixiang Jiang, Lingbo Liu, and Changwen Chen. Clip-count: Towards text-guided zero-shot ob- ject counting. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, pp. 4535-4545, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9798400701085. doi: 141145/3581783.333 1789. 3581783. 3611789
URL nttps://dci.oow 00..:/..rrg/001.11/......
Seunggu Kang, WonJun Moon, Euiyeon Kim, and Jae-Pil Heo. Vlcounter: Text-aware visual rep- resentation for zero-shot object counting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.
2024.
Haopeng Li, Lingho Liu, Kunlin Yang. Shinan Liu, Junyu Gao, Bin Zhao, Rui Zhang, and Jun Hou. Video crowd localization with multifocus gaussian neighborhood attention and a large-scale benchmark. IEEE Transactions on Image Processing. 31:6032-6044 2022. 2022.3205210.
doi: 10.1109/TIP.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1091-1100, 2018. doi: 10. 10.1109/CVPR.2018.001111
Dingkang Liang, Xiwu Chen, Wei Xu, Yu Zhou, and Xiang Bai. Transcrowd: weakly-supervised crowd counting with transformers. Science China Information Sciences, 65(6):1-14, 2022.
Hui Lin, Zhiheng Ma, Rongrong Ji, Yaowei Wang, and Xiaopeng Hong. Boosting crowd counting via multifaceted attention. In CVPR, 2022.
Hui Lin, Zhiheng Ma, Xiaopeng Hong, Qinnan Shangguan, and Deyu Meng. Gramformer: Learning crowd counting via graph-modulated transformer, 2024. URL https::/aczxv.orgg/
2401. 03870
Chengxin Liu, Hao Lu, Zhiguo Cao, and Tongliang Liu. Point-query quadtree for crowd counting, localization, and more. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Weizhe Liu, Mathieu Salzmann, and Pascal Fua. Context-aware crowd counting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Weizhe Liu, Mathieu Salzmann, and Pascal Fua. Estimating people flows to better count them in crowded scenes. In The European Conference on Computer Vision (ECCV), August 2020.
Chen Change Loy, Shaogang Gong, and Tao Xiang. From semi-supervised to transfer counting of crowds. In 2013 IEEE International Conference on Computer Vision, pp. 2256-2263, 2013. doi: 10. 10.11009/C
Zhiheng Ma, Xing Wei, Xiaopeng Hong, and Yihong Gong. Bayesian loss for crowd count estima- tion with point supervision. In Proceedings of the IEEE International Conference on Computer Vision, pp.
Shiqiao Meng, Jiajie Li, Weiwei Guo, Lai Ye, and Jinfeng Jiang. Phnet: Parasite-host network for video crowd counting. In 2020 25th International Conference on Pattern Recognition (ICPR), pp- 19941992.
1956-1963, 2021. doi: 10. Ww..1199//CPR880000
Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal al- gorithms. Physica D: Nonlinear Phenomena, 60(1):259-266) 1992. ISSN 0167-2789. doi: https: //bb:.or//1..1011//6667777711109 (Tt.0000166777229999999229220200
URL https: www. wciencedirect. com/
sci ooiiennce/artilllee////111121119
12

---

Published as a conference paper at ICLR 2025
Xingjian Shi, Zhourong Chen, Hao Wang. Dit-Yan Yeung, Wai-kin Wong, and Wang-chum Woo. Convolutional Istm network: a machine learning approach for precipitation nowcasting. In Pro- ceedings of the 28th International Conference on Neural Information Processing Systems Vol- ume 1, NIPSS'S 15, pp. 802-810, Cambridge, MA, USA, 2015. MIT Press.
Weibo Shu, Jia Wan, Kay Chen Tan, Sam Kwong, and Antoni B Chan. Crowd counting in the frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19618-19627, 2022.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Ye Tian, Xiangxiang Chu, and Hongpeng Wang. Cctrans: Simplifying and improving crowd count- ing with transformer. arXiv preprint urXiv:2109.14483 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data- efficient learners for self-super video pre-training. In Advances in Neural Information Pro- cessing Systems, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning. ICML '08. pp. 1096-1103, New York, NY, USA, 2008. Asso- doi: 10.1 1411119919116666 1390294. URL 3901 56. 1390294]
ciation for Computing Machinery. ISBN 978 605582054.
https: //doi www./orgg...oo 0. . 1145/139015551
Hongchang Wang. Huaxiang Lu, Huimin Guo, Haifang Jian, Chuang Gan, and Wu Liu. Bird-count: mulli--mdddlle benchmark and system for bird population counting in the wild. Multimedia Tools and Applications, Apr 2023. ISSN 1573-7721. doi: 10. 1..11000111444 1042-0223222-200 14833-z. URL
https: //o.//do//.. ooo.orgglo
/10. /00 1007 / 1042 02.3 4833-z
Xingjiao Wu, Baohan Xu, Yingbin Zheng, Hao Ye, Jing Yang, and Liang He. Fast video crowd counting with a temporal aware network. Neurocomputing, 403:13-20, 2020.
The Wu, Xinfeng Zhang, Geng Tian, Yaowei Wang, and Qingming Huang. Spatial-temporal graph network for video crowd counting. IEEE Transactions on Circuits and Systems for Video Tech- nology, 33(1)2228-241, 2023. doi: 10.11099CSSV.22222..1188 1944
Feng Xiong, Xingjian Shi, and Dit- Yan Yeung. Spatiotemporal modeling for crowd counting in videos. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 5161-5169, 2017. doi: 10.1000000222
Yifan Yang, Guorong Li, Zhe Wu, Li Su, Qingming Huang, and Nicu Sebe. Reverse perspective network for perspective-awer aware object counting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1374-4383, 2020.
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd count- ing via multi-column convolutional neural network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Joey Tianyi Zhou, Le Zhang, Jiawei Du, Xi Peng. Zhiwen Fang, Zhe Xiao, and Hongyuan Zhu. Locality crowd counting. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 44(7):360236133 2022. doi: 00.1009/PAMI..220133556661
Pengfei Zhu, Tao Peng. Dawei Du, Hongtao Yu, Libo Zhang, and Qinghua Hu. Graph regularized flow attention network for video animal counting from drones. IEEE Transactions Image Processing, 30:5339-5351, 2021. doi: 10. 2TT.TPP..02..30822999
Zhikang Zou, Huiliang Shao, Xiaoye Qu, Wei Wei, and Pan Zhou. Enhanced 3d convolutional networks for crowd counting, arXiv preprint arxiv: 000800412 2019.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DRONEBIRD DATASET
Sunny; Crowded; Side view; Far range; Standing
Backlight; Crowded; Side view; Far range; Flying
Sunny; Crowded; Top view; Fur range; Flying
Sunny: Sparse; Side view; Close range; Standing
Figure 6: Visualization of partial examples of DroneBird.
Our DroneBird dataset is captured by cameras mounted on drones using consumer drones such as DJI Mavic Pro, Phantom Pro, etc. DroneBird captures wide range scenarios, including rivers, wetlands, lakes, ice, and other common bird habitats. The data captured in DroneBird is primarily obtained by the drone from 30 meters or 60 meters in the air, with a small portion of the data captured close to the ground. DroneBird's ' collection time is during the daytime in the early evening when the weather conditions are favorable, and the view relatively clear.
Table 4: Existing bird datasets and our proposed DroneBird dataset.
Dataset
Type
Trajectory
Highest Resolution
Frames
Ave count
Total count
Penguin Arteta et al. 2016 Bird-Count (Wang al. 2023
Image Image
1536 x 2048
33, 405
178.4
5,970, 899
768 1024
1,372
131.1
173, 458
DroneBird
Video
V
2160 4096
21,500
171.5
6866 409
20 pixels 160 pixels
Figure 7: Visualization of pixel occupied by the target in DroneBird and existing crowd dataset.
14

---

Published as a conference paper at ICLR 2025
DroneBird captured 50 videos of mi- gratory birds and segmented them. Specifically, we used 30 of the vidco data as a train set, 10 of the remain- ing 20 videos a test set, and a0 validate set. We cut the 40 videos in the train and test sets to 500 frames
Table 5: Transfer performance on bird data.
Exp.
Original dataset
Target dataset
MAE
RMSE
1
VSCrowd
DroneBird
183.31
217.42
2
DroneBird
DroneBird
38.72
42.92
per video (around 17s), and cut the 10 videos in the validate set to 150 frames per video (around 5s) accomplish a reasonable data division. The train set, test set and validate set after the divi- sion is completed contain 15, 000 frames, 5, 000 frames and 1, 500 frames, respectively. It is worth noting that the data scenarios for each of the two divisions the Irain, test. and validate sets are different, as way to ensure that the data will not be leaked during the training process.
We have compiled the dataset's statistics and compared them with existing bird datasets, with the results shown in Table 4 A few examples of DroneBird are demonstrated in Fig. 6 Most of the targets in DroneBird are small in size. We compared the pixel height occupied by individual bird targets in DroneBird with that occupied by individuals in existing crowd data, and visualized this The pixel height occupied by individual bird targets is significantly smaller
comparison in Fig. 7 than that of crowd individuals, posing significant challenge for the target counting task.
We summarize the contributions and new challenges brought by DroneBird into the following four points:
Small target: We compared the number of foreground pixels occupied by independent targets in the DroneBird data and the existing crowd data at the same resolution, and the independent targets in DroneBird occupy fewer pixels (less than 20 pixels height) than the independent targets in the existing dataset (more than 160 pixels height). Note that although the counting target in the crowd counting task is the human head, the human body still provides important information, and the human body occupies more pixels compared to birds, our dataset provides data on small targets for counting. On the other hand, to avoid drone interference with bird activity, DroneBird's 's drone was photographed farther away from the flock, resulting in a further reduction of pixels occupied by the photographed individual birds.
Sparse distribution: Unlike humans, birds in nature need enough space to move around, which means birds are more sparsely distributed in space. Coupled with the smaller tar- get size of birds, the sparseness of the foreground portion is even more pronounced in DroneBird compared the already existing dataset.
Fast movement: Unlike existing video crowd-sourced data, the unique flight movements of birds allow them to do fast movements, posing a challenge for better temporal modeling.
New counting category: DroneBird fills the gap of missing video bird data in the field. The models trained on crowd data fail to conduct bird counting directly (See Table We evaluated the performance of our method trained on the VSCrowd dataset (Exp. T) and the DroneBird training set (Exp. 2) on the DroneBird testing set. The results show that the model trained on the VSCrowd dataset failed to perform well on the bird data, which indicates the large gap between crowd and bird data.
A.2 DETAILED METHOD
We detail the training process of our E-MAC method Algorithm The training process of DEMO is represented in Algorithm 2
As shown in Algorithm I our model requires two frames (11)) and their corresponding den- sity maps (De, D,-1) as input. The image and density map pairs S, {Le, D.} and S (J--1, ) are fed to the DEMO to predict the density map D, and D. Meanwhile, a pretrained optical flow estimation network (Sun et al. 2018) is performed on D, and D.-1 to generate optical flow M, which is used to warp the D.-1 to D. Then the D and D, are fed into a cross- attention layer to calculate the residual Di of adjacent predicted density map. The final predicted density map Drsse of T, is then calculated by a wise add of D, and Dçe.
15

---

Published as a conference paper at ICLR 2025
Algorithm Framework Workflow in Training Phase
Ensure: {4. Dr). (4a, D
Require: Druse
1: for all epoch do
2:
D @DENo (Le, D+)
3:
D. f- chenn (I: I 1t11, Dz-1)
4:
M + OppticalFion (1, IT))
D +- @warpl (M, D(t)
5:
6:
DrS t- ( (D warp D D.)
7:
Druse &- Dt Dees
8:
Cluse: Feurs Cops, LTv t- 0Loss
9:
C C Ctuse + X2Ceur + >z + XnCTv
10: end for
Algorithm DEMO workflow in training phase
Ensure: D,D
Require: 1)
Tr; TD, potch: D patch < patchift (1, D)
2: VD Osum (D patch)
3: N = random (0,
4: if N < 1 then
5:
K < argsort (VD)
6: else
7: K. argsort (D)
8: end if
9: for all
t; E Ty do if KC then
11:
M +0
else
13:
M +
14: end
15: end for
16: T T + mask(T, Madaptive) mask(Tp, Mandom
17: T Oconcate T T
18: T < Dencuder T
19: THt T <- S
20: Mask random
21: Tn + Óan (T Mask)
22: TD + dcal (T Tee)
23: D +- ddeccdder (T))
In the DEMO, video frame I and its corresponding density map D performed patchify and projection operation to get the token set T,, Tn and patch set Iucch: D Dpatch- Then we perform mask operation get the retained tokens Tist Tee The number of retained tokens for the image and density map modalities are generated by the Dirichlet distribution, denoted as Nr and A respectively. Specifically, for image modality, we use an adaptive mask obtain T from Ty. Firstly, we calculate the number of targets V in the i-th density map patch D Patchh and Vi 2smm ). where @sm represents the pixel-wise sum operation each density map patch, and the result represents the number of targets in the corresponding patch. To focus on the foreground, sort the image tokens according the number of targets V", in the corresponding density map patch. Although the foreground provides more valid information, the background should not be completely ignored. Therefore, introduce background information, probability (BRP) P to control the sorting manner. The tokens are sorted in ascending order with a probability of P (focus background) or in descending order with probability of P (focus
background retention
16

---

Published as a conference paper at ICLR 2025
Sum Density Map Patchess Dparch
Deseending Soct with Probability
Resture
+ 26
Shuffle
NO
Restore
Envoder
1616 13 20
Predict
Figure 8: The detailed process of SAM. For ease of expression, we crop the image and the density map into 25 patches, as shown in the figure, this number varies according to the size of input images (each patch is set to 16 16). We first calculate the Vi of each density map patch D patch- and sort the token according to the number of targets Vb. To balance the fore- and back-ground informa- tion, we set a background retention probability (BRP) P to determine the sorting manner, which is detailed Sec. Sec. For tokens from image modality, we keep the first M tokens after sorting. For tokens from density map modality, ie., randomly select 7\ density tokens. Since the masked tokens are filled by learnable tokens in the decoder, first restore the retained image and density tokens to their original order before sorting. Then, we concatenate and feed them into the encoder. Note that the restoration follows the same setting with Bachmann al. (2022), which can be performed in the decoder well. The full set of retained tokens and filled density map tokens are then fed into the decoder to predict the density map. Specifically, the full of retained tokens are treated as key and value vectors, and the filled density map tokens are treated as query vector in the cross-altention layer in the decoder. Then, the output of the cross-attention layer is fed into two self-attention layers perform the final prediction.
randomly shuffle their order and keep the first Au tokens,
foreground). The first Nyet tokens after sorting is then retained as TT. For density map modality, randomly shuffle the order of Tn and retain the first Aftt tokens (randomly mask) T The retained tokens The and THE are jointly fed into the encoder, while the remaining tokens are discarded and not passed into the model. The output tokens Tee encoder are then split to Tp and T The learnable random mask tokens are filled at the masked positions in T as placeholders, and the filled tokens set denoted as TD. TD and Tee are then fed in a cross-attention layer, in which TD servers as query and Toe servers as key and value. Then, the reconstructed density map δ is regressed by two transformer layers.
We have illustrated the detailed SAM process in Fig. For ease of expression, we crop the image and density map into 25 patches. Each patch from the image and the density map at the same position is paired, and we have noted an index for each pair of patches in Fig. g For each pair of patches (Ipanch Djusch). we calculate the sum of the density map patch Vb. sort the token pair (Tr, Tp) according the number of targets Vi, We set background retention probability (BRP) P to determine the sorting manner, which is detailed in Sec. 4.3 For tokens from image modality, retain the first A/ tokens in the sorted Ty. For tokens from density map modality, randomly shuffle their order and retain the first / tokens. Since the masked tokens will be filled by learnable tokens in the decoder, we first restore the retained image and density tokens their original order
17

---

Published as a conference paper at ICLR 2025
Table 6: Quantitative comparison between the proposed method and existing methods the Mall dataset with metries MAE and RMSE.
Method
Type
MAE
RMSE
CSRNet Li et al.
2018
Image
2.46
4.70
RPNet Yang et al. 2020
Image
2.20
3.60
TAN (Wu
2020
Image
2.03
2.60
HMoDE (Du et al. 20233
Image
2.82
3.41
PET Liu al. 20233)
Image
1.89
2.46
Gramlormer (Lin et al.
2024
Image
1.69
2.14
ConvLSTM (Xiong et al.
2017
Video
2.24
8.50
LSTN (Fang el al. (019))
Video
2.00
2.50
E3D Zou et al. 2019)
Video
1.64
2.13
MLSTN (Fng et al. (2220)
Video
1.80
2.42
MOPN (oossnn
2020
Video
1.78
2.25
Monet
Bai Chan
2021
Video
1.54
2.02
PFTF (vvenutt al.
2022
Video
2.99
3.72
FRVCC (Hou et al. 2023)
Video
1.41
1.79
STGN (Wu al. 2023)
Video
1.53
1.97
Ours
Video
1.35
1.76
Table Quantitative comparison between our proposed method and existing methods on the FDST dataset with metrics MAE and RMSE, lower metrics better.
Method
Type
MAE
RMSEL
MCNN Zhang et al. 2016)
Image
3.77
4.88
CSRNet
2018
Image
2.56
3.12
Chfl Shu et al. 2022
Image
3.33
4.38
MAN (Lin
al. 2022
Image
2.79
4.21
HMoDE (Du al. 2023
Image
2.49
3.51
PET Liu et al. 2023)
Image
1.73
2.27
Gramformer (Lin et al.
2024
Image
5.15
6.32
COnVLSTM (Xiong et
2017
Video
4.48
5,82
LSTN (Fang et al. 20199
Video
3.35
4.45
MLSTN Fang al. (2020)
Video
2.35
3.02
EPF (Liu et al. 2020)
Video
2.17
2.62
MOPN (Hossain et al.
2020
Video
1.76
2.25
PHNet Meng et al. (02T
Video
1.65
2.16
GNANer (L. et al.
2022)
Video
2.10
2.90
PFTF (Avvenuti al.
2022
Video
2.07
2.69
FRVCC Hou et al. 2023)
Video
1.88
2.45
STGN Wu et al. 2023)
Video
1.38
1.82
Ours
Video
1.29
1.69
before sorting. Then, we concatenate and feed them into the encoder. The full set of retained tokens and filled density map tokens are then fed into the decoder to predict the density map. Specifically, the full set of retained tokens are treated as key and value vectors, and the filled density map tokens are treated as query vector in the cross-attention layer in the decoder. Then, the output of the cross- attention layer is fed into two self-attention layers perform the final prediction.
A.3 MORE EXPERIMENT RESULTS
Additional results on the Mall, FDST, and VSCrowd datasets are provided Tabs. and In extensive survey, method consistently achieved competitive results.
18

---

Published as a conference paper at ICLR 2025
Table 8: Quantitative comparison between proposed method and existing methods the VSCrowd dataset with metrics MAE and RMSE, lower metrics better.
Method
Type
MAEI
RMSEL
MCNN Zhang et al. 2016)
Image
27.1
46.9
CSRNet Li al. 2018)
Image
13.8
21.1
Bayesian (Ma ct al. 2019
Image
8.7
11.8
MAN (Lin et al.
Image
8.3
10.4
HMoDE (Du et al. 2023
Image
19.8
39.5
PET (Liu et
2023)
Image
6.6
11.0
Granformer (Lin et al.
2024
Image
8.09
15.65
EPF (Liu et al. 2020)
Video
10.4
14.6
GNANet (Li al.
2022)
Video
8.2
10.2
STGN (Wu et al.
2023)
Video
9.6
12.5
Ours
Video
6.0
10.3
Table 9: Effect of TCF,
Table 10: Effect of E-MAC architecture.
Exp.
Model
MAE,
RMSEJ
Exp.
Model
MAE
RMSE
VIT
2.45
3.22
vanilla VIT
2.63
3.31
VIT TCF
2.32
2.69
MultiMAE
2.45
3.22
5
E-MAC
1.51
2.03
E-MAC w/ vanilla VIT weight
1.49
1.93
6
E-MAC ICE
1.29
1.69
10
E-MAC w/ MultiMAE weight
1.29
1.69
A.4 IMPACT OF IMAGE SIZE
We conducted experiments on the FDST dataset to explore the effect of image size on the perfor- mance of our E-MAC module. We tried a variety of image input sizes and compared their exper- imental results to validate the effect of input image size. When we increased the image size used for training from 224 224 480 x 480, the final MAE showed downward trend in the figure and decreased from 1.93 to 1.47, which improved by 24%. However, the computational cost of the model increases exponentially the size of the image increases (Dosovitskiy et al., 2020). Inte- grating temporal information further increases the computational cost, thereby affecting the overall performance. Therefore, the performance and computational cost.
the input image size 320 320 on the FDST dataset, balancing
A.5 IMPACT OF TCF
Additionally, we have conducted additional ablation studies the TCF module, shown in Table We compare the performance of the original ViT architecture (Exp. in Table the VIT framework augmented with TCF (Exp. 4 in Table and E-MAC with without TCF (Exp. 5 and in Table The results demonstrate that TCF provides performance improvements of 5% and 15% on the two architectures, respectively. To further demonstrate how the fusion module works, we visualized the input D,, and output D of the fusion module as well as the intermediate variable D under three datasets, as shown in Fig(9] we provide the original image I, and corresponding ground truth D, in the first two rows of the figure for reference. Compared with D., the fusion module can well realize the correction of the current density map by correlating the residuals obtained from the previous and current predicted density map, which makes the integration of the fused density map Dusee closer to the ground truth. We zoom in some of the background interference by correlating the front and back predicted density maps, which makes the final predicted density map better.
areas, and notice that the fusion module remove
19

---

Published as a conference paper at ICLR 2025
SCRMEI
SCENE3
SCINIT
Inugh
Qlosall TreiLA
22.00
RadlDonie Aap0p
21.40
35.45
32.07
Prolexe
27.99
42/44
33.89
Proticaa Ktonkal Melp 02
4.45
Figure 9: Visualization of the output and intermediate variables in the Fusion Module.
mask visualization
predicted donsity map
random mask SAM (P-1) SAM (P-0.8)
100
80
random musk
60
20
SAM
hackground patch
foreground patch
Figure 10: Visualization of the predicted density map w/ and w/o SAM, and statistical results of the number of foreground and background patches w/ and w/o SAM.
A.6 VISUALIZATION SAM
We visualized the differences between the density maps predicted from the image w/ and w/o SAM and the ground truth. presented in Fig. 10 The zoom-in regions display intuitive and significant visual differences. E-MAC trained w/ SAM effectively counts the targets, while E-MAC using random masking fails to count some targets. Besides, we conducted statistical comparison of the number of foreground and background patches in the images after applying random masks and different P (BRP) settings with SAM, as presented in Fig. [0] Obviously, SAM significantly reduces the proportion background regions, thereby balancing positive and negative samples.
20

---

Published as a conference paper at ICLR 2025
Image
Truth
Predicted Density
Predicted Dansiry (E-MAC w TCF)
Predicted (E-MAC wio SAM)
Density (E-MAC)
(MuliMAB)
Figure 11: Visualization of the predicted results of our key components (TCF, DEMO, SAM),
Table 12: Comparison on FLOPs, FPS, and the number of Parameters.
Method
FLOPs(G)
FPS 1 Parameters(M) 1. MAE (DroneBird) J RMSE (DroneBird) J
EPF [Liu et al. 2020 PFTF Avvenun et 2022
815
19
20
97.22
133.01
1075
13
23
89.76
101.02
STGN eW al. 2023
742
30
13
92.38
124.67
Ours
811
16
38.72
42,92
We have also visualized some of the predicted results of key components (TCF, DEMO, SAM) with our baseline, as shown in Fig. II We gradually add the components (TCF, DEMO, SAM) to the baseline, resulting in increasingly accurate predictions and better visual results.
Table 11: Result with different random seeds.
Metries
Average
MAE
1.29
1.28
1.33
1.30
RMSE
1.69
1.66
1.69
1.68
A.7 EFFECT OF E-MAC ARCHITECTURE.
We provide additional quantitative experiments demonstrate the performance sources of our pro- posed framework. As shown in Table IO, we provide a comparison of the performance of the vanilla VIT architecture using different pre- pre-training weights (Exp. 7 and 8 in TableT00 as well as our frame- work using different pre-traine traitrain weights (Exp. and 10 in Table T0P. As can be seen from the table: using a larger pre-training weight ViT does provide better performance, but the improvement is very limited (6% improvement between Exp. 7 and 8). In contrast, our unique design tailored to the data and video counting tasks significantly enhances performance (43% between Exp. 7 and 9 and 47% between Exp. 8 and 10 respectively).
A.8 IMPACT OF RANDOM SEED.
To explore the impact of different random seeds on the overall performance of our model, we have also conducted more experiments the FDST dataset, which is relatively small dataset for quick validation. The average MAE and RMSE scores are 1.30=z.022 and 1.68 The results in Table 11 show that different random seeds do not significantly affect the performance of model.
A.9 MODEL. EFFICIENCY DISCUSSION
We have compared FLOPs, FPS, and the number of parameters with some other video counting models and reported the results in Table 12 All the experiments were conducted on an NVIDIA RTX 3090 GPU. It is worth noting that the FLOPs and FPS of our method and the competing methods are comparable, although we first adopted the vision foundation model. Compared to EPF (Liu et al. (220)) and PFTF (Avvenuti et al., (022), our E-MAC achieves superior performance with less required computations. STGN (Wu et al.] 2023) achieves higher FPS with fewer parameters, but its performance is limited. Compared to STGN (Wu et al. 2023), our method achieved over 58% performance improvement with only 9% more FLOPS.
A.10 DYNAMICS DATA
Here, we explore the intra-frame dynamics of the foreground regions in video data. We processed data in the FDST dataset and made analyses. We first utilized a 60 x 60 win- dow to crop the images into patches and then counted the number of people each patch.
21

---

Published as a conference paper at ICLR 2025
Statistical result is shown in Fig. the hori- zontal axis coordinates are the density of people in each patch, calculated from the correspond- ing density map. The left vertical coordinate the number of patches for each crowd den- sity. We fold a portion of the axis as the number gap is large. The heterogeneous density dis- tribution across image patches indicates inher- ent dynamism in the intra-framt density char- acteristics. Additionally, due to the presence of large background areas, the model should fo- cus more on the foreground regions of the sam- ples to extract the most informative and rele- vant information. The utilization of fixed fo- cus region across different samples may result in the loss of critical information about the fore- ground areas. Similarly, the adoption of com- pletely random focus regions unable con- sistently capture the salient information within the foreground regions. Thus, suggest the employment of dynamic masking mechanism obtain the foreground focus regions for different samples.
955
9055
305
65%
60%
81
02
05
96
0J
08
00
12
13
14
1.5
Figure 12: Density distribution. The bar graph portion (blue) represents the number of patches corresponding to the crowd density. The line graph portion (red) represents the percentage of the number of patches whose density is less than the current density.
22


Reviews:
[{"review_id": "BuYedvtM4c", "reviewer": "Reviewer_3dEQ", "summary": "This paper proposes a large video bird counting dataset DroneBird, in natural scenarios for migratory bird protection. In addition, an efficient masked autoencoder for video crowd counting framework is proposed. Specifically, to integrate the foundational model and take the density map as an auxiliary modality to perform self-representation learning, a density-embedded efficient masked autoencoder counting framework is proposed. Moreover, the authors propose an efficient spatial adaptive masking method to overcome the dynamic density distribution and make the model focus on the foreground regions. Extensive experiments on three crowd datasets and our DroneBird validate the effectiveness.", "questions": "- One open question: As we all know that MAE pre-training can improve many downstream task fine-tuning. For video crowd counting, does the current approaches also benefit from the MAE pre-trained models (e.g., MAE ViT-base)? i.e., using them as the initialization weights for video counting fine-tuning.\n\nAs mentioned in the Weaknesses, my main concerns lie in the ambiguous performance gains and methodology design. Since the proposed approach uses the pre-trained MultiMAE model and jointly follows a masked autoencoder framework for fine-tuning, it is not clear whether the performance improvements stem from the pre-trained model or downstream masked fine-tuning with the designed framework from the authors.  I would like to see the author rebuttal in terms of this part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n- A large bird video counting dataset is proposed, which is helpful for bird activities analysis;\n- Applying masked authencoding learning for video crowd counting, which is interesting;\n- The proposed approach achieves SOTA video counting performance on several benchmarks, especially for the proposed bird dataset;", "weaknesses": "Weaknesses: \n- Ambiguous performance gains: The proposed approach integrates a masked autoencoder for representation learning and formulates the previous density regression task as \"density reconstruction\" within an MAE-like framework for crowd counting. Additionally, the authors use the pre-trained ViT-B from MultiMAE as the encoder for discriminative feature extraction. However, given that pre-trained MAE models typically lead to significant performance improvements during downstream task fine-tuning, it remains unclear whether the observed performance gains stem from the designed new masked autoencoder-based framework or the usage of pre-trained MultiMAE. One suggestion: compared w/ a variant w/o using the pre-trained ViT;\n- For the proposed Temporal Collaborative Fusion, it is unclear why we need to calculate $\\hat{D}_{t}^{res}$ again after obtaining $\\hat{D}_{t}$ and $\\hat{D}_{t-1}$ from the decoder. If $\\hat{D}_{t}$ and $\\hat{D}_{t-1}$ are not accurate enough, then $\\hat{D}_{t}^{res}$ calculated using them will also be inaccurate. Conversely, if $\\hat{D}_{t}$ and $\\hat{D}_{t-1}$ are accurate, there seems to be no need for the Temporal Collaborative Fusion module.\n- A comparison of model complexity with the other approaches in Table 1 should be provided to ensure a fair evaluation, as the current work may use a relatively larger model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zndYkpdEKW", "reviewer": "Reviewer_P9ZV", "summary": "This paper presents a method and a new dataset for video object counting. The model performs well in multiple object-counting benchmarks using video as input.", "questions": "- In video counting, how is the count of a video defined? Is it the average count of objects, or the number of different instances that appear in the video?\n- The TCF module presented in Fig. 2 is inconsistent with the corresponding description. In Eq. (1), the product of the \"warp\" operator is added to $\\hat{D}_t$ to generate $\\hat{D}_{\\text{fuse}}$, and the \"warp\" operation involves $\\hat{D}_t$. In the description, a cross-attention is applied to generate $\\hat{D}_t^{\\text{res}}$, which is then combined with $\\hat{D}_t$ to produce $\\hat{D}_{\\text{fuse}}$. In Fig. 2, the generation of $\\hat{D}_t^{\\text{res}}$ does not involve $\\hat{D}_t$ according to the arrows that demonstrate the data flow.\n- In Eq. (2), $\\mathbf{T}_D \\in \\mathbb{B}^{B \\times \\mathcal{N}_D \\times C}$ is a token set including vectors. How are these vectors sorted?\n- The description of SAM is also different from that in Fig. 2. The description takes the ground truth $D_t$ to mask tokens, while SAM in Fig. 2 involves $I_{t-1}$, $D_{t-1}$, and a random mask, which does not appear in Sec. 4.3.\n- During training, $\\mathbf{S}_t = \\{I_t, D_t\\}$ contains the input image and GT labels. How does the model work in inference, without GT as input?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- A video object counting benchmark, Dronebird, is established.\n- A corresponding object counting method to deal with video data is proposed.", "weaknesses": "Many descriptions are not inconsistent, making it difficult to understand the paper. See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eHLd5cbR5w", "reviewer": "Reviewer_Dnim", "summary": "This paper presents the E-MAC framework, which addresses the dynamic imbalance of fore-background in video object counting through density-embedded Efficient Masked Autoencoder Counting. By leveraging optical flow-based temporal collaborative fusion, E-MAC aligns features across frames to improve counting accuracy. The approach introduces Density-Embedded Masked Modeling (DEMO) to enhance intraframe representation of dynamic foreground objects. To mitigate redundant background information, an efficient spatial adaptive masking derived from density maps is employed. Additionally, the paper introduces DroneBird, a large-scale bird counting dataset for migratory bird protection, showcasing the framework’s effectiveness in natural scenarios and validating its superiority through extensive experiments on three crowd datasets.", "questions": "1.In Table 1, do the compared methods use the same backbone network (ViT-B) as this paper's method? If not, please specify the reasons.\n\n2.Why does the proposed method show higher RMSE on the VSCrowd dataset compared to GNANet in Table 1?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The efficient spatial adaptive masking method adeptly handles dynamic density distribution, ensuring focus on foreground regions and tackling the imbalance of fore-background effectively.\n\n2.Introduction of the DroneBird dataset, the first video bird counting dataset, significantly broadens the application domain, demonstrating the framework’s versatility and impact through extensive validation against competing methods.", "weaknesses": "1.The manuscript lacks visual analyses of the three key components, which would illustrate the targeted effectiveness of each module.\n\n2.It is recommended to provide code or critical sections of the code for review to enable thorough evaluation of the methodology and results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2kvVJ37SAw", "reviewer": "Reviewer_rdoG", "summary": "1. This paper addresses video object counting by introducing the DroneBird dataset for bird-counting scenarios and proposes a masked autoencoder-based counting framework.\n\n2. The framework uses optical flow for temporal alignment and density maps for spatial focus to improve counting performance.\n\n3. While each component is technically feasible, they lack integration to showcase a comprehensive advantage for crowd counting as a whole.\n\n4. The dataset offers potential application value, though its academic innovation and contribution remain to be further substantiated.", "questions": "**Questions:**\n\n1. In Table 1, video-based methods underperform image-based methods on the DroneBird dataset. What might be the reasons for this discrepancy, and could this be further elaborated?\n  \n2. Optical flow is typically more effective for capturing dynamic objects in scenes with a static camera. In the drone viewpoint of DroneBird, how does the model address the challenge where all objects in the field of view are in motion? Particularly, TCF’s improvements are minor according to the ablation study—how might this be explained?\n\n3. How is Exp. III in Table 2 implemented? Given my understanding, SAM’s application without DEMO may lack practical relevance—how does it contribute in the absence of DEMO?\n\n4. Could you further articulate the scientific value of DroneBird, especially in terms of the new challenges it introduces for video-based research? Specifically, what novel research questions or difficulties does it bring?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Introduces a novel bird-counting dataset, DroneBird, captured from a drone’s perspective. \n\n2. Proposes optical flow-guided Temporal Collaborative Fusion (TCF) to extract temporal information from adjacent frames, enhancing object counting accuracy.\n\n3. Integrates Density-Embedded Masked Modeling (DEMO), using density maps as an auxiliary modality with image data, helping balance foreground and background in the model.\n\n4. Employs Spatial Adaptive Masking (SAM) to focus on foreground areas, potentially improving model efficiency and counting accuracy.", "weaknesses": "1. The methodology's components (such as DEMO, SAM, and TCF) lack cohesive integration, appearing independently implemented without clearly demonstrated necessity for crowd counting.\n\n2. Although DroneBird introduces a new category, the dataset’s difficulty and uniqueness are not strongly evident, and it lacks clear differentiation in research value compared to existing datasets beyond adding a bird-counting scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)