Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark

Authors: Bing Cao, Quanhao Lu, Jiekang Feng, Qilong Wang, Pengfei Zhu, Qinghua Hu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video object counting, masked autoencoder, multimodal self-representation learning

Abstract:
The dynamic imbalance of the fore-background is a major challenge in video object counting, which is usually caused by the sparsity of target objects. This remains understudied in existing works and often leads to severe under-/over-prediction errors. To tackle this issue in video object counting, we propose a density-embedded Efficient Masked Autoencoder Counting (E-MAC) framework in this paper. To empower the model’s representation ability on density regression, we develop a new Density-Embedded Masked mOdeling (DEMO) method, which first takes the density map as an auxiliary modality to perform multimodal self-representation learning for image and density map. Although DEMO contributes to effective cross-modal regression guidance, it also brings in redundant background information, making it difficult to focus on the foreground regions. To handle this dilemma, we propose an efficient spatial adaptive masking derived from density maps to boost efficiency. Meanwhile, we employ an optical flow-based temporal collaborative fusion strategy to effectively capture the dynamic variations across frames, aligning features to derive multi-frame density residuals. The counting
accuracy of the current frame is boosted by harnessing the information from adjacent frames. In addition, considering that most existing datasets are limited to human-centric scenarios, we propose a large video bird counting dataset, DroneBird, in natural scenarios for migratory bird protection. Extensive experiments on three crowd datasets and our DroneBird validate our superiority against
the counterparts. The code and dataset are available.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730466439180

thread_type:
official_review

<thread_object>
creator: reviewer_rdog

<content>
summary:
1. This paper addresses video object counting by introducing the DroneBird dataset for bird-counting scenarios and proposes a masked autoencoder-based counting framework.

2. The framework uses optical flow for temporal alignment and density maps for spatial focus to improve counting performance.

3. While each component is technically feasible, they lack integration to showcase a comprehensive advantage for crowd counting as a whole.

4. The dataset offers potential application value, though its academic innovation and contribution remain to be further substantiated.

strengths:
1. Introduces a novel bird-counting dataset, DroneBird, captured from a drone’s perspective. 

2. Proposes optical flow-guided Temporal Collaborative Fusion (TCF) to extract temporal information from adjacent frames, enhancing object counting accuracy.

3. Integrates Density-Embedded Masked Modeling (DEMO), using density maps as an auxiliary modality with image data, helping balance foreground and background in the model.

4. Employs Spatial Adaptive Masking (SAM) to focus on foreground areas, potentially improving model efficiency and counting accuracy.

weaknesses:
1. The methodology's components (such as DEMO, SAM, and TCF) lack cohesive integration, appearing independently implemented without clearly demonstrated necessity for crowd counting.

2. Although DroneBird introduces a new category, the dataset’s difficulty and uniqueness are not strongly evident, and it lacks clear differentiation in research value compared to existing datasets beyond adding a bird-counting scenario.

questions:
**Questions:**

1. In Table 1, video-based methods underperform image-based methods on the DroneBird dataset. What might be the reasons for this discrepancy, and could this be further elaborated?
  
2. Optical flow is typically more effective for capturing dynamic objects in scenes with a static camera. In the drone viewpoint of DroneBird, how does the model address the challenge where all objects in the field of view are in motion? Particularly, TCF’s improvements are minor according to the ablation study—how might this be explained?

3. How is Exp. III in Table 2 implemented? Given my understanding, SAM’s application without DEMO may lack practical relevance—how does it contribute in the absence of DEMO?

4. Could you further articulate the scientific value of DroneBird, especially in terms of the new challenges it introduces for video-based research? Specifically, what novel research questions or difficulties does it bring?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625566404

thread_type:
official_review

<thread_object>
creator: reviewer_dnim

<content>
summary:
This paper presents the E-MAC framework, which addresses the dynamic imbalance of fore-background in video object counting through density-embedded Efficient Masked Autoencoder Counting. By leveraging optical flow-based temporal collaborative fusion, E-MAC aligns features across frames to improve counting accuracy. The approach introduces Density-Embedded Masked Modeling (DEMO) to enhance intraframe representation of dynamic foreground objects. To mitigate redundant background information, an efficient spatial adaptive masking derived from density maps is employed. Additionally, the paper introduces DroneBird, a large-scale bird counting dataset for migratory bird protection, showcasing the framework’s effectiveness in natural scenarios and validating its superiority through extensive experiments on three crowd datasets.

strengths:
1.The efficient spatial adaptive masking method adeptly handles dynamic density distribution, ensuring focus on foreground regions and tackling the imbalance of fore-background effectively.

2.Introduction of the DroneBird dataset, the first video bird counting dataset, significantly broadens the application domain, demonstrating the framework’s versatility and impact through extensive validation against competing methods.

weaknesses:
1.The manuscript lacks visual analyses of the three key components, which would illustrate the targeted effectiveness of each module.

2.It is recommended to provide code or critical sections of the code for review to enable thorough evaluation of the methodology and results.

questions:
1.In Table 1, do the compared methods use the same backbone network (ViT-B) as this paper's method? If not, please specify the reasons.

2.Why does the proposed method show higher RMSE on the VSCrowd dataset compared to GNANet in Table 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698248644

thread_type:
official_review

<thread_object>
creator: reviewer_p9zv

<content>
summary:
This paper presents a method and a new dataset for video object counting. The model performs well in multiple object-counting benchmarks using video as input.

strengths:
- A video object counting benchmark, Dronebird, is established.
- A corresponding object counting method to deal with video data is proposed.

weaknesses:
Many descriptions are not inconsistent, making it difficult to understand the paper. See Questions.

questions:
- In video counting, how is the count of a video defined? Is it the average count of objects, or the number of different instances that appear in the video?
- The TCF module presented in Fig. 2 is inconsistent with the corresponding description. In Eq. (1), the product of the "warp" operator is added to $\hat{D}_t$ to generate $\hat{D}_{\text{fuse}}$, and the "warp" operation involves $\hat{D}_t$. In the description, a cross-attention is applied to generate $\hat{D}_t^{\text{res}}$, which is then combined with $\hat{D}_t$ to produce $\hat{D}_{\text{fuse}}$. In Fig. 2, the generation of $\hat{D}_t^{\text{res}}$ does not involve $\hat{D}_t$ according to the arrows that demonstrate the data flow.
- In Eq. (2), $\mathbf{T}_D \in \mathbb{B}^{B \times \mathcal{N}_D \times C}$ is a token set including vectors. How are these vectors sorted?
- The description of SAM is also different from that in Fig. 2. The description takes the ground truth $D_t$ to mask tokens, while SAM in Fig. 2 involves $I_{t-1}$, $D_{t-1}$, and a random mask, which does not appear in Sec. 4.3.
- During training, $\mathbf{S}_t = \{I_t, D_t\}$ contains the input image and GT labels. How does the model work in inference, without GT as input?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730821783689

thread_type:
official_review

<thread_object>
creator: reviewer_3deq

<content>
summary:
This paper proposes a large video bird counting dataset DroneBird, in natural scenarios for migratory bird protection. In addition, an efficient masked autoencoder for video crowd counting framework is proposed. Specifically, to integrate the foundational model and take the density map as an auxiliary modality to perform self-representation learning, a density-embedded efficient masked autoencoder counting framework is proposed. Moreover, the authors propose an efficient spatial adaptive masking method to overcome the dynamic density distribution and make the model focus on the foreground regions. Extensive experiments on three crowd datasets and our DroneBird validate the effectiveness.

strengths:
Strengths:
- A large bird video counting dataset is proposed, which is helpful for bird activities analysis;
- Applying masked authencoding learning for video crowd counting, which is interesting;
- The proposed approach achieves SOTA video counting performance on several benchmarks, especially for the proposed bird dataset;

weaknesses:
Weaknesses: 
- Ambiguous performance gains: The proposed approach integrates a masked autoencoder for representation learning and formulates the previous density regression task as "density reconstruction" within an MAE-like framework for crowd counting. Additionally, the authors use the pre-trained ViT-B from MultiMAE as the encoder for discriminative feature extraction. However, given that pre-trained MAE models typically lead to significant performance improvements during downstream task fine-tuning, it remains unclear whether the observed performance gains stem from the designed new masked autoencoder-based framework or the usage of pre-trained MultiMAE. One suggestion: compared w/ a variant w/o using the pre-trained ViT;
- For the proposed Temporal Collaborative Fusion, it is unclear why we need to calculate $\hat{D}_{t}^{res}$ again after obtaining $\hat{D}_{t}$ and $\hat{D}_{t-1}$ from the decoder. If $\hat{D}_{t}$ and $\hat{D}_{t-1}$ are not accurate enough, then $\hat{D}_{t}^{res}$ calculated using them will also be inaccurate. Conversely, if $\hat{D}_{t}$ and $\hat{D}_{t-1}$ are accurate, there seems to be no need for the Temporal Collaborative Fusion module.
- A comparison of model complexity with the other approaches in Table 1 should be provided to ensure a fair evaluation, as the current work may use a relatively larger model.

questions:
- One open question: As we all know that MAE pre-training can improve many downstream task fine-tuning. For video crowd counting, does the current approaches also benefit from the MAE pre-trained models (e.g., MAE ViT-base)? i.e., using them as the initialization weights for video counting fine-tuning.

As mentioned in the Weaknesses, my main concerns lie in the ambiguous performance gains and methodology design. Since the proposed approach uses the pre-trained MultiMAE model and jointly follows a masked autoencoder framework for fine-tuning, it is not clear whether the performance improvements stem from the pre-trained model or downstream masked fine-tuning with the designed framework from the authors.  I would like to see the author rebuttal in terms of this part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732252591501

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
Dear PCs, SACs, ACs, and Reviewers,

We would like to than you for your valuable feedback and insightful reviews, which have greatly contributed to imporving the paper. In this paper, we proposed an **interesting** (Reviewer 3dEQ) framework for video object counting, which **tackles the imbalance of fore-background effectively** (Reviewer Dnim) and **handles dynamic density distribution** (Reviewer Dnim), experiments on multiple dataset validata E-MAC's **efficiency**  (Reviewer rdoG) and **feasible** (Reviewer rdoG). We also proposed a **new** (Reviewer P9ZV) **novel** (Reviewer rdoG) and **helpful bird counting dataset** (Reviewer 3dEQ), which **significantly broadens the application domain** (Reviewer Dnim).

We hope that our responses will satisfactorily address your questions and concerns. We address the reviewers' concerns with the following updates and improvements:

1. **Extended Analysis and Experiments**: More ablation studies including the impact of pretrained weight, effect of $\texttt{TCF}$ and visual analyses of the key components ($\texttt{TCF}, \texttt{DEMO}, \texttt{SAM}$).
2. **Detailed Descriptions**: More detailed descriptions of the key components ($\texttt{TCF}, \texttt{DEMO}, \texttt{SAM}$) and more detailed framework diagram.
3. **More dataset analysis**: Detailed analysis of the DroneBird dataset, including the comparison with existing datasets and the challenges of the new dataset.
4. **Point-by-Point Responses**: Comprehensive responses to specific reviewer comments.

We sincerely appreciate the time and effort you have dedicated to reviewing our submission, along with your invaluable suggestions. If you have any other concers, please feell free to contact us. We look forward to discussing with you. We have released an anonymous [URL](https://anonymous.4open.science/r/E-MAC-4631/README.md) with our code of model for better understanding, and we have also prepared an updated manuscript and supplementary material that addresses the reviewers' concerns.

Sincerely,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733138588877

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A gentle reminder for the close of the author-reviewer discussion & summary of the rebuttal

comment:
Dear PCs, SACs, ACs, and Reviewers,

As the author-reviewer discussion period is closing at the end on December 2nd, we would like to call for any further discussion or comments on our submission.

In our rebuttal, we addressed the following raised concerns/misunderstandings:

- We have provided more experimental results and visualizations for validating each component.
- We have provided more explanations for the pipeline of E-MAC and more details for some figures.
- We have added experiments on the efficiency of our method.
- We have added comparisons with some competing methods on different pre-trained models.
- We have clarified the new challenges of our DroneBird dataset for video counting.
- We have carefully checked and revised all the representation issues throughout the paper.

We believe that these clarifications and additional details strengthen our paper and address the reviewers' concerns. We would like to thank the reviewer for recognizing our work **tackling the imbalance of fore-background effectively** and that our dataset **significantly broadens the application domain** (Reviewer Dnim). We are also delighted that the reviewers believe our responses have **addressed their concerns** (Reviewer 3dEQ, rdoG), and consider our submission to be **a nice work in video counting** (Reviewer P9ZV). 

We understand the constraints of time and workload that reviewers and AC face, and we appreciate the effort already put into evaluating our work. If there are any additional insights, questions, or clarifications on our responses/submission that you would like to discuss with us, we would be very grateful to hear them, your feedback is invaluable for the improvement of our research.


Best regards,

Authors of submission 678
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper initially received mixed reviews 5563. The major concerns were:

1. missing ablation study showing the effect of the Multi-MAE backbone [3dEQ] 
2. missing comparison of model complexity [3dEQ]
3. motivation for TCF, and needs better description? [3dEQ, P9ZV]
4. comparison using MAE for other video counting methods [3dEQ]
5. how to do inference without GT input? [xwMA]
6. are experiments using the same backbone in Table 1? [Dnim]
7. method components lack cohesive integration [rdoG]
8. what is the value of the new dataset? [rdoG]
9. video-based methods underperform image ones on DroneBird, why? [rdoG]

The authors wrote a response, and had significant discussions with the reviewers. After the discussion, ratings were increased to 6865. The remaining negative reviewer thought that there was little theoretical contribution and incremental novelty, but also conceded that the proposed method could be used in other areas or vision applications, such as tiny object detection and segmentation, and thus could generally be interesting to the community. 

Overall, the reviewers noted that the use of MAE architecture for video crowd counting was interesting, and achieved SOTA results. The AC agrees and recommend accept. The authors should revise the paper according to the reviews, response, and helpful discussions.