Scale-Invariant Continuous Implicit Neural Representations For Object Counting

Authors: Siyuan Xu, Yucheng Wang, Xihaier Luo, Byung-Jun Yoon, Xiaoning Qian
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: scale invariance, implicit neural representation, object counting

Abstract:
Many object counting methods rely on density map estimation (DME) using convolutional neural networks (CNNs) on discrete grid image representations. However, these methods struggle with large variations in object size or input image resolution, typically due to different imaging conditions and perspective effects. Worse yet, discrete grid representations of density maps result in information loss with blurred or vanished details for low-resolution inputs.
To overcome these limitations, we design Scale-Invariant Implicit neural representations for counting (SI-INR) to map arbitrary-scale input signals into a continuous function space, where each function produces density values over continuous spatial coordinates. SI-INR achieves robust counting performances with respect to changing object sizes, extensive experiments on commonly used diverse datasets have validated the proposed method.

Decision: reject
Reviewer scores: 5, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730218605214

thread_type:
official_review

<thread_object>
creator: reviewer_fhtu

<content>
summary:
This paper introduces a framework, Scale-Invariant Implicit Neural Representations (SI-INR), for object counting across varying image scales and resolutions. SI-INR addresses limitations in current methods, particularly the challenge of scale invariance in object counting tasks involving dense object scenes. The approach combines a scale-equivariant backbone with implicit neural representations, achieving high accuracy across benchmarks like the RSOC and CARPK datasets.

strengths:
- The framework’s focus on handling scale invariance through a continuous mapping function and modular structure is particularly valuable for applications involving heterogeneous datasets or remote sensing imagery with objects of varying sizes.

- Experiments across multiple datasets, such as RSOC and CARPK, showcase the generalizability of SI-INR. The paper also provides sufficient details on network configurations, making the work reproducible.

weaknesses:
- The paper does not fully explain the operational details of the "scale-invariant continuous mapping," especially regarding how SI-INR preserves fine details for low-resolution inputs. A more comprehensive description would enhance reproducibility.

- While computational demands are briefly mentioned, there is no clear comparison of SI-INR's runtime performance against baselines in various resolutions. This omission makes it difficult to assess scalability for real-time applications.

- The paper could benefit from additional experiments testing SI-INR’s robustness with images at extreme resolutions, especially low resolutions (e.g., <200 pixels), to provide a more complete understanding of its limitations.

- Missing some important previous work:

[1] Learning spatial awareness to improve crowd counting. (2019). ICCV 2019

[2] Rethinking spatial invariance of convolutional networks for object counting. (2022).CVPR 2022

questions:
- Could you provide further clarification on the continuous mapping mechanism in SI-INR? Specifically, how does it handle low-resolution inputs without sacrificing accuracy?

- Would you consider evaluating SI-INR’s performance in sequential data or video-based counting tasks?

- Have you identified any potential biases in SI-INR when applied to diverse environmental conditions, such as different lighting or weather effects in remote sensing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730340436541

thread_type:
official_review

<thread_object>
creator: reviewer_dyoe

<content>
summary:
This paper introduces a novel framework, Scale-Invariant Implicit Neural Representation (SI-INR), aimed at addressing significant challenges in object counting under varying scales and image resolutions. Traditional CNN-based counting methods often suffer from performance degradation when encountering objects at unseen scales or perspectives due to their reliance on discrete grid representations and non-scale-invariant models. The proposed SI-INR framework leverages a continuous function space, transforming discrete image signals into scale-invariant, continuous representations to improve accuracy and generalizability in object counting tasks.

strengths:
1. The formulas are sufficient, demonstrating a solid mathematical foundation.  
2. The method is reasonable.  
3. The experiments are comprehensive, proving the effectiveness of the method through empirical validation.

weaknesses:
1. After reading the paper, my understanding is that a VAE-like approach is used to obtain a fixed-size output, ensuring that inputs of different scales can be mapped to the same output, thus addressing the issues mentioned in the paper. However, the paper is written in a very complex manner and requires careful reading to understand. I suggest redrawing Figure 1. First, the new Figure 1 should allow readers to immediately grasp your network, especially the input and output. Additionally, it should include both the overall framework and detailed components of the network. The current Figure 1 does not provide a clear understanding of your work, making it difficult to reproduce.

2. Although your method is reasonable, it does not achieve the result mentioned in line 139 of the paper, i.e., obtaining the same output from inputs of different scales. In previous methods, multi-scale image augmentation was commonly used during training to address this problem. Although your approach is different, the goal remains the same.

3. In line 150, it is mentioned that GT is continuous. Why is GT continuous, whereas the output of previous methods is discrete? And why is your method’s output continuous?

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730603619674

thread_type:
official_review

<thread_object>
creator: reviewer_maum

<content>
summary:
This paper proposes a scale-invariant method that maps discrete grid image signals into a continuous 2D function space. This approach allows the model to represent an image as a continuous function rather than fixed discrete pixels. The model learns through supervised learning to capture and represent scale-invariant features.

strengths:
1. The proposed mehtod achieves better performance than several baseline on some simple datasets.

weaknesses:
1. The comparison methods are very limited; many state-of-the-art methods are not included, such as those using optimal transport and point-to-point matching.
2. As shown in Figure 2, the objects are of similar size, making it difficult to justify that the method effectively addresses the multi-scale challenge.
3. The proposed method should be evaluated on a typical dataset with dense crowds, which are known to have multi-scale individuals.
4. The description of the scale-invariant model is unclear.

questions:
1. Why can it be used to extract scale-invariant features? 
2. Since the multi-scale challenge has been investigated for a long time, how does the performance compare to other approaches?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657585917

thread_type:
official_review

<thread_object>
creator: reviewer_jvjv

<content>
summary:
The paper use implicit functions for resolution-agnostic scene representation.They first exact deep-features of the input images and map them into a "scale-invariant" latent space and finally, decode them back to density map for counting. The method is tested for several REMOTE SENSING datasets where it achieves competitive results.

strengths:
Counting objects of varying scales is challenging.
Using implicit neural representation for scale variations is reasonable.

weaknesses:
The presentation could be significantly improved to enhance clarity. Currently, the methodology section is challenging to follow. For example, the paragraph from lines 137 to 142 is particularly dense:  "Here h denotes an element of the Scale-Translation group H and represents one scale-translation operator, p1(·) denotes the corresponding group actions of h acting on the image domain."  - what does it means by "Scale-Translation group H" or "scale-translation operator"? In the following paragraph, the authors bring up the "continuous function space" without any explanation and with many notations are not clearly defined including I_a or D^gt (unclear where does this continuous ground-truth coming from?).  Overall, the writing creates several logical gaps that make it difficult to fully grasp the method. To the best of my understanding, the method is basically: 1) extract deep-features using a scale-equivariant backbone B, 2) using an encoder to map output of B into a latent space and 3) decode this latent representation into a fixed scale for counting. Could the authors confirm whether this understanding is accurate?

The problem statement also seems somewhat misleading. The authors should clarify that the paper addresses scenarios where the input image scale is unknown, rather than scale variations within a single image. However, the method has only been tested on remote sensing datasets, where this specific problem may not be prominent; in many cases, images are often at a uniform scale, or metadata about the sensor is available. Thus, it is not immediately clear in which practical scenarios the proposed method would be applicable. 

The evaluation is also questionable. It is unclear to me why it is not tested on crowd-counting dataset and also few-shot counting datasets (FSC-147) where the scale-invariant issue is particularly relevant. If the method is designed only for remote sensing data then the title should reflect that. The method doesn't actually achieve state-of-the-art performance (in both RSOC and CARPK) since the SOTA method [1][2] are not included in the table. Can the authors provide explanation for this?
[1] A Lightweight Multi-scale Feature Fusion Network for Remote Sensing Object Counting
[2] Few-shot Object Counting with Similarity-Aware Feature Enhancement

questions:
Intuitively, can the authors comment on why using implicit function is able to extract robust scale-invariant representations? Using implicit function is time-consuming and the speed test should be included.

Why not test the method on crowd-counting and few-shot counting datasets?

Why are SOTA methods not included in the evaluation?

For remote sensing data, can we simply use object detection or some simple baseline to infer the object scale?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732061217907

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal for All Reviewers

comment:
We thank all four reviewers **Jvjv, mauM, dyoE, fHTu** for their encouraging comments and constructive feedback. We here provide our general responses to all the reviewers for some of the raised common points. 

**1) Why SI-INR leads to continuous function representation:** 
1. Implicit Neural Representations (INRs) model a continuous function $ u: \mathbb{R}^d \to \mathbb{R} $, parameterized by $\theta_{\text{INR}} $, where $ u(x; \theta_{\text{INR}}) $ takes spatial coordinates $ x \in \mathbb{R}^d $ as input. Unlike grid-based representations, INRs are inherently resolution-agnostic, as they predict the signal value $ u(x) $ at any arbitrary $ x $ within the domain. This property enables continuous feature generation, as the model can be queried at finer resolutions to produce high-quality outputs regardless of the input resolution.

2. In SI-INR, the function extends to $u(x; z, \theta_{\text{INR}}) $, where $z \in \mathbb{R}^m $ represents the latent features extracted by the encoder. Thus, SI-INR learns a conditional continuous representation of the input by optimizing over $ \theta_{\text{INR}} $ and $ z $. This allows for task-specific predictions such as continuous density estimation.

3. For example, in density estimation tasks, the ground-truth density map $\rho(x) $ is often defined as a continuous function, commonly represented as a mixture of Gaussians: $ \rho(x) = \sum_{i=1}^N \mathcal{N}(x; \mu_i, \Sigma_i), $where $ \mu_i $ and $\Sigma_i $ denote the mean and covariance of the $i $-th Gaussian. Traditional methods discretize $\rho(x) $ onto a fixed grid, leading to potential information loss. In contrast, SI-INR directly models $ \rho(x) $ as a continuous function and evaluates predictions at arbitrary points $ x_n $, sampled from the domain.

**2) Inference speed comparison:**  
We sincerely appreciate the reviewers' suggestion to include a speed test. To clarify, all experiments in our study were conducted on a workstation equipped with an NVIDIA V100 32GB GPU. For the RSOC small-vehicle dataset, we observed the following inference times: ASPD-Net requires approximately 15.13 seconds, PSGC-Net takes around 2.47 seconds, eFreeNet takes around 3.84 seconds, and our SI-INR model requires about 3.87 seconds. 

| Model        | Inference Time (seconds) |
|--------------|---------------------------|
| eFreeNet     | 3.84                      |
| ASPD-Net     | 15.13                     |
| PSGC-Net     | 2.47                      |
| SI-INR (Ours)| 3.87                      |

SI-INR does take longer during the inference phase compared to PSGC-Net due to the integration of scale-equivariant models and the use of stacks of linear layers in the INR. However, thanks to the design of our INR decoder, which consists of only 4 linear layers, and our lightweight scale-equivariant backbone, the inference cost remains manageable and acceptable for practical use. 

Implicit neural representation (INR) models are known to be computationally intensive, particularly for high-frequency signals. Models such as NeRF [1], SIREN [2], and Fourier Feature Networks [3] excel in rapidly converging on coarse structures, like the overall geometry of objects, but require significantly more iterations to resolve fine details, such as intricate textures. In contrast, SI-INR is optimized for detection tasks, which primarily involve lower-frequency signals, leading to reduced computational demands. Additionally, detection tasks generally require less resolution fidelity than rendering tasks, further improving efficiency. These details and comparisons will be included in the revised manuscript. We will include these details in the revised manuscript to provide a clearer comparison.

[1] "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[2] "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473.

[3]"Fourier features let networks learn high frequency functions in low dimensional domains." Advances in neural information processing systems 33 (2020): 7537-7547.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732088542886

thread_type:
public_comment

<thread_object>
creator: ~wei_lin2

<content>
title:
Uniform sampled $\mathbf{x}$

comment:
Continuous representation of the crowd is a reasonable and nice proposal, but I have a problem in understanding the difference between it and the traditional grid-based method. In line-308, the authors described that "$\mathbf{x}$ is uniformly sampled from a pre-defined grid.'' This part is not clear so I have the following questions:

1. How many grids are sampled during training?
2. Is the number of sampled grids also random?
3. During inference, how is the count estimated? If a density map is output, how many grids are required? Does the number of grids have an impact on the performance?

It would be very helpful to understand this paper better if the authors could take some time to address my questions. 
However, if they are too busy, it is fine to leave my comments unaddressed.

Thanks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732671492351

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of updates in the revised manuscript

comment:
Dear Reviewers,

Thank you for your valuable feedback and thoughtful suggestions, which have been instrumental in improving our work. We have carefully revised the manuscript to address the concerns raised, and the key updates are summarized below:

**Major Changes**
1. Narrow the research scope from general object counting to remote sensing object counting, revise the related title, abstract, introduction, and presentations in the paper. [reviewer Jvjv & reviewer MauM]
2. New benchmark results on the CARPK dataset in Tables 2 and Sec 4.2. [Reviewer Jvjv]
3. New experimental analysis on the UCF-QNRF crowd-counting dataset in Appendix B5. [Reviewer Jvjv & reviewer MauM & reviewer fHTu]
4. New discussions on handling different resolution inputs in Sec 4.2 's 'Generalization Results'. [Reviewer Jvjv & reviewer fHTu]
5. New discussions on inference speed comparison in Sec 4.2 's 'Inference Efficiency'. [Reviewer Jvjv & reviewer fHTu]
6. New visual analysis on inter-image scale variation in Appendix B1 and Figure 4. [Reviewer MauM]
7. New discussion on the computational infeasibility of SESN to achieve a truly scale-invariance model in Section 4.2's 'Generalization Results'.

8. New discussion on the comparison with different methods for handling multi-scale challenges in Section 4.2's 'Generalization Results' and Appendix B6. [Reviewer Jvjv & reviewer MauM & reviewer dyoE & reviewer fHTu]
9. New discussion and visualization of generating arbitrary resolution outputs in Section 4.3 'Effect of sampling rate' and Appendix B7. [Reviewer Jvjv & reviewer MauM & reviewer dyoE & reviewer fHTu]

**Other Changes**
1. Provide a more detailed introduction about scale-equivariance/invariance theory in Appendix A1. [Reviewer Jvjv]
2. Enhanced readability for the reason why implicit neural representation achieves continuous function in Section 3.2.1.  [Reviewer Jvjv & reviewer MauM & reviewer dyoE & reviewer fHTu]
3. Correct the presentation of experiment results in Section 4.2 and Section 4.3. [Reviewer Jvjv]


To make it easier to identify the changes, all revised parts are highlighted in blue in the manuscript.

With these updates, we feel the depth and quality of our paper have been meaningfully improved. We hope that these revisions, along with our responses in the review threads, adequately address the concerns raised by the Reviewers.


We sincerely appreciate your time and thoughtful feedback.


Best Regards,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733073587236

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers, 

As the author-reviewer discussion period is approaching the deadline soon, we kindly request you to review our responses to your comments, concerns and suggestions. If you have further questions or comments, we will do our best to address them before the discussion period ends. If our responses have resolved your concerns, we would greatly appreciate it if you could update your evaluation of our work accordingly.
Thank you once again for your valuable time and thoughtful feedback.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a scale-invariant implicit neural representation for object counting using a continuous function space. The proposal aims to achieve robust counting with respect to variable object sizes. The method extracts features from images and maps them into a scale-invariant latent space, which is later decoded into a density map for counting. Experiments are conducted to validate the proposal.

Strengths:
- The use of implicit neural representations is a reasonable approach for scale-invariant counting.
- The presented results are better than some baselines for simple datasets.

Weaknesses:
- The method has been tested on remote sensing datasets where the image scale variation problem may not be prominent.
- There are missing comparisons on crowd-counting and few-shot counting datasets, and the baselines in the experiments are simple and do not include other recent methods.
- The experiments do not fully support the paper's claims.
- The method produces the same output from inputs of different scales.
- The presentation could be significantly improved. In particular, implementation details are missing, which limits the reproducibility of the paper.

The strengths seem to be limited to introducing a new approach for counting. However, the weaknesses, particularly the lack of experimentation in traditional crowd-counting datasets, are more extensive. The major concerns are that the experiments do not support the claims due to missing comparisons in crowd-counting datasets and with other methods. Thus, I recommend rejecting the paper.