Prototype-Guided Dual-Transformer Reasoning for Video Individual Counting

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:40:49

Abstract: Video Individual Counting (VIC), which focuses on accurately tallying the total number of individuals in a video without duplication, is crucial for urban public space management and densely-populated areas planning. Existing methods suffer from limitations in terms of expensive manual annotation, and the efficiency of location or detection algorithms. In this work, we contribute a novel Prototype-guided Dual-Transformer Reasoning framework, termed PDTR, which takes both similarity and difference of adjacent frames into account to achieve accurate counting in an end-to-end regression manner. Specifically, we first design a multi-receptive field feature fusion module to acquire initial comprehensive representations. Subsequently, the dynamic prototype generation module memorizes consistent representations of similar information to generate prototypes. Additionally, to further dig out the shared and private features from different frames, a prototype cross-guided decoder and a privacy-decoupling module are designed. Extensive experiments conducted on two existing VIC datasets, consistently demonstrate the superiority of PDTR over state-of-the-art baselines.

Main paper: 
 introduction: 
 With the ongoing increase in urban population density, effective crowd management and safety control in densely populated public areas are of utmost importance. Consequently, Video Crowd Counting (VCC) is developed to estimate the number of individuals in each frame, which has been widely used in various applications [7, 9, 40] . However, despite the fact that VCC has achieved amazing improvement, it fails to accurately count the total number of individuals appearing in a video, which follows a constraint that each pedestrian is counted only once. Therefore, a more challenging task termed Video Individual Counting (VIC) is proposed. Obviously, removing duplication counting and achieving accurate total enumeration from a video play as a critical issue in VIC. Over the decades, VCC [1, 9, 42, 46] has witnessed significant advancements. Current methodologies generally fall into two categories: single frame-based counting and cross-line counting, neither of which is directly applicable to the VIC task. The single framebased counting methods [11, 20, 29, 37, 52] inevitably lead to repetitive counting of the same target in adjacent frames, resulting in an inaccurate representation of the total count of people in a video, as shown in Fig. 1(a) . The cross-line methods [4, 28, 47, 48] solely focus on counting the pedestrians crossing predetermined lines, disregarding pedestrians in stationary or intermittent motion states, as shown in Fig. 1(b) . Additionally, the manual marking operation required by cross-line methods necessitates labor-intensive effort, and lead to prohibitive costs when dealing with highly dense conditions. Inspired by multi-object tracking (MOT) methods, VIC was initially defined by Han et al. [12] who proposed a new counting paradigm DRNet to tally the individuals in the initial video frame and increment this count with new entrants in subsequent frames. DRNet extracts head descriptors from density maps of adjacent frames and uses feature matching inference to infer inflow (the number of people entering the current frame) and outflow (the number of people who left the previous frame). Recently, Liu et al. [26] also proposed a weakly supervised method based on contrastive learning with group-level matching, to further reduce the reliance of MOT-based methods on trajectory labels. Unfortunately, these MOT-based methods (shown in Fig. 1(c )) heavily depend on the accuracy of density maps and descriptors.In addition, the aforementioned methods all rely on convolutional neural network architecture and involve additional preprocessing operations. To reduce the reliance on these operations, Liang et al. [22] takes the lead in applying DETR [3] to crowd localization, thereby achieving an end-to-end regression localization. Despite the fact that this straightforward and efficient method is primarily suited for single image and cannot be directly applied to VIC, it prompts us to pose the hypothesis: can video individual counting also be addressed using a uncomplicated regression approach? Generally, the VIC task has higher difficulty and meets two key challenges: i) if the model can generate semantic and consistent representations of individuals across consecutive frames, it can effectively address the issue of target repetition in temporal individual counting; and ii) Given the continual motion state of target in a video, it becomes even more critical to thoroughly address the fusion of information pertaining to the same target across different frames.Inspired by aforementioned analysis, we propose the Prototypeguided Dual-Transformer Reasoning (PDTR) framework, fully leveraging the advantages of prototype learning and dual-stream inference, as shown in Fig. 1(d) . In particular, our PDTR is composed of four main components: i) multi-receptive field feature fusion module (MRF 3 ), ii) dynamic prototype generation (DPG), iii) prototype cross-guided decoder (PCD), and iv) privacy-decoupling module (PDM). Obviously, the sizes of targets in a frame are rich diverse for the sake of distances between targets and the camera. We propose the MRF3 to acquire and merge multi-scale representations. Subsequently, these representations are input into DPG to extract probability distribution of similar information across different frames, thereby directing the model's attention towards similarity and aiding in prototypes generation. Afterwards, a prototypes cross-guided mechanism is designed to efficiently train the transformer decoder to extract shared features from comprehensive representations of different frames, thereby inferring targets that present in both frames. Lastly, PDM is utilized to separate private features from each frame for complementary accounting.The core contributions of this work are as follows:• We propose a novel prototype-guided dual-transformer reasoning framework for VIC, which converts the feature matching process of conventional models to an end-to-end regression reasoning procedure. To the best of our knowledge, this represents the initial endeavor to employ Transformer in a dual-stream cross-guidance manner for VIC. • A novel dynamic prototype generation module is deployed to bridge and mine consistency information from comprehensive representations of adjacent frames, assisting decoder in cross-generating semantic consistency features, thereby reasonably utilizing the motion information of targets between frames to reduce duplicate counting.  
related work 2.1 video individual counting: 
 Video individual counting (VIC), as it involves counting each person in a video only once, presents a higher level of complexity than frame-by-frame video crowd counting (VCC). One possible solution is the multi-object tracking methods [2, 23, 27, 30, 33, 34, 38] , which detect and track multiple objects in consecutive frames of a given video, thereby achieving pedestrians counting. This scheme has made some research progress in VCC. Ren et al. [30] modeled the detection, counting, and tracking problems as a network flow problem. Sundararaman et al. [34] developed two frameworks for head detection and tracking, based on motion models and a color histogram-based re-identification module. However, multi-object tracking methods need to capture continuous trajectories of objects and consider frequent ID switching, they cannot effectively be applied to VIC. Motivated by these methods, Han et al. [12] innovatively converted VIC into a feature matching task by utilizing the descriptors of each located head in adjacent frames to match and determine pedestrian inflow. Liu et al. [26] proposed a novel baseline equipped with a newly designed group level matching soft contrastive loss. To our knowledge, these are the only two studies that completely focus on VIC. Unlike these methods, our approach reformulates the mainstream object matching process into a more reliable, prototype-guided similarity and difference reasoning procedure.Prototype refers to the feature representations of instances within the same class [31] . Thanks to its exemplar-driven nature and simpler inductive bias [14] , it holds significant potential across various tasks. In image classification, each prototype acted as a representation of a specific category, enhancing model performance through the regulation of inter-prototype distances [31, 43] . In semantic segmentation, prototype vectors represented the masked object features of support images and were utilized to search for pixel positions with similar features in the query image, thereby aiding in accurate target segmentation. [18, 39, 50] . In crowd counting, Huang et al. [15] established a weather bank for gathering various weather prototypes, and devised prototype loss to improve the adaptability under various weather conditions. Our study adopts the concept of prototypes to signify similarity and implements prototype aggregation within dual-transformer framework.Transformer, initially proposed for modeling sequential data in machine translation [35] , has recently demonstrated remarkable success in various tasks, including object detection [3, 5, 44] , image recognition [8, 32] , and semantic segmentation [49] . This success has led to increased interest in exploring transformer-based architectures for crowd counting [21, 22, 24] . For instance, Liang et al. [22] devised a novel approach by treating crowd localization as a regression task and developed an end-to-end transformer framework. Lin et al. [24] enhanced transformer by introducing learnable region attention and an instance attention mechanism, which effectively integrated global and local features to address scale differences. Moreover, researchers have explored the potential of transformers in VCC. Fang et al. [9] thoroughly investigated the complementarity between density maps of consecutive frames, and addressed pedestrian occlusion in videos. Inspired by those methods, we first attempt to apply transformer to VIC and design a dual-transformer framework to handle temporal information in videos. 
method 3.1 problem formulation: 
 Suppose that we have a video 𝑰 = {𝐼 𝑖 } 𝑇 -1 𝑖=0 containing 𝑇 frames. Given the similarity between adjacent frames, it is possible that the flow of people has remained constant due to minor variations in information. Thus, we sample every 𝜀 frames to create frame pairs {𝐼 𝑧 , 𝐼 𝑧+𝜀 }. For each frame in {𝐼 𝑧 , 𝐼 𝑧+𝜀 }, taking 𝐼 𝑧 as an example, its corresponding label is𝐿 𝑧 = 𝑐 𝑧 𝑛 , 𝑠 𝑧 𝑛 𝑁𝑧 𝑛=1. Specifically, 𝑐 𝑧 𝑛 = (𝑥 𝑧 𝑛 , 𝑦 𝑧 𝑛 ) represents the coordinates of all 𝑁 𝑖 pedestrians in frame 𝐼 𝑧 , while 𝑠 𝑧 𝑛 ∈ {0, 1} denotes corresponding state of pedestrian 𝑛. Here, 0 indicates the pedestrian appears in both frames, and 1 denotes that it only exists in frame 𝐼 𝑧 . The objective of this paper is to accomplish video individual counting (VIC), which involves accurately determining the total number of individuals in video 𝑰 while ensuring no duplication in the count.Building upon the works of [12, 26] , VIC is broken down into two sub-problems: counting the total number of people 𝑀 (𝐼 0 ) in the initial frame, and inferring the number of new individuals 𝑀 𝑖𝑛 (𝐼 𝑧 , 𝐼 𝑧+𝜀 ) who appear in subsequent frames. The total number of people in video 𝐼 can be calculated:𝑇𝑜𝑡𝑎𝑙 ≈ 𝑀 (𝐼 0 ) + 𝑀 𝑖𝑛 (𝐼 𝑧 , 𝐼 𝑧+𝜀 ) = 𝑐𝑜𝑢𝑛𝑡 (𝑐 0 ) + 𝑜=(𝑇 -𝜀 )/𝜀 ∑︁ 𝑜=0 𝑐𝑜𝑢𝑛𝑡 ( 𝑐 𝑧+𝜀 |𝑠 𝑧+𝜀 = 1, 𝑧 = 𝑜𝜀 ),where 𝑐𝑜𝑢𝑛𝑡 (•) is the count operation and 𝜀 denotes the sample interval.Based on the analysis in Sec. 3.1, we have designed an end-to-end regression network, named PDTR, which can directly predict the point coordinates of pedestrians belonging to different states. For ease of understanding, we will use superscripts ( * ) in the following description to uniformly represent different frames. Specifically, as shown in Fig. 2 , PDTR is composed of four components: (a) MRF 3 , which includes a feature extractor (backbone) F 𝜃 1 and a fusion encoder F 𝜃 2 . Extractor F 𝜃 1 extracts multi-scale feature from input frames 𝐼 1 and 𝐼 2 ∈ R 𝐻 ×𝑊 ×3 respectively. Encoder F 𝜃 2 captures long-distance global information and integrates them into extracted features to yield multi-receptive field fusion representations 𝐹 ( * ) ∈ R ℎ×𝑤×𝑐 . (b) DPG, which learns and stores shared prototypical representations X ( * ) from different frames. In this paper, 𝑘 prototypes 𝑋 = {𝑥 1 , ..., 𝑥 𝑘 } ∈ R 𝑘 ×𝑐 are learned from features 𝐹 ( * ) via EM mechanism, to excavate consistent representations of the same target across neighboring frames. We expect that these learned prototypes can assist the model in tackling challenges arising from scene 𝑐 ∈ R ℎ×𝑤×𝑐 in a cross guidance manner, i.e., U 𝜗 : ((𝐹 1 , X 2 ), (𝐹 2 , X 1 )) → 𝑍 ( * )𝑐 . (d) PDM, which decouples discriminative and private features contained in different frames from comprehensive representations: V 𝜓 : (𝐹 ( * ) , 𝑍 ( * ) 𝑐 ) → 𝑍 ( * ) 𝑠 . Finally, a decoder D 𝜙 is adopted to map the learned features to final point coordinates and confidence scores A, i.e., D 𝜙 : (𝑍 ( * ) 𝑐 , 𝑍 ( * ) 𝑠 ) → A. It is noteworthy that our model, configured as a dual-stream transformer for neighboring frames, shares weights across components since each branch executes identical functions.Existing studies [21, 22] have illustrated a direct approach to capture fine-grained image features for overcoming the limitation of relying solely on local information. They employ neural networks to extract detailed information, and then integrate them into the transformer encoder to model pixel-wise spatial dependencies. However, these approaches overlook the irrationality of feature extraction at the same scale, as objects of various sizes and distances have different scales. To this end, we incorporate multi-scale information into transformer reasoning to comprehensively capture features of targets with different sizes, as depicted in Fig. 3(a) . By involving multi-receptive field information, our model effectively identifies target regions targets with various sizes.Specifically, we design MRF 3 to fuse multi-scale and global context information, comprising two main components: feature extractor F 𝜃 1 and fusion encoder F 𝜃 2 . The former enhances the receptive field by progressively deepening network layers, offering diverse multi-scale information, whereas the latter emphasizes global context and integrates varied spatial details. Initially, extractor F 𝜃 1 takes a pair of adjacent frames 𝐼 ( * ) ∈ R 𝐻 ×𝑊 ×3 as inputs, generating 𝑐-dimensional multi-scale feature sets F1 , F2 , F3 , F4( * ) with resolutions of 1/32, 1/16, 1/8 and 1/4 respectively. Subsequently, MRF 3 employs a feature alignment pyramid to align embeddings of various scales F ( * ) 𝑖 (excluding F ( * ) 4 ) to F ( * ) 𝑖 of the same resolution. Different layers of feature alignment pyramid incorporate varying numbers of feature alignment (FA) blocks, according to the feature size. The specific configuration is outlined below:EQUATIONwhere the FA block comprises an average pooling layer and a convolution layer, and 𝜃 𝐹𝐴 𝑖 represents trainable parameters of the 𝑖-th FA block. Finally, the encoder F 𝜃 2 adopts 𝐻 transformer blocks to comprehensively integrate feature representations of varying scales:EQUATIONwhere 𝑄 ℎ , 𝐾 ℎ and 𝑉 ℎ denote learnable parameters for the ℎ-th head, and 𝑄 ℎ , 𝐾 ℎ and 𝑉 ℎ denote 𝑞𝑢𝑒𝑟𝑦, 𝑘𝑒𝑦 and 𝑣𝑎𝑙𝑢𝑒, respectively. 𝐻 = 8 heads are used in our implementation. It is important to highlight that multi-scale features are integrated solely into the initial three encoder layers, and subsequent blocks maintain selfattention mechanism. By employing cross attention mechanism, we obtain fusion features 𝐹 ( * ) that amalgamate rich spatial information from multiple receptive fields.Effectively utilizing similar information between adjacent frames to solve duplicate counting is a key insight in the VIC tasks. To delve deeper into the consistency information across frames in varied scenes, it is crucial to thoroughly consider the high-level semantic details within the scene. However, for the task of counting individuals in videos, traditional techniques like global average pooling are unreliable due to object-to-image issues and the dynamic nature of targets' motion [51] . Consequently, we employ metric learning to craft a dynamic prototype generation module (DPG), denoted as P 𝛿 , to capture discriminative prototypical representations X ( * ) , as shown in Fig. 3(b) . Motivated by the expectation-maximization (EM) strategy [6] , DPG applies iterative clustering to dynamically group similar features and address motion-induced uncertainty.The feature inputs from different frames can then be expressed as the weighted sum of respective stored prototypes. This approach ensures the generation of semantic consistency representations for the same targets across neighboring frames.To begin, reshape the given feature embedding 𝐹 ( * ) ∈ R ℎ×𝑤×𝑐 into a set of local features by collapsing the spatial dimension of 𝑖 ∈ R 𝑐 . Next, establish a collection of 𝑘 learnable visual atoms stored in an external memory as 𝑀 = 𝑚 𝑗 𝑘 1 . To continually enhance memory performance, we initially sample from a normal distribution with a mean of 0 and a standard deviation of √︁ 2/𝑘 to initialize the prototype embedding, i.e., 𝑚 𝑗 ∼ N (0, √︁ 2/𝑘), and calculate the correlation map 𝐶 by:EQUATIONwhere 𝑓 ( * )∈ 𝐹 ( * ) indicates the 𝑖-th feature, 𝑚 𝑗 ∈ 𝑀 denotes the 𝑗-th memory item, and 𝜌 represents an adjusting parameter. In order to better fit the sample distribution in the input feature space, we update the memory 𝑀 based on the correlation map between samples and memory items calculated above:EQUATIONWe then normalize the updated vector x 𝑗 to maintain unit length, which ensures placement in a standardized space, and contributes to model convergence and generalization:EQUATIONwhere 𝜖 = 1𝑒 -6, serves as a coefficient to prevent the denominator from being 0. Followed by multiple iterations of the operations described above, the initial cluster centroids 𝑋 = {𝑥 1 , ..., 𝑥 𝑘 } ∈ R 𝑘 ×𝑐 that have adjusted to the input feature distribution are acquired. To enhance the discriminative ability of these centroids, we introduce prototyping multi-layer perceptron (ProMLP) blocks to enhance representation X ( * ) for individual frames:EQUATIONwhere ProMLP consists of a stack of N = 2 identical MLP with residual connections, and 𝜃 𝑚𝑙𝑝 represents the trainable parameters. Note that ProMLP is the only part of PDTR that does not share weights.Ultimately, DPG generates the prototypical representations X ( * ) , capturing the similar semantics across various frames. In accordance with [44] , an unsupervised prototyping loss is introduced to enforce a significant separation between the acquired prototypes:EQUATION2 ), 0), (7) where 𝑤 is a pre-set hyperparameter.The prototype representations acquired through DPG contain rich inter-frame consistency information, which help model infer pedestrians that exist in both frames. To investigate shared information across adjacent frames utilizing the extracted features 𝐹 ( * ) and prototype X ( * ) , we develop a prototype cross-guided decoder (PCD) denoted as U 𝜗 . The decoder fully combines all available information and produces unified representations, i.e., U 𝜗 : ((𝐹 1 , X 2 ), (𝐹 2 , X 1 )) → (𝑍 1 𝑐 , 𝑍 2 𝑐 ). Leveraging the effectiveness of transformers in modeling long-distance dependencies, PCD employs a standard transformer architecture to capture inter-frame similarity and global information, as shown in Fig. 3(b) . In contrast to prior methods [9, 22] that focused on learning close relationships within patches of single frame, we implement a cross-guidance strategy using prototypes from different frames to emphasize common regions. For frame 𝐼 1 in PCD, multi-head cross-attention mechanism computes the 𝑞𝑢𝑒𝑟𝑦 using the prototype X 2 from the current frame 𝐼 2 and derives the 𝑘𝑒𝑦 and 𝑣𝑎𝑙𝑢𝑒 from the multi-scale feature 𝐹 1 :EQUATIONwhere Qℎ , Kℎ and Ṽℎ represent the learnable parameters of project layers corresponding to the ℎ-th head. Additionally, we also make the same cross-attention operation on the current frame 𝐼 2 utilizing information from previous frame 𝐼 1 . In a word, the semantic information from various frames can be amalgamated into the shared features 𝑍 1 𝑐 and 𝑍 2 𝑐 through a prototypes cross-guided crossattention operation. This strategy enables model to focus more effectively on similar regions and maintain semantic consistency.Following the inference of shared pedestrians between two frames, it is necessary to additionally separate private features from comprehensive representation to infer pedestrians that only exist in their respective frames. To distinguish their distinct private domain information, we developed a privacy-decoupling module (PDM) V 𝜓 for decoupling comprehensive feature 𝐹 ( * ) and shared features𝑍 ( * )𝑐 , sufficiently examining the disparities between two frames, i.e., V 𝜓 : (𝐹 ( * ) , 𝑍 ( * ) 𝑐 ) → 𝑍 ( * ) 𝑠 . Specifically, we assess spatial positional variances through feature subtraction, represented as:EQUATIONwhere 𝑛𝑜𝑟𝑚(•) is LayerNorm and 𝑑𝑟𝑜𝑝 (•) is Dropout [41] . At this point, the model outputs shared features 𝑍 ( * ) 𝑐 and private features 𝑍 ( * ) 𝑠 from adjacent frames, which are used to predict the point coordinates (regression head) and confidence scores (classification head). A decoder D 𝜙 uses a simple MLP layer and a linear projection layer to achieve coordinate prediction and confidence scoring, respectively, i.e, D 𝜙 : (𝑍 ( * ) 𝑐 , 𝑍 ( * ) 𝑠 ) → A. To address the problem of ambiguous matching between adjacent targets in the video, we utilize KMO-based Hungarian bipartite matching [22] to guarantee the quality and rationality of the matching outcomes. The proposed PDTR is optimized using multiple loss functions, namely prototyping loss, regression loss, and classification loss. The first loss function has been defined in Section 3.4. The regression loss L 𝑟𝑒𝑔 aims to regulate the learning of coordinate positioning and we employ the commonly-used 𝑀𝐴𝐸 loss:L 𝑟𝑒𝑔 = A ( * ) 𝑐 - Â ( * ) 𝑐 1 + A ( * ) 𝑠 - Â ( * )𝑠 1 (10) where Â ( * ) 𝑐 and Â ( * ) 𝑠 represent the true coordinate sets of shared and private targets, respectively. In addition, A ( * )𝑐 and A ( * ) 𝑠 denote respective predicted subsets obtained through the KMO-based Hungarian bipartite matching.Moreover, we employ the focal loss [25] as the classification loss L 𝑐𝑙𝑠 to distinguish whether predicted point belongs to key target or "background". The overall loss function of PDTR is:EQUATIONwhere 𝛼 1 and 𝛼 2 are scaling factors.  
experiments 4.1 experimental settings: 
 Datasets. Two publicly available benchmark datasets, namely CroHD [34] and SenseCrowd [19] , are utilized for performance evaluation. Both datasets include annotations for point coordinates as well as inflow and outflow statistics. CroHD comprises 11,463 frames distributed over 9 full-HD resolution sequences, with 4 videos allocated for training and validation, and 5 videos designated for testing. SenseCrowd, a large-scale dataset with various scenes categorizations (e.g., density, time, and space), encompasses 634 sequences totaling 62,938 frames. The distribution of training, validation, and testing sets aligns with the experimental setups in [12] . Evaluation Metric. We uses Mean Absolute Error (MAE), Mean Square Error (MSE) and Weighted Relative Absolute Errors (WRAE) to evaluate performance. The first two metrics represent fundamental measures for crowd counting [46] . Additionally, in line with [12] , we utilize WRAE to precisely assess the variance between video length and pedestrian count:EQUATIONwhere 𝐾 signifies the total count of videos, 𝑇 𝑖 denotes the overall number of frames in 𝑖-th video, while 𝑁 𝑖 and N𝑖 represent the actual and predicted pedestrian quantities in 𝑖-th video. Implementation Details. During training, the underlying extractor F 𝜃 1 is initialized by ResNet-50 [13] pre-trained on ImageNet [17] , and the remaining modules (i.e., encoder F 𝜃 2 , DPG P 𝛿 , transformer decoder U 𝜗 and decoder D 𝜙 ) are randomly initialized. Encoder F 𝜃 2 and transformer decoder U 𝜗 both consist of 3 transformer layer with 8 heads. The learning rate is set as 1𝑒 -4 along with Adam [16] with weight decay 5𝑒 -4. The proposed network is trained on the Nvidia RTX 3090 GPU and implemented using the PyTorch framework. The scaling factor of losses 𝛼 1 and 𝛼 2 are both set to 0.5. The sampling interval for image pairs ranges from 2s to 8s, while it is set to 3s during testing.Comparison methods. Currently, there is limited research on VIC tasks, DRNet [12] and GCNet [26] are the only two for VIC. We categorize such approaches as feature matching-based methods, labeled as M. Furthermore, to assess the efficacy of our method, we also compared it with two distinct types of approaches: those based on multi-object tracking and those based on cross-line counting. The tracking-based method employs statistical analysis of object trajectories, including PHDTT [36] , FairMOT [45] , and HeadHunter-T [34] , denoted as T. The cross-line method, LOI [48] , involves counting the individuals crossing predefined lines, denoted as L.Results on CroHD. In Table 1 , we present a comparison of our PDTR with 6 existing methods using three commonly-used metrics.The results indicate that our PDTR outperforms all competitors by a significant margin. For example, MAE and MSE of PDTR are remarkably low at 60.6 and 73.7, representing approximately onefourth and one-fifth of those obtained by the tracking-based and cross-line methods, respectively. At the same time, it significantly surpasses feature matching-based methods, which proves the importance of reasoning over the similarity and difference between adjacent frames. For specific performance on five testing scenes, our method achieves the best prediction results, with the exception of CroHD11 and CroHD15. We achieve similar prediction results to the state-of-the-art on CroHD11 and CroHD15, mainly due to diversity of perspectives in the scene. As PDTR is trained solely on data from a fixed perspective, it exhibits limited adaptability to variations in shooting angles. Furthermore, PDTR demonstrates outstanding performance on several other high-density videos (e.g., CroHD12: 183.0, CroHD13: 259.6, and CroHD14: 245.9 person/frame), underscoring the effectiveness of our method in managing scenes with high density.Results on SenseCrowd. problems become more prevalent, which leads to insufficient availability of feature information. This hampers the fulfillment of detection and association operations as necessitated by alternative methods, ultimately resulting in suboptimal model performance.On the contrary, our method's performance advantage becomes increasingly evident, as it directly converts input into coordinate points by leveraging the similarity and difference between two frames. Fig. 4 displays qualitative results of three representative scenarios (i.e., square, mall and street), demonstrating our model's ability to achieve accurate inference across various environments. Nonetheless, some erroneous examples may arise, as observed in current frame of second scene, where predicted inflow exceeds true inflow. Upon comparing adjacent frames, it becomes evident that PDTR provides more plausible prediction results.Impact of FAP. The variety of perspectives across different scenarios result in pedestrian targets suffer from inconsistent scales. To assess the effectiveness of FAP developed in this study, we excluded this module from the original framework. Instead, we directly employed features obtained from extractor F 𝜃 1 as input for encoder F 𝜃 2 to generate comprehensive representations, as shown in Table 3 (i.e., w/o FAP). The counting performance exhibits a notable decline, particularly in CroHD13 and CroHD14 where scale issues are more prominent. Thus, the effectiveness and robustness of feature alignment pyramid in integrating multi-scale features and global information have been confirmed. Impact of L 𝑝𝑟𝑜 . The prototyping loss is designed to enhance the discriminative ability of prototypical representations obtained from the memory module by maximizing their inter-distance. We remove it from the overall model to illustrate its role in constraining prototype generation, as shown in Table 3 (i.e., w/o L 𝑝𝑟𝑜 ). Clearly, the model trained without prototyping loss function exhibits a notable performance decrease compared to the full-equipped model. Additionally, Table 4 demonstrates that setting 𝑘=256 visual atoms empowers our model to achieve outstanding performance. Impact of Cross-guide. Effectively fusing information from adjacent frames using cross-guidance mode is another crucial aspect of achieving rational reasoning. To further clarify the role of crossguidance mode in aiding generation of shared representations, we omit it and instead utilize outputs of encoder F 𝜃 2 to direct the decoding processes of each frame, as depicted in Table 3 (w/o Cross-Guide). Without cross-guidance mode, the model regresses into a single-frame counting model, lacking the ability to supplement information from adjacent frames and incapable of addressing interframe similarities and differences, consequently leading to a substantial decline in model performance.Increasing the number of transformer layer can enhance the model's representational capacity and more effectively capture the intricate relationships within input sequences [10, 35] . However, the excessive number of transformer layers may inflate computational costs and parameter numbers, increasing the difficulty of model optimization. We examine the impact of the encoder/decoder layer number on counting performance, as shown in Table 4 . It is evident that with increasing layers, network performance decreases. At L=3, the network's counting performance peaks and generally surpasses that of higher layer counts. 
conclusion: 
 In this paper, we presented a prototype-guided dual transformer reasoning (PDTR) framework to solve video individual counting (VIC) in an end-to-end regression manner. PDTR fully leveraged the advantages of prototype learning and dual-stream inference, innovatively using transformer to capture similarity and differences between adjacent frames for accurate counting. Initially, a multi-receptive field feature fusion module (MRF 3 ) learned comprehensive representations of various frames. Subsequently, a dynamic prototype generation module (DPG) extracted similar information from these representations to cross-guide decoder (PCD) for shared features. Finally, a privacy-decoupling module (PDM) was implemented to extract frame-specific private information from comprehensive features. Extensive experiments conducted on two public datasets showed promising performance compared to state-of-theart methods on VIC.  
 
 Appendix:  


Decision: This work is focused on individual counting in videos. The authors propose a prototype guided approach to solve this problem. This work initially received 3x borderline accepts and the reviewers were mainly concerned by the complex nature of the approach. The authors provided a rebuttal and most of the concerns were addressed in the response. The final ratings are weak accept and 2x borderline accept. The AC agrees with this assessment and even though the solution is little complex it is performing well and the evaluation is comprehensive. The AC recommends acceptance of this work.

Reviews: review: paper_summary: paper_summary: The paper introduces a Prototype-Guided Dual-Transformer Reasoning (PDTR) framework designed to count individuals in videos accurately without duplication. This method leverages a dual-transformer approach to capture similarities and differences between adjacent frames in an end-to-end regression manner. Key components include:

Multi-receptive Field Feature Fusion Module (MRF3): Acquires initial comprehensive representations.
Dynamic Prototype Generation (DPG): Generates consistent representations across frames.
Prototype Cross-Guided Decoder (PCD): Uses shared features for improved counting.
Privacy-Decoupling Module (PDM): Separates private features to enhance accuracy.
Extensive experiments on two VIC datasets demonstrate the PDTR's superior performance over existing methods.
strength_weakness: strength_weakness: S1: Innovative dual-transformer framework that effectively handles the similarities and differences between adjacent video frames for accurate individual counting.

S2: The dynamic prototype generation module enhances consistency in representations across frames, reducing duplicate counting.

S3: Extensive use of privacy-decoupling ensures that private features are well-separated, aiding in better individual identification.

S4: Comprehensive testing on multiple datasets shows robust performance across different scenarios.

S5: The multi-receptive field feature fusion module integrates multi-scale and global context information, improving feature representation.
limitations: limitations: W1: The framework is specialized for counting tasks and does not address other potential tasks like tracking or behavior analysis, limiting its broader applicability.

W2: The addition of multiple complex modules, such as DPG and PCD, increases computational requirements, potentially impacting real-time performance.

W3: While tested on various datasets, the method's performance in highly diverse and unforeseen environments is not thoroughly examined, raising questions about its generalizability.

Overall Review
The paper presents a novel and effective approach to video individual counting using a Prototype-Guided Dual-Transformer Reasoning framework. Its strengths lie in the innovative use of dual transformers and prototype generation to enhance counting accuracy while maintaining privacy. The method shows significant improvements over existing techniques, especially in handling frame-to-frame consistency and privacy concerns. However, the added complexity and computational demands could hinder real-time applications, and broader testing is needed to ensure its robustness in diverse settings. Overall, the PDTR framework is a promising advancement in the field of video individual counting.
paper_summary: paper_summary: The paper presents a novel framework, Prototype-Guided Dual-Transformer Reasoning (PDTR), for Video Individual Counting (VIC). The approach aims to accurately count the total number of individuals in a video without duplication. The PDTR framework leverages prototype learning and dual-stream inference to achieve this goal. It includes four main components: multi-receptive field feature fusion (MRF3), dynamic prototype generation (DPG), prototype cross-guided decoder (PCD), and privacy-decoupling module (PDM). Extensive experiments demonstrate that PDTR outperforms state-of-the-art baselines on existing VIC datasets.
strength_weakness: strength_weakness: - Innovative approach
- handling shared and private features can provide a better solution than current approaches
- e2e regression reasoning
- highly experimental and ablation studies
limitations: limitations: - to complex approach.
- paper lack conceptual innovation, but the components used looks more of an engineering effort
- If the paper mainly combines well-known techniques (e.g., transformers, prototype learning) in a new context (video individual counting), this combination might be viewed as evolutionary rather than revolutionary
- Yao, T., Li, Y., Pan, Y., Wang, Y., Zhang, X. P., & Mei, T. (2023). Dual vision transformer. IEEE transactions on pattern analysis and machine intelligence. This paper can be useful.
paper_summary: paper_summary: The paper introduces a novel framework named Prototype-Guided Dual-Transformer Reasoning (PDTR) aimed at addressing the challenge of accurately counting individuals in video footage. This task, known as Video Individual Counting (VIC), is essential for managing densely populated urban areas and public spaces.
strength_weakness: strength_weakness: 1. The authors propose a new framework that uses prototype learning and dual-stream inference to count individuals in a video. The framework is designed to handle the challenges of target repetition and motion state continuity in videos by converting the feature matching process into an end-to-end regression reasoning procedure.
2. The PDTR framework was evaluated on two public datasets, CroHD and SenseCrowd. The results consistently demonstrated the superiority of PDTR over state-of-the-art baselines, showcasing its effectiveness in challenging scenarios with occlusions, dynamic traffic, and complex layouts. Further, the authors conducted ablation studies to assess the contribution of each module within the PDTR framework.
limitations: limitations: 1. The PDTR might be more complex compared to simpler counting methods, which could impact its ease of implementation and computational efficiency. The use of transformers and multiple loss functions might increase the computational cost, which could be a limitation for users with limited computational resources.
2. The underlying extractor is initialized with a pretrained model on ImageNet, which may introduce dependencies on external resources and the quality of pretrained weights.
3. The paper could benefit from a more detailed comparative analysis with existing methods, particularly in terms of computational efficiency and scalability.
4. While the paper demonstrates the effectiveness of the framework in controlled settings, its effectiveness in real-world applications and its integration into practical systems could be further explored.