Unsupervised Multi-view Pedestrian Detection

Authors: mengyin  liu
chao  zhu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:16:46

Abstract: With the prosperity of the intelligent surveillance, multiple cameras have been applied to localize pedestrians more accurately. However, previous methods rely on laborious annotations of pedestrians in every frame and camera view. Therefore, we propose in this paper an Unsupervised Multi-view Pedestrian Detection approach (UMPD) to learn an annotation-free detector via vision-language models and 2D-3D cross-modal mapping: 1) Firstly, Semantic-aware Iterative Segmentation (SIS) is proposed to extract unsupervised representations of multi-view images, which are converted into 2D masks as pseudo labels, via our proposed iterative PCA and zero-shot semantic classes from vision-language models; 2) Secondly, we propose Geometry-aware Volume-based Detector (GVD) to end-to-end encode multi-view 2D images into a 3D volume to predict voxel-wise density and color via 2D-to-3D geometric projection, trained by 3D-to-2D rendering losses with SIS pseudo labels; 3) Thirdly, for better detection results, i.e., the 3D density projected on Birds-Eye-View, we propose Vertical-aware BEV Regularization (VBR) to constrain pedestrians to be vertical like the natural poses.  Extensive experiments on popular multi-view pedestrian detection benchmarks Wildtrack, Terrace, and MultiviewX, show that our proposed UMPD, as the first fully-unsupervised method to our best knowledge, performs competitively to the previous state-of-the-art supervised methods. Code is available at https://github.com/lmy98129/UMPD.

Main paper: 
 related works 2.1 multi-view pedestrian detection: 
 To detect the pedestrians in multi-view images, various detection methods have been proposed based on different architectures.Following the pedestrian detection for monocular image, RCNN & Clustering [36] firstly detects pedestrians by a supervised 2D detector in each view, and then fuses them via clustering. Similarly, POM-CNN [4] fuses 2D masks by supervised model as final results. Due to weaker cross-view consistency of the single-view 2D boxes or masks, their performances are worse than BEV-based methods.In an anchor-based style, DeepMCD [5] predicts the BEV positions by anchors. Deep-Occlusion [2] trains Conditional Random Field (CRF) on anchor features. In an anchor-free style, MVDet [12] firstly uses homography mapping from 2D features onto BEV to predict positions. Following MVDet, SHOT [31] makes multi-height projection. 3DROM [27] learns with a random occlusion augmentation. More augmentation is adopted by MVAug [7] . MVDeTr [11] predicts pedestrian directions via Transformer. With source domain labels, GMVD [33] is still inferior to in-domain methods. Recently, an unsupervised view fusion component is applied to supervised detection [38] . All these methods rely on laborious BEV labels.Differently, without any manual labels, our proposed volumebased detector GVD performs 2D-3D cross-modal mapping to predicting the 3D density of pedestrians, learned from our 2D SIS masks as pseudo labels with our VBR as an extra regularization.In the past decade, unsupervised learning of feature representations has achieved powerful performance. Specifically, popular contrastive learning discriminates paired samples, which is capable of more unsupervised zero-shot tasks than supervised fine-tuning.For single modality, DINOv1 [3] learns a self-distillation for only positive samples, which is utilized by CutLER [34] to segment all salient objects in a single 2D image rather than multi-view ones. Recently, DINOv2 [25] with better data quality and learning scheme yields more robust representations, which segments shared concepts across images, by dividing the 1 st PCA values of features.For multiple modalities, CLIP [28] learns on paired images and texts collected from the internet. Furthermore, MaskCLIP [40] predicts 2D masks queried by input texts, and CrowdCLIP [20] merely predicts people count. But these unsupervised methods are too coarse-grained for accurate masks as single-modal DINOs [3, 25] .In this paper, we complement these powerful unsupervised models as our proposed SIS to obtain pseudo labels for Unsupervised Multi-view Pedestrian Detection (UMPD), which iteratively segments PCA of DINOv2 features for 2D masks, with zero-shot semantic capability of vision-language model CLIP to identify pedestrians.For various 3D perception tasks, it is crucial to construct an explicit 3D volume from multi-view images. For example, 3D pose estimation methods [14] or 3D object detectors [30, 32] assigns 2D features to their corresponding 3D voxels as an explicit volume.Neural Radiance Field (NeRF) [17, 23, 24] learns a neural network to represent implicit volume. However, typical ≤ 10 [4, 12] or even 4 views [8] in multi-view pedestrian detection datasets are too sparse for RFP [23] that needs 20∼80 views, and LERF [17] that fits videos with 400∼600 views. They distill implicit semantics of DINO [3] and CLIP [28] , while our SIS yields explicit 2D labels by [25, 28] . Implicit volume is also inapt for vertical constraints like our VBR. Another explicit technique is unsupervised Multi-View Stereo (MVS) [19, 35] , but 3D volume of the whole scene is constructed from images, rather than only foreground like pedestrians. Differently, for 3D volume by our proposed GVD, differentiable rendering framework PyTorch3D [15] renders 3D densities and colors into both 2D masks and images, learned via our SIS pedestrian masks. 
proposed method: 
 As is illustrated in Figure 2 , our proposed UMPD approach comprises three key components: 1) Semantic-aware Iterative Segmentation (SIS) segments the PCA values into masks iteratively, with zero-shot semantic capability of CLIP to identify pedestrians; 2) Geometric-aware Volume-based Detector (GVD) encodes multiview 2D images into a 3D volume via geometric correspondence, and learns to predict density and color by the rendering losses with SIS masks; 3) Vertical-aware BEV Regularization (VBR) method constrains the 3D density from GVD to be vertical on the BEV plane. More details will be introduced in the following sections.With the powerful unsupervised methods like DINOv2 [25] , similar PCA values of cross-image features indicate the same concepts, which can be better controlled than the merely single-image salient objects in the previous unsupervised methods [3, 34] . In the figures of DINOv2 paper, even abstract concepts like "wings" of airplane and bird in different images are co-segmented. Thus, it is adopted to identify pedestrians based on multi-view cross-image information. 𝑍 (1) as a linear mapping from X to 𝜙 (1) = (𝜙 (1)EQUATIONEQUATIONP ) ⊤ , where P 𝑝=1 𝑥 𝑝𝑑 = 0 after the Zero Standardization of each 𝑋 𝑝 :𝑍 (1) = 𝜙(1)EQUATIONEQUATIONLemma 1. The 1 st PCA vector 𝜙 (1) of X: (a) maximizes the global variance var(𝜙 (1) ) = 1 P P 𝑝=1 (𝜙 (1) 𝑝 ) 2 ; (b) minimizes the reconstruction loss min 𝑍 (1) ∥X -X𝑍 (1) (𝑍 (1) ) ⊤ ∥ 2 , where 𝑍 (1) is a bi-direction mapping between high-dimensional X and low-dimensional 𝜙 (1) .Given the foreground or background that co-exist in multiple images, if their DINOv2 features X are similar, according to the bi-direction mapping ensured by a minimized reconstruction loss in Lemma 1, the 1 st PCA values 𝜙 (1) of these features are also similar and distinguished from the opposite parts, as is illustrated by the two peaks in the histogram of Figure 3 . Then, 𝜙 Therefore, given the pixel-wise PCA values Φ 𝑛𝑖 𝑗 of Φ ∈ R N×H×W , where Φ are reshaped from the vector 𝜙 (1) ∈ R P , they are segmented into 2D masks M 𝑡 =1 𝑛 of each view 𝑛 in Figure 3 via the threshold value 𝜙 thr , which is formulated as:M 𝑡 =1 𝑛𝑖 𝑗 =1.0, Φ 𝑛𝑖 𝑗 > 𝜙 thr , as foreground, 0.0, Φ 𝑛𝑖 𝑗 ≤ 𝜙 thr , as background.(2) † For further proofs, experimental results, and visualizations, please refer to our supplementary materials: https://lmy98129.github.io/academic/src/UMPD-Appendix.pdf. if UseCLIP then 12:M 𝑡 ← Select foreground by S of CLIP in Eq. 313:if t>1 then ⊲ Update foreground 14:M 𝑡 -1 [1(M 𝑡 -1 = 1.0)] ← M 𝑡 ; M 𝑡 ← M 𝑡 -115:X 𝑡 +1 ← X 𝑡 [1(M 𝑡 = 1.0)] ⊲ Update features 16:return M T PCA ⊲ Output pseudo labels 3.1.2 Semantic-aware Iterative Segmentation of PCA. Recalling the Lemma 1, DINOv2 features are bi-directionally mapped to their PCA values. However, as is illustrated in the Figure 3 , they represent the ground plane as the most distinguished background, but the remaining parts are pedestrians and their contexts. Thus, more PCA iterations in Algorithm 1 is used to further segment the pedestrians and non-human background inside these coarse-grained features. Firstly, "foreground" features X 𝑡 [1(M 𝑡 = 1.0)] ∈ R P ′ ×D , P ′ ≪ P are collected to calculate the 1 st PCA values and yield new mask M 𝑡 by Eq.2. Differently, the mask is only updated inside foreground of 𝑡 -1, where new background is thus merged into the previous ones. However, in Figure 4 (a), the ≤ 𝜙 thr part is the real foreground, which is inconsistent with Eq. 2, because merely inter-image concepts are segmented, regardless of which part means the real pedestrians.Inspired by the zero-shot semantic capability to recognize the object classes [20] of vision-language models like CLIP [28] , the original pooling is modified [40] to obtain 2D masks. Denoted as "⊛" in Figure 2 M 𝑡 𝑛 = M 𝑡 𝑛 , if S human 𝑛 ∩ M 𝑡 𝑛 > S human 𝑛 ∩ (1 -M 𝑡 𝑛 ) , 1 -M 𝑡 𝑛 , otherwise.(3)Manual BEV labels and 2D pedestrian masks from SIS can represent different observations of 3D pedestrians, i.e., the former are the top-down observations, and the latter are the surrounding ones.To learn from these pseudo labels and predict the BEV pedestrian positions, a 3D volume as a "bridge" between them is constructed from encoding the input multi-view 2D images by our proposed fully-unsupervised detector, which is a 2D-3D cross modal mapping. Geometric Projection for 3D Volume. Inspired by the previous multi-view 3D methods [30, 32] for supervised tasks, we extract 2D features F 𝑛 ∈ R C×H×W of each view 𝑛 via a visual encoder [10] as is shown in Figure 5 (a), where C is the channel number. Then, pixel-wise feature 𝐹 𝑛 𝑢𝑣 ∈ R C is back-projected into potential 3D voxels via the geometry of 2D-3D cross modal mapping.Formally, given a pinhole camera calibrated with intrinsic and extrinsic matrices {K 𝑛 , Ψ 𝑛 }, its geometric model to obtain a 2D pixel (𝑢, 𝑣) ⊤ captured from a 3D voxel position (𝑥, 𝑦, 𝑧) ⊤ is:EQUATIONwhere Π 0 is an auxiliary matrix to obtain (𝑥 ′ , 𝑦 ′ , 𝑧 ′ ) ⊤ from the 3D homogeneous coordinate (𝑥 ′ , 𝑦 ′ , 𝑧 ′ , 1) ⊤ after the transformation of extrinsic matrix Ψ 𝑛 , and 𝜆 is the depth distance between this 3D voxel and camera optical origin. 1 𝜆 ensures the 2D homogeneity of (𝑢, 𝑣, 1) ⊤ from (𝑢 ′ , 𝑣 ′ , 𝜆) ⊤ after applying intrinsic matrix K.In 3D geometry, there are various 3D voxels (𝑥, 𝑦, 𝑧) ⊤ with different potential depths 𝜆 in Eq. 4, that derive the same 2D pixel (𝑢, 𝑣, 1) ⊤ . For such a one-to-many 2D-3D cross-modal correspondence, the 2D pixel feature 𝐹 𝑛 𝑢𝑣 is assigned to all these 3D voxel positions to form a 3D volume V 𝑛 ∈ R C×X×Y×Z from each view 𝑛:EQUATIONTo fuse the 3D volumes V 𝑛 of each view 𝑛 in Figure 5 (b) into a unified V, a soft-max function is used like supervised volume-based methods [14] to re-weight V 𝑛 , where the highest feature values at its voxel in all the views 𝑛 are assigned with the largest weights:EQUATIONwhere V ∈ R C×X×Y×Z and • denotes Hadamard Product. To predict the 3D density D ∈ [0, 1] X×Y×Z and color C ∈ [0, 1] 3×X×Y×Z from the volume V, 3D-convolution with a 5 × 5 × 5 kernel is adopted as a decoder for a larger 3D receptive field, like the large-kernel 2D-convolution of the supervised methods [12] on the BEV features.Rendering Losses with Pseudo Labels. With the 2D pedestrian masks M as pseudo labels, differential rendering framework PyTorch3D [15] renders the predicted 3D density D into 2D masks M by the pinhole camera model in Eq. 4 .In details, given a 2D pixel position (𝑢, 𝑣) ⊤ in a view 𝑛, the 3D direction of the ray 𝑟 from the 3D position of camera optical origin 𝑜 through (𝑢, 𝑣) ⊤ is 𝛿. Any 3D position along this ray is 𝑟 (𝜏) = 𝑜+𝜏 (𝛿). 𝜏 is the distance between 𝑟 (𝜏) and 𝑜, whose maximum is Λ. The rendered 2D mask pixel M𝑛 (𝑢, 𝑣) = M(𝜏) from D is denoted as:EQUATIONwhere T (𝜏) is the transmittance (i.e., optical opacity) at distance 𝜏.Similarly, to discriminate crowded pedestrians by their appearances, the predicted 3D colors C are also rendered into 2D images Ĩ. Each 2D pixel Ĩ𝑛 (𝑢, 𝑣) = Ĩ(𝜏) of rendered image is formulated as:EQUATION)Note that predicted 3D color C is view-invariant like the real world, i.e., it is identical in any observations, thus input is 3D position 𝑟 (𝜏) without view direction 𝛿 as NeRF [24] . Since only foreground have colors, the pseudo labels are images I masked by M.Following the official instruction of PyTorch3D [15] , Huber Loss L ℎ𝑢𝑏𝑒𝑟 [13] is used with inputs θ𝑛𝑖 𝑗 ∈ Θ, 𝜃 𝑛𝑖 𝑗 ∈ Θ as 2D predictions and pseudo labels for 2D mask M or image I in Figure 5(c) :EQUATIONAs is shown in Figure 2 (c) and 5(d), if predicted pedestrians are leaning or laying down, their BEV occupancies, i.e., maximized 3D density along Z-axis, are unnaturally larger than standing. Thus, Vertical-aware BEV Regularization (VBR) is proposed to regularize the pedestrian poses with minimized BEV occupancy:EQUATIONFinally, the overall loss function L that optimizes the parameters 𝜃 * = arg min L of our UMPD to predict the best 3D color and density based on rendered Ĩ and M in Eq. 8 and 7 is formulated as: where weights = 1.0 of all loss item follow popular supervised multiview 3D detectors [30, 32] . For inference, the detection results of our proposed UMPD are the predicted 3D density D projected on BEV, i.e., Ω = max(D 𝑥 𝑦𝑧 ) 𝑧 ∈ R X×Y , where D 𝑥 𝑦𝑧 ∈ D.EQUATION 
experiments: 
 In this section, extensive experiments are conducted on popular multi-view pedestrian detection benchmarks, i.e., Wildtrack, Terrace, and MultiviewX, to evaluate our proposed UMPD. Ablation study is performed on our proposed key components SIS, GVD, and VBR. Furthermore, qualitative analysis and state-of-the-art comparisons on these benchmarks are also reported.Wildtrack [4] and Terrace [8] datasets are collected from real-world interested regions surrounded by multiple calibrated and synchronized cameras, where the unscripted pedestrians are naturally walking or standing. MultiviewX [12] is a challenging dataset using Unity 3D Engine to simulate more populated scenes, featured with higher crowdedness than [4, 8] . Multiple Object Detection Accuracy and Precision [16] (MODA and MODP), Precision, and Recall are evaluation metrics. Table 1 shows more details of these datasets. [34] segments salient objects in red and green circles, while our multi-view SIS avoids them. Grounded-SAM [29] ignores small pedestrians in green circle. Supervised DINOv2-based Mask2Former [6] yields fine-grained masks in orange circles. Our proposed UMPD method is based on PyTorch [26] framework. For our SIS, unsupervised features for PCA with 𝜙 𝑡ℎ𝑟 = 2 are extracted by ViT-B/14 DINOv2 model [25] . S human is generated by ResNet-50 [10] of unsupervised vision-language model CLIP [28, 40] . The visual encoder of our GVD is ResNet-18 [10] following [12] . The result Ω is post-processed with a 3×3 Gaussian Kernel and > 0.4 threshold used by [12] . 4×A5000 GPUs are used for training with a mini-batch 4 and learning rate 1 × 10 -2 , and 1×GPU for testing. Our SIS and VBR are only for training. The inference time of GVD is ∼1.0s/frame. Each frame comprises multi-view images.The ablation study is performed on the popular real-world dataset Wildtrack. In Table 2 , for our proposed SIS, different PCA iteration T PCA and the zero-shot semantic capability of CLIP are experimented. "T PCA =1" performs worse with coarse-grained foreground as shown in Figure 3 Meanwhile, "T PCA =3" causes oversegmentation. Although CLIP fixes some mistakes by keeping some Table 5 : Different 2D mask segmentation methods for pseudo labels: our proposed SIS, single-image method CutLER [34] , supervised Mask2Former [6] based on DINOv2 [25] , and Grounded-SAM [29] with Grounding-DINO [22] prompts. pedestrians parts as the foregrounds, the results are still not ideal. Finally, our SIS is equipped with CLIP to identify pedestrians and "T PCA =2" as a proper setting. For the threshold 𝜙 𝑡ℎ𝑟 of our SIS, the overall best 𝜙 𝑡ℎ𝑟 = 2 is obtained among all datasets in Table 3 . Meanwhile, Table 4 evaluates our proposed GVD and VBR by: 1) 2D-to-3D geometric projection with different volume fusion; 2) 3D-to-2D rendering losses such as L color and L VBR , since L mask is inevitable to predict detection results Ω. For the volume fusion operations, the soft-max re-weighting in Eq. 6 achieves higher MODA and MODP, while simpler adding and concatenation are insufficient to directly handle such complex 3D volumes, which is consistent with the experiments on other multi-view tasks [14] .For the loss functions, L color assists to detect better by discriminating the different appearances of pedestrian instances than merely L mask . Note that our L VBR follows the natural vertical human poses and thus brings significant improvements (+49.3 MODA and +9.1 MODP) on the default L mask and L color following PyTorch3D [15] .As is discussed in Section 2.2, CutLER [34] based on DINOv1 [3] is different from our proposed SIS with DINOv2 [25] , which segment all salient objects in a single image, rather than inter-image concepts. Released by the DINOv2 official code [25] , a new version of Mask2Former [6] is initialized with DINOv2 pre-training, and finetuned with supervision. Recently, Grounded-SAM [29] embraces SAM [18] with boxes from Grounding-DINO [22] as prompts.In Table 5 , we compare different 2D masks from these methods, as pseudo labels to train our UMPD. Single-view masks from CutLER without cross-view information performs worse. Similarly, singleview object boxes by Grounding-DINO [22] mislead Grounded-SAM [29] to yield low-quality masks. Mask2Former achieves better results by powerful pretraining and supervised fine-tuning, while our proposed fully-unsupervised SIS also performs competitively. In summary, the quality of 2D masks greatly affects the performance of our UMPD, which is worth future researches for better masks.In Figure 6 , we visualize the segmentation results of our unsupervised SIS with different PCA iteration T PCA and whether to use the vision-language model CLIP. At T PCA =1, foregrounds in Figure 6 2. At T PCA =2, the foreground is corrected by CLIP in Figure 6 (a), while > 𝜙 𝑡ℎ𝑟 in Eq. 2 is used in Figure 6 (b) and yields wrong mask, since PCA performs naïve segmentation regardless of semantic classes. Under the over-segmentation at T PCA =3, CLIP decreases some mistakes, which is consistent with the results in Table 2 . In Figure 7 , 2D masks of different segmentation methods are visualized. Noisy single-view masks by CutLER [34] comprise nonhuman salient objects like tent and truck. Wrong boxes without small pedestrians from Grounding-DINO [22] mislead the Grounded-SAM [29] . With DINOv2 pre-training and supervised fine-tuning, Mask2Former [6] performs better than our unsupervised SIS, e.g., pedestrians with various scales in the orange circles of Figure 7 (f).Moreover, Figure 8 and 9 shows the detection results by our unsupervised UMPD on both real-world [4, 8] and simulated datasets [12] . There are still some wrong results near the edges of region, where less overlapped camera views are insufficient for detection, especially on more populated MultiviewX in Figure 9 (b). Such challenges remain to be tackled by unsupervised works in the future.In Table 6 , we compare our proposed unsupervised approach UMPD with fully-supervised state-of-the-art methods on Wildtrack, Terrace, and MultiviewX benchmarks: detection-based methods RCNN & Clustering [36] and POM-CNN [4] ; anchor-based methods Deep-MCD [5] and Deep-Occlusion [2] ; anchor-free methods MVDet [12] , SHOT [31] , MVDeTr [11] , 3DROM [27] , and MVAug [7] ; crossdomain method GMVD [33] . " ‡" are results by training on simulated MultiviewX labels without real Wildtrack labels. For real-world datasets Wildtrack and Terrace, our proposed UMPD surpasses RCNN & Clustering, POM-CNN, Deep-Occlusion by MODA and MODP, as well as DeepMCD and GMVD by MODA, where UMPD better generalizes without any source domain. As is shown in Table 1 and Figure 9 (b), MultiviewX has higher crowdedness. For this challenging dataset, UMPD out-performs RCNN & Clustering by MODA and MODP, as well as DeepMCD and Deep-Occlusion by MODP. In summary, our fully-unsupervised UMPD achieves competitive performances on all real-world and simulated datasets, without any manual labels essential to mainstream supervised methods. 
conclusion: 
 In this paper, we have proposed a novel unsupervised multi-view pedestrian detection approach UMPD, which eliminates the heavy burden of manual annotations. For such a challenging task, three key components are proposed: SIS segments the PCA values of DINOv2 features iteratively, with zero-shot semantic capability of CLIP to identify pedestrians. Based on 2D-3D cross-modal mapping, GVD encodes multi-view images into a 3D volume, and learns to predict 3D density and color by the rendering losses with SIS pseudo labels. VBR constrains the predicted 3D density to be naturally vertical on the BEV ground plane. With these powerful components, our proposed UMPD achieves competitive performances on challenging benchmarks Wildtrack, Terrace, and MultiviewX. We hope this work, as the first fully-unsupervised method in this field, could be a start and inspire more interesting future researches. 
: 
 1 IntroductionDetecting pedestrians is fundamental in various real-world applications, especially where the fine-grained positions of pedestrians on Birds-Eye-View (BEV) are required rather than coarse-grained bounding boxes [21] , such as crowd forecasting [1] for safety and customer behavior analysis [9] for retailing. To avoid occlusion or smaller scales, as shown in Figure 1 (a), multiple cameras around an interested region are introduced to better detect pedestrians. Figure 2 : The architecture of our proposed UMPD approach. (a) Semantic-aware Iterative Segmentation (SIS) iteratively segments the PCA values of the DINOv2 [25] features into pseudo labels, based on vision-language model CLIP [28] as a foreground selector. (b) Geometric-aware Volume-based Detector (GVD) encodes multi-view 2D images into a 3D volume via 2D-3D crossmodal mapping, and learns to predict 3D density and color by L color and L mask with SIS pseudo labels (cameras mean rendering). (c) Vertical-aware BEV Regularization (VBR) constrains the predicted 3D density to be "human-like" vertical on BEV.Most previous methods depend on supervised learning, including the classic detection-based [4, 36] and anchor-based detectors [2, 5] , as well as more recent perspective-based anchor-free ones [7, 11, 12, 27, 31, 38] . They require supervised models to obtain 2D detection, segmentation or feature encoding, which are mapped onto the BEV for detection. Thus, pedestrian BEV positions are annotated.However, heavy human labors are essential to annotate BEV labels of all-view video frames [4, 8] . Recently, with tinier pedestrians that harder to label, multi-view counting dataset [37, 39] to count coarse number is introduced to evaluate accurate BEV detection [38] . Although game engines can auto-generate images and labels [12, 39] , domain gaps like different image styles or camera poses hinder better cross-domain performance [33] towards in-domain real scenes [4] . Therefore, manual labels are still inevitable.As is shown in Figure 1 (b) and (c), we found a potential solution without manual labels via 2D-3D cross-modal mapping. For 3D pedestrian density, the BEV labels mean a top-down observation, and 2D masks from cameras mean the surrounding observations. Hence, the 3D density becomes a "bridge", to be predicted by a volume-based detector from 2D images, learned from unsupervised 2D masks, and finally projected onto BEV as the detection results.For 2D pedestrian masks, we notice the recent powerful unsupervised vision-language models like DINOv2 [25] and CLIP [28] . Robust representations with inter-image co-exist concepts can be extracted by DINOv2, and converted into the 1 st Principal Component Analysis (PCA) values to be segmented as foreground masks. Meanwhile, CLIP model can identify object classes by texts, e.g., "A picture of a human", following the default template of CLIP [28] .To construct a 3D volume from multi-view images, some supervised methods for 3D object detection [30, 32] or 3D pose estimation [14] project each pixel of 2D features into potential 3D voxels based on geometric correspondence. Such powerful 3D volume frameworks inspire us to design a novel fully-unsupervised detector. Furthermore, differentiable rendering framework [15] can render 3D density predicted by a volume-based detector into the 2D mask of each view, which is learned via unsupervised 2D masks. To better discriminate the pedestrian instances from their appearances, the colors are also rendered and learned via original 2D images.In summary, we have observed a high dependency of the current mainstream supervised methods on the laborious manual labels. As is illustrated in Figure 2 , we propose a novel approach to tackle this problem via Unsupervised Multi-view Pedestrian Detection (UMPD). Our main contributions are:• Firstly, Semantic-aware Iterative Segmentation (SIS) method is proposed to extract the PCA values of DINOv2 representations, and segment them into 2D masks as pseudo labels.To identify the pedestrians, iterative PCA is adopted with zero-shot semantic classes of vision-language model CLIP.• Secondly, we propose Geometric-aware Volume-based Detector (GVD) to encode multi-view 2D images into a 3D volume via geometry, and learn to predict 3D density and color from this volume via rendering losses with SIS pseudo labels. • Thirdly, Vertical-aware BEV Regularization (VBR) method is further proposed to constrain the predicted 3D density to be vertical on BEV, following the natural pedestrian poses. • Finally, formed by these key components, our proposed UMPD, as the first fully-unsupervised method in this field to our best knowledge, performs competitively on popular Wildtrack, Terrace, and MultiviewX datasets, especially compared with the previous supervised methods. 
 
 Appendix:  


Decision: Initially, the paper was met with interest due to its innovative approach to unsupervised pedestrian detection, a significant step forward in reducing the labor intensity of annotations. However, concerns were raised about the generalization ability of the model, the robustness against various environmental factors, and the lack of detailed analysis on these aspects. The initial average rating leaned towards acceptance with reservations.

Post-rebuttal, the authors addressed several concerns by providing additional clarifications on the method's dependency on existing technologies and its performance in unsupervised settings. They also committed to extending their experiments to larger datasets and including more recent methods for a fair comparison. Despite these improvements, some concerns about the method's robustness and generalization remained partially addressed. The final average rating shifted slightly to a weak accept due to the novelty and potential impact of the research.

AC noticed and considered the authors' email during the rebuttal. Looking forward to seeing the final improved version of this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel unsupervised multi-view pedestrian detection method, aiming to reduce the labor intensity of annotations. This method includes three main modules: 1) Semantic-aware Iterative Segmentation (SIS), 2) Geometry-aware volume-based Detector (GVD), and 3) Vertival-aware BEV Regularization (VBR). The authors conduct extensive experiments on the mainstream multi-view pedestrian detection datasets, such as Wildtrack, Terrace, and MultiviewX, demonstrating the competitive performance compared to previous fully supervised methods.
strength_weakness: strength_weakness: 1.	Innovative: This paper proposes a completely unsupervised multi-view pedestrian detection method, which is an important innovation in the field of pedestrian detection.
2.	Depth of technology: UMPD combines multiple advanced technologies such as DINOv2, CLIP and 3D volume construction, showing deep technical research.
3.	Experimental validation: The authors conducted extensive experiments on multiple datasets to verify the effectiveness of the proposed method.
4.	Performance Competition: UMPD shows competitive performance compared to existing supervised learning methods, which indicates the potential of unsupervised learning in this area.
limitations: limitations: 1.	The generalization ability of the model is not discussed in detail in the paper, especially in different environments and different pedestrian densities.
2.	For the factors that affect the accuracy of pedestrian detection, such as occlusion, illumination change and angle of view change, the paper lacks an in-depth analysis of the robustness of the model.
3.	How to determine the threshold used in subsection 3.1.1?
4.	The related metrics (e.g., MODA, MODP) should be clarified clearly.
5.	Figure 8 lacks the result visualization of the Terrace dataset. 
6.	In formula (11), there is no corresponding weight coefficient for the loss function, and the ablation experiment for each weight of the loss function is lacking.
7.  The method is only evaluated on small datasets, such as Wildtrack and MultiviewX. I doubt whether this method can be performed on large datasets such as CVCS or CityStreet as in "Multi-View People Detection in Large Scenes via Supervised View-Wise Contribution Weighting".
8. There is no possible baseline methods for unsupervised settings.
paper_summary: paper_summary: This paper presents UMPD, a method that eliminates the need for manual annotations in pedestrian detection using multiple camera views. UMPD consists of three components: Semantic-aware Iterative Segmentation (SIS) for generating 2D masks as pseudo labels, Geometry-aware Volume-based Detector (GVD) for predicting 3D density and color from multi-view images, and Vertical-aware BEV Regularization (VBR) to maintain vertical alignment in BEV projections. Evaluations on Wildtrack, Terrace, and MultiviewX datasets show UMPD performs competitively with state-of-the-art supervised methods. This approach is the first fully unsupervised method in the field, demonstrating significant potential for future research.
strength_weakness: strength_weakness: - The proposed approach represents a valuable adaptation of unsupervised learning to the multi-view pedestrian detection task, which is essential when considering the potential labeling costs from a real-world perspective.

- In conjunction with the CLIP model, the approach effectively leverages multi-modal knowledge to achieve notable performance.
limitations: limitations: - While this work (unsupervised/self-supervised) learning can be the first work in the multi-view pedestrian detection, the authors need to compare the existing unsupervised/self-supervised learning for the various multi-view tasks (e.g., [Ref_1, Ref_2, Ref_3]). If not, the effectiveness of the proposed unsupervised learning cannot be known.

- As mentioned in the first bullet in Sec. 1, one of the main novelty of this work is utilizing DINO and CLIP to generate pseudo-labels. However, this method is not novel. The previous works (e.g., [Ref_4] already adopt this scheme (difference between the existing works and this paper seems only the task-difference). What is the difference?

- The proposed method is heavily rely on the existing visual-encoder and text-encoder.

- The performance depends on the template used in the clip text encoder.

- In an unsupervised setting, the effectiveness of the proposed method relies on the choice of $\phi_{thr}$. The authors need to conduct experiments by varying $\phi_{thr}$.

- Why the authors does not adopt the recent unsupervised works for the multi-view pedestrian detection for comparison.

[Ref_1] X. Liu et al., "Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation," in NeurIPS 2022.

[Ref_2] J. Li et al., "DS-MVSNet: Unsupervised Multi-view Stereo via Depth Synthesis," in ACM MM 2022.

[Ref_3] K. Xiong et al., "CL-MVSNet: Unsupervised Multi-view Stereo with Dual-level Contrastive Learning," in ICCV 2023.

[Ref_4] J. Kerr et al., "LERF: Language Embedded Radiance Fields," in ICCV 2023.
paper_summary: paper_summary: This paper proposes a novel approach to pedestrian detection using multiple camera views without the need for manual annotations. The method, termed UMPD, leverages vision-language models and 2D-3D cross-modal mapping to detect pedestrians in 3D space. The experiments have demonstrated the acceptable performance of the method and the effectiveness of the three components.
strength_weakness: strength_weakness: 1. The proposed UMPD is the first fully-unsupervised multi-view pedestrian detection method, addressing the challenge of manual annotation. Experimental results demonstrate its achievement of acceptable results compared to other non-fully-unsupervised methods.
2. The paper provides relevant code and results, making it more credible.
3. The article is well-written.
limitations: limitations: I'm not an expert in this field, but I believe the innovation and significance of this work are commendable. The only minor weakness is that the results in Table 7, while acceptable, are not particularly outstanding.