Multi-View Multiple People Detection, Labelling and Tracking
Creators: name: Zhang, Yue
Publisher: name: University of Surrey
Year: 2023
Subjects: Multi-view detection
Multi-view labelling
Multi-view tracking

description: Multi-view multiple people detection, labelling and tracking has attracted considerable interest
description: in recent years and has been employed in many application domains such as video surveillance,
description: human pose estimation and 3D human modelling. However, correctly detecting, labelling
description: and tracking multiple people from different viewpoints is a challenging task, especially in
description: complex scenes, due to occlusions, visual ambiguities, as well as variations in appearance and
description: illumination. Recent approaches established based on deep learning such as object detection
description: and semantic segmentation have proved very successful at improving the performance of a wide
description: range of computer vision tasks. By contrast, limited works have been done with deep learning
description: in multi-view multiple people detection, labelling and tracking, due to the lack of multi-view
description: datasets. This thesis therefore introduces novel algorithms to advance the state-of-the-art in
description: multi-view detection, labelling and tracking on both images and videos based on deep learning,
description: whilst also establishing new synthetic datasets to address the lack of multi-view data in this
description: field.<br>
description: <br>
description: Firstly, we propose a multi-view framework for joint object detection and labelling based on
description: pairs of images. As the lack of multi-view data prevents training an end-to-end network for the
description: task, we therefore leverage some of the benefits of deep learning by introducing a novel singleview convolutional neural network (CNN) with a series of multi-view constraints. The proposed
description: framework extends the single-view Mask R-CNN approach to multiple views to achieve multiview object detection and labelling without need for additional training, thus overcoming the lack
description: of multi-view data. Dedicated components are embedded into the framework to match objects
description: across views by enforcing epipolar constraints, appearance feature similarity and class coherence.
description: By jointly processing detection and labelling in a unified network, the multi-view extension
description: enables the proposed framework to detect objects which would otherwise be mis-detected in a
description: single view, and achieves coherent object labelling across views.<br>
description: <br>
description: Next, to fundamentally address the shortage of multi-view dataset and leverage the advantage
description: of deep neural network directly into multi-view detection and labelling, we generate a largescale synthetic dataset named MV3DHumans for the multi-view multiple people detection and
description: labelling task by combining 3D human models, panoramic backgrounds, human poses, and
description: appearance detail rendering. The proposed MV3DHumans dataset consists of various scenes with
description: different degrees of crowdedness, providing sufficient data for training deep neural networks. We
description: then develop a novel end-to-end multi-view framework named Multi-View Labelling network
description: (MVL-net), using a CNN to learn multi-view consistency based on the generated synthetic
description: dataset. A novel labelling branch with a matching net is introduced in the MVL-net to predict
description: matching confidence scores for pairs of people from two views, thus solving the problem of label
description: prediction for an unknown number of people. In addition, a calibration component based on the
description: epipolar geometry is integrated in the proposed framework to further improve the performace.<br>
description: <br>
description: Finally, we extend our work from multi-view labelling on images to the more challenging tracking problem on multi-view videos. A synthetic multi-view video dataset named
description: MV3DHumansVideo is established by extending data generation from images to videos, consisting of hundreds of short multi-view video sequences for training and several long sequences
description: for testing. Based on the newly proposed multi-view video dataset, we propose a novel spatiotemporal correlation (STC) network, jointly achieving detection, temporal feature learning and
description: spatial feature learning based on multi-view videos. The proposed STC network combines
description: detection and feature learning in a unified network and seamlessly integrates temporal and spatial context information to enhance temporal feature and spatial feature learned for multi-view
description: video tracking. An efficient multi-view tracking framework is then developed based on learned
description: features to obtain consistent people labels for multi-view videos. We demonstrate that this
description: method can learn effective features for multi-view tracking and obtain consistent people labels
description: for entire multi-view videos.<br>