nils-herrmann/clean_scirepeval huggingface dataset:
Interaction recognition in wide areas using audiovisual sensors

We present an event recognition framework to detect interactions among objects, for example people, using a network of cameras and associated microphone pairs. The complementarity of the video and audio modalities is exploited to cover wide areas. In particular, object movements in portions of the scene that are not covered by the cameras' fields of view are estimated using the input from microphones. After estimating trajectories using audio-visual features, we recognize interactions based on a Coupled Hidden Markov Model Maximum a Posteriori (CHMM-MAP) approach. The states of the CHMM are initialized via Gaussian Mixture Model (GMM) clustering on a multi-dimensional feature space. Evaluation and comparison with three alternative methods demonstrate the effectiveness of the proposed CHMM-MAP trained on multiple features on both synthetic and real data.
Video-based multi-agent traffic surveillance system

This paper describes Monitorix, a video-based traffic surveillance multi-agent system. Monitorix agents are grouped in four tiers, according to the kind of information processing they perform: the sensors and effectors tier, the objective description tier, the application assistant tier, and the user assistant tier. The video analysis algorithms use an adaptive, data-driven, application independent approach to extract features from the video raw data. In spite of the diversity of agent tasks, adaptive learning algorithms are used in most cases. The integration of video analysis algorithms and agent technology is made via a special middle agent called Proxy. Monitorix is a fully decentralised multi-agent system living in a FIPA Platform and using FIPA Agent Communication Language. Tracking of vehicles across nonoverlapping cameras is performed by the Tracker agent, using a traffic model and learning algorithms that tune the model parameters.
Multiview Matching of Articulated Objects

We address the problem of multiview association of articulated objects observed using possibly moving and hand-held cameras. Starting from trajectory data, we encode the temporal evolution of the objects and perform matching without making assumptions on scene geometry and with only weak assumptions on the field-of-view overlaps. After generating a viewpoint invariant representation using self-similarity matrices, we put in correspondence the spatio-temporal object descriptions using spectral methods on the resulting matching graph. We validate the proposed method on three publicly available real-world datasets and compare it with alternative approaches. Moreover, we present an extensive analysis of the accuracy of the proposed method in different contexts, with varying noise levels on the input data, varying amount of overlap between the fields of view, and varying duration of the available observations.
Guest Editorial Introduction to the Special Issue on Group and Crowd Behavior Analysis for Intelligent Multicamera Video Surveillance

Despite significant progress in human behavior analysis over the past few years, most of today’s state-of-the-art algorithms focus on analyzing individual behavior in a simple environment monitored by a single camera. Recently, the widespread availability of cameras and a growing need for public safety have shifted the attention of researchers in video surveillance from individual behavior analysis to group and crowd behavior analysis in multicamera networks. Group behavior analysis provides a novel level for describing events, which are semantically more meaningful, highlighting barely visible relational connections among people. Crowd behavior analysis can also be used for anomaly detection such as panic scenarios, dangerous situations, and illegal behaviors in public spaces.
Content-aware ranking of video segments

We present an algorithm for ranking videos from multi-camera settings. The algorithm is based on the analysis of objects and events associated to them. Object analysis uses multiple features and is based on motion segmentation using color-based change detection and then trajectories for each target are generated using multi-frame graph matching. Event detection employs a hidden semi-Markov model with duration distribution that generates a sequence of activities performed by each target using Viterbi decoding. For each object size, pose and its associated events (including their duration) are used to rank each frame of a video. The proposed method for ranking is deadline driven. The performance of the proposed approach is demonstrated on standard multi-camera datasets as well as on simulated 3D multi-camera scenarios.
Re-identification with multiple source-cameras

Target re-identification approaches generally perform association between camera pairs only. However, in a multi-camera system many-to-one camera associations are needed when targets transit from multiple source-cameras to a destination-camera. To address this problem, we propose a person re-identification approach that generates camera-invariant object matching scores, which are based on re-identification score variations in multiple camera pairs. Each camera pair is represented with two parametric distribution models obtained by curve fitting on intra-class and inter-class target matching scores. These two models are combined to generate the likelihood of a correct match between a new target in the destination-camera and those in all source-cameras. We show the improvement in the performance of the proposed re-identification approach compared to existing pairwise approaches on two publicly available datasets.
Assessing tracking assessment measures

We propose a methodology to quantitatively compare the relative performance of tracking evaluation measures. The proposed methodology is based on determining the probabilistic agreement between tracking result decisions made by measures and those made by humans. We use tracking results on publicly available datasets with different target types and varying challenges, and collect the judgments of 90 skilled, semi-skilled and unskilled human subjects using a web-based performance assessment test. The analysis of the agreements allows us to highlight the variation in performance of the different measures and the most appropriate ones for the various stages of tracking performance evaluation.
Shadow identification and classification using invariant color models

A novel approach to shadow detection is presented. The method is based on the use of invariant color models to identify and to classify shadows in digital images. The procedure is divided into two levels: first, shadow candidate regions are extracted; then, by using the invariant color features, shadow candidate pixels are classified as self shadow points or as cast shadow points. The use of invariant color features allows a low complexity of the classification stage. Experimental results show that the method succeeds in detecting and classifying shadows within the environmental constrains assumed as hypotheses, which are less restrictive than state-of-the-art methods with respect to illumination conditions and the scene's layout.
Accurate video object segmentation through change detection

We propose an algorithm for the accurate extraction of video objects from color sequences. The semantics defining the video objects is motion, and the extraction algorithm is based on change detection. The color difference between frames is modeled so as to separate the contributions caused by sensor noise and illumination variations from those caused by meaningful objects. Sensor noise is eliminated by using a probability-based classification, and local illumination variations are removed using a knowledge-based approach that is formulated as a hypothesize-and-test scheme. Experimental results show that the proposed method provides accurate contours of multiple deformable objects, thus providing a reliable input to object-based applications such as those supported by the MPEG-4 and MPEG-7 standards.
Audio-assisted trajectory estimation in non-overlapping multi-camera networks

We present an algorithm to improve trajectory estimation in networks of non-overlapping cameras using audio measurements. The algorithm fuses audiovisual cues in each camera's field of view and recovers trajectories in unobserved regions using microphones only. Audio source localization is performed using Stereo Audio and Cycloptic Vision (STAC) sensor by estimating the time difference of arrival (TDOA) between microphone pair and then by computing the cross correlation. Audio estimates are then smoothed using Kalman filtering. The audio-visual fusion is performed using a dynamic weighting strategy. We show that using a multi-modal sensor with combined visual (narrow) and audio (wider) field of view can enable extended target tracking in non-overlapping camera settings. In particular, the weighting scheme improves performance in the overlapping regions. The algorithm is evaluated in several multi-sensor configurations using synthetic data and compared with state of the art algorithm.
Rate-adaptive multicast video streaming from teams of micro aerial vehicles

Video multicasting from cameras mounted on micro aerial vehicles (MAVs) is desirable for applications such as search and rescue, surveillance and disaster management. Because of the mobility of the video sources and the high data-rate of videos, the transmission rate should be adapted to the task at hand. Rate-adaptive video multicast streaming in 802.11 requires wireless link estimation as well as frequent feedback from multiple receivers. We propose an application layer rate-adaptive video multicast streaming framework using 802.11 adhoc network that is applicable when both the sender and the receiver nodes are mobile. The receiver nodes of a multicast group are dynamically elected based on their changing link conditions to gain feedback. An Application Layer Video Multicast Gateway (ALVM-GW) adapts the transmission rate and the video encoding rate based on the received feedback. Emulation results show that the proposed approach has balanced performance in terms of goodput, delay and packet loss.
Local Abnormality Detection in Video Using Subspace Learning

On-line abnormality detection in video without the use ofobject detection and tracking is a desirable task in surveillance.We address this problem for the case when labeledinformation about normal events is limited and informationabout abnormal events is not available. We formulatethis problem as a one-class classification, where multiplelocal novelty classifiers (detectors) are used to first learnnormal actions based on motion information and then todetect abnormal instances. Each detector is associated toa small region of interest and is trained over labeled samplesprojected on an appropriate subspace. We discover thissubspace by using both labeled and unlabeled segments.We investigate the use of subspace learning and comparetwo methodologies based on linear (Principal ComponentsAnalysis) and on non-linear subspace learning (LocalityPreserving Projections), respectively. Experimental resultson a real underground station dataset shows that the linearapproach is better suited for cases where the subspacelearning is restricted to the labeled samples, whereas thenon-linear approach is preferable in the presence of additionalunlabeled data.
PFT: A protocol for evaluating video trackers

The growing interest in developing video tracking algorithms has not been accompanied by the development of commonly used evaluation criteria to assess and to compare their performance. Researchers often present trackers' results on different datasets and evaluate them with different performance measures thus hindering both formative and summative quality assessment. In this paper, we present a protocol to evaluate the performance of tracking algorithms that tests video trackers using a set of trials and a pre-defined set of sequences and that enables objective and reproducible performance evaluation of trackers using ground truth information. Each trial highlights strengths and weaknesses of a tracker on simulated test scenarios on real sequences that represent real-world scenarios. Moreover a new evaluation measure is introduced that allows us to summarize the performance of a tracker based on the lost-track-ratio curve. The validation and the effectiveness of the proposed protocol is demonstrated experimentally on three trackers and its implementation is made available online to the research community.
Video-Based Human Behavior Understanding: A Survey

Understanding human behaviors is a challenging problem in computer vision that has recently seen important advances. Human behavior understanding combines image and signal processing, feature extraction, machine learning, and 3-D geometry. Application scenarios range from surveillance to indexing and retrieval, from patient care to industrial safety and sports analysis. Given the broad set of techniques used in video-based behavior understanding and the fast progress in this area, in this paper we organize and survey the corresponding literature, define unambiguous key terms, and discuss links among fundamental building blocks ranging from human detection to action and interaction recognition. The advantages and the drawbacks of the methods are critically discussed, providing a comprehensive coverage of key aspects of video-based human behavior understanding, available datasets for experimentation and comparisons, and important open research issues.
Microphone-Array Ego-Noise Reduction Algorithms for Auditory Micro Aerial Vehicles

When a micro aerial vehicle (MAV) captures sounds emitted by a ground or aerial source, its motors and propellers are much closer to the microphone(s) than the sound source, thus leading to extremely low signal-to-noise ratios (SNR), e.g., −15 dB. While microphone-array techniques have been investigated intensively, their application to MAV-based ego-noise reduction has been rarely reported in the literature. To fill this gap, we implement and compare three types of microphone-array algorithms to enhance the target sound captured by an MAV. These algorithms include a recently emerged technique, time-frequency spatial filtering, and two well-known techniques, beamforming and blind source separation. In particular, based on the observation that the target sound and the ego-noise usually have concentrated energy at sparsely isolated time-frequency bins, we propose to use the time-frequency processing approach, which formulates a spatial filter that can enhance a target direction based on local direction of arrival estimates at individual time-frequency bins. By exploiting the time-frequency sparsity of the acoustic signal, this spatial filter works robustly for sound enhancement in the presence of strong ego-noise. We analyze in details the three techniques and conduct a comparative evaluation with real-recorded MAV sounds. Experimental results show the superiority of blind source separation and time-frequency filtering in low-SNR scenarios.
Low-cost multi-camera object matching

We propose an object matching approach aimed at smartphone cameras that exploits the well-known concept of local sets of features for object representation. We also enable the temporal alignment of cameras by exploiting the frames of detected objects to group objects appeared in the same time interval for the assignment within each camera. The proposed approach does not need training thus making it suitable for matching during short temporal intervals. We use both outdoor and indoor datasets for the evaluation, and show that the proposed method reduces up to 95% the amount of information to be stored and communicated.
A batch asynchronous tracker for wireless smart-camera networks

Distributed tracking in wireless smart-camera networks is affected by varying local processing delays that generally depend on the current scene complexity. As a consequence, each camera makes target information available to the network at different time instants. These unknown delays compound the drifts caused by local clocks and may induce tracking failures when target information is fused. To address this problem, we propose a distributed batch asynchronous tracker for fully connected wireless smart-camera networks. The cameras use the information filter to estimate the target state information and to predict corresponding information of other cameras based on the asynchronous information received from them. Finally, the temporally aligned information is fused. We show that the proposed approach achieves higher tracking accuracy than the state of the art under varying degrees of asynchronism.
Distributed target tracking under realistic network conditions

Distributed target tracking has been a widely studied problem for different applications, more recently considering distributed particle filter techniques for Wireless Sensor Networks (WSNs). However, the adaptation of distributed tracking filters to sensor networks with limited field-of-view and large raw-data management constraints under realistic networking conditions has not yet received enough attention. In this work, we extend the distributed particle filter formulation and enable its operation in realistic scenarios by introducing a next-hop selection mechanism for the aggregation chain and a target hand- over strategy that is capable of handling detection misses and target losses. The filter is tested using a network simulation environment over an area of 6000 m2 and networks of up to 1000 sensors. (5 pages)
Inertial-Vision: Cross-Domain Knowledge Transfer for Wearable Sensors

Multi-modal ego-centric data from inertial measurement units (IMU) and first-person videos (FPV) can be effectively fused to recognise proprioceptive activities. Existing IMU-based approaches mostly employ cascades of handcrafted triaxial motion features or deep frameworks trained on limited data. FPV approaches generally encode scene dynamics with motion and pooled appearance features. In this paper, we propose a multi-modal ego-centric proprioceptive activity recognition that uses a convolutional neural network (CNN) followed by a long short-term memory (LSTM) network, transfer learning and a merit-based fusion of IMU and/or FPV streams. The CNN encodes short-term temporal dynamics of the ego-motion and the LSTM exploits the long-term temporal dependency among activities. The merit of a stream is evaluated with a sparsity measure of its initial classification output. We validate the proposed framework on multiple visual and inertial datasets.
Multi-Camera Scene Analysis using an Object-Centric Continuous Distribution Hidden Markov Model

We propose a multi-camera event detection framework that can operate on a common ground plane as well as on the image plane. The proposed event detector is based on an object-centric state modeling that uses a continuous distribution hidden Markov model (CDHMM). Video objects are first detected using statistical change detection and then tracked using graph matching. Next, the algorithm recognizes events by estimating the most likely object state sequence using a HMM decoding strategy, based on the Viterbi algorithm. We demonstrate and evaluate the proposed framework on standard event detection datasets with single and multiple cameras, with both overlapping and non-overlapping fields of view.
An Iterative Approach to Source Counting and Localization Using Two Distant Microphones

We propose a time difference of arrival (TDOA) estimation framework based on time-frequency inter-channel phase difference (IPD) to count and localize multiple acoustic sources in a reverberant environment using two distant microphones. The time-frequency (T-F) processing enables exploitation of the nonstationarity and sparsity of audio signals, increasing robustness to multiple sources and ambient noise. For inter-channel phase difference estimation, we use a cost function, which is equivalent to the generalized cross correlation with phase transform (GCC) algorithm and which is robust to spatial aliasing caused by large inter-microphone distances. To estimate the number of sources, we further propose an iterative contribution removal (ICR) algorithm to count and locate the sources using the peaks of the GCC function. In each iteration, we first use IPD to calculate the GCC function, whose highest peak is detected as the location of a sound source; then we detect the T-F bins that are associated with this source and remove them from the IPD set. The proposed ICR algorithm successfully solves the GCC peak ambiguities between multiple sources and multiple reverberant paths.
Facial scan change detection

We present a method for quantifying and localising changes in two facial scans of the same person taken at two different time instants. The method is based on rigid registration and semantic feature extraction, followed by discrepancy computation. The proposed method combines the Landmark Transform (LT) method, which is applied on semantic feature points, and the Iterative Closest Point (ICP) algorithm, which is performed on semantic regions. Finally, the discrepancy between the two scans is computed using the Symmetric Hausdorff distance. Experimental results with both synthetic and real data show the effectiveness of the proposed method which has also been validated by an experienced clinical scientist. Moreover, the method is being used as support in clinical studies on a 3D object database with more than 1000 facial scans.
Adaptive Online Performance Evaluation of Video Trackers

We propose an adaptive framework to estimate the quality of video tracking algorithms without ground-truth data. The framework is divided into two main stages, namely, the estimation of the tracker condition to identify temporal segments during which a target is lost and the measurement of the quality of the estimated track when the tracker is successful. A key novelty of the proposed framework is the capability of evaluating video trackers with multiple failures and recoveries over long sequences. Successful tracking is identified by analyzing the uncertainty of the tracker, whereas track recovery from errors is determined based on the time-reversibility constraint. The proposed approach is demonstrated on a particle filter tracker over a heterogeneous data set. Experimental results show the effectiveness and robustness of the proposed framework that improves state-of-the-art approaches in the presence of tracking challenges such as occlusions, illumination changes, and clutter and on sequences containing multiple tracking errors and recoveries.
Target Detection and Tracking With Heterogeneous Sensors

We present a multimodal detection and tracking algorithm for sensors composed of a camera mounted between two microphones. Target localization is performed on color-based change detection in the video modality and on time difference of arrival (TDOA) estimation between the two microphones in the audio modality. The TDOA is computed by multiband generalized cross correlation (GCC) analysis. The estimated directions of arrival are then postprocessed using a Riccati Kalman filter. The visual and audio estimates are finally integrated, at the likelihood level, into a particle filter (PF) that uses a zero-order motion model, and a weighted probabilistic data association (WPDA) scheme. We demonstrate that the Kalman filtering (KF) improves the accuracy of the audio source localization and that the WPDA helps to enhance the tracking performance of sensor fusion in reverberant scenarios. The combination of multiband GCC, KF, and WPDA within the particle filtering framework improves the performance of the algorithm in noisy scenarios. We also show how the proposed audiovisual tracker summarizes the observed scene by generating metadata that can be transmitted to other network nodes instead of transmitting the raw images and can be used for very low bit rate communication. Moreover, the generated metadata can also be used to detect and monitor events of interest.