nils-herrmann/clean_scirepeval huggingface dataset:
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition

Automatic affect analysis has attracted great interest in various contexts including the recognition of action units and basic or non-basic emotions. In spite of major efforts, there are several open questions on what the important cues to interpret facial expressions are and how to encode them. In this paper, we review the progress across a range of affect recognition applications to shed light on these fundamental questions. We analyse the state-of-the-art solutions by decomposing their pipelines into fundamental components, namely face registration, representation, dimensionality reduction and recognition. We discuss the role of these components and highlight the models and new trends that are followed in their design. Moreover, we provide a comprehensive analysis of facial representations by uncovering their advantages and limitations; we elaborate on the type of information they encode and discuss how they deal with the key challenges of illumination variations, registration errors, head-pose variations, occlusions, and identity bias. This survey allows us to identify open issues and to define future directions for designing real-world affect recognition systems.
Online Cross-Modal Adaptation for Audio–Visual Person Identification With Wearable Cameras

We propose an audio-visual target identification approach for egocentric data with cross-modal model adaptation. The proposed approach blindly and iteratively adapts the time-dependent models of each modality to varying target appearance and environmental conditions using the posterior of the other modality. The adaptation is unsupervised and performed online; thus, models can be improved as new unlabeled data become available. In particular, accurate models do not deteriorate when a modality is underperforming thanks to an appropriate selection of the parameters in the adaptation. Importantly, unlike traditional audio-visual integration methods, the proposed approach is also useful for temporal intervals during which only one modality is available or when different modalities are used for different tasks. We evaluate the proposed method in an end-to-end multimodal person identification application with two challenging real-world datasets and show that the proposed approach successfully adapts models in presence of mild mismatch. We also show that the proposed approach is beneficial to other multimodal score fusion algorithms.
Multi-Tracker Partition Fusion

We propose a decision-level approach to fuse the output of multiple trackers based on their estimated individual performance. The proposed approach is composed of three main steps. First, we group trackers into clusters based on the spatiotemporal pair-wise correlation of their short-term trajectories. Then, we evaluate performance based on reverse-time analysis with an adaptive reference frame and define the cluster with trackers that appear to be successfully following the target as the on-target cluster. Finally, the state estimations produced by trackers in the on-target cluster are fused to obtain the target state. The proposed fusion approach uses standard tracker outputs and can therefore combine various types of trackers. We tested the proposed approach with several combinations of state-of-the-art trackers and also compared it with individual trackers and other fusion approaches. The results show that the proposed approach improves the state estimation accuracy under multiple tracking challenges.
Perceptual prefiltering for video coding

Semantic segmentation is generally associated with second generation video coders, or object-based coders. Object-based coders encode different video objects separately in order to achieve lower bitrates and to enable object-based functionalities. In this paper, we present an encoding framework that uses semantic segmentation to improve the performance of first generation video coders, or frame-based coders. Semantic segmentation is exploited in a prefiltering step prior to encoding. This prefiltering step mimics the way humans treat visual information by separating relevant information from contextual information. Contextual information is then simplified, thus reducing the information to be coded. Experimental results on indoor as well as on outdoor test sequences, when the semantics is defined by motion, show that the proposed prefiltering improves the perceived quality with respect to traditional video coders.
Multi-camera Scheduling for Video Production

We present a novel algorithm for automated video production based on content ranking. The proposed algorithm generates videos by performing camera selection while minimizing the number of inter-camera switch. We model the problem as a finite horizon Partially Observable Markov Decision Process over temporal windows and we use a multivariate Gaussian distribution to represent the content-quality score for each camera. The performance of the proposed approach is demonstrated on a multi-camera setup of fixed cameras with partially overlapping fields of view. Subjective experiments based on the Turing test confirmed the quality of the automatically produced videos. The proposed approach is also compared with recent methods based on Recursive Decision and on Dynamic Bayesian Networks and its results outperform both methods.
Constrained Optimization for Plane-Based Stereo

Depth and surface normal estimation are crucial components in understanding 3D scene geometry from calibrated stereo images. In this paper, we propose visibility and disparity magnitude constraints for slanted patches in the scene. These constraints can be used to associate geometrically feasible planes with each point in the disparity space. The new constraints are validated in the PatchMatch Stereo framework. We use these new constraints not only for initialization, but also in the local plane refinement step of this iterative algorithm. The proposed constraints increase the probability of estimating correct plane parameters, and lead to an improved 3D reconstruction of the scene. Furthermore, the proposed constrained initialization reduces the number of iterations before convergence to the optimal plane parameters. In addition, as most stereo image pairs are not perfectly rectified, we modify the view propagation process by assigning the plane parameters to the neighbors of the candidate pixel. To update the plane parameters in the plane refinement step, we use a gradient free non-linear optimizer. The benefits of the new initialization, propagation, and refinement schemes are demonstrated.
Efficient Multitarget Visual Tracking Using Random Finite Sets

We propose a filtering framework for multitarget tracking that is based on the probability hypothesis density (PHD) filter and data association using graph matching. This framework can be combined with any object detectors that generate positional and dimensional information of objects of interest. The PHD filter compensates for missing detections and removes noise and clutter. Moreover, this filter reduces the growth in complexity with the number of targets from exponential to linear by propagating the first-order moment of the multitarget posterior, instead of the full posterior. In order to account for the nature of the PHD propagation, we propose a novel particle resampling strategy and we adapt dynamic and observation models to cope with varying object scales. The proposed resampling strategy allows us to use the PHD filter when a priori knowledge of the scene is not available. Moreover, the dynamic and observation models are not limited to the PHD filter and can be applied to any Bayesian tracker that can handle state-dependent variances. Extensive experimental results on a large video surveillance dataset using a standard evaluation protocol show that the proposed filtering framework improves the accuracy of the tracker, especially in cluttered scenes.
Time-frequency processing for sound source localization from a micro aerial vehicle

We address the problem of sound source localization with a microphone array mounted on a micro aerial vehicle (MAV). Due to the noise generated by motors and propellers, this scenario is characterized by extremely low signal-to-noise ratios (SNR). Based on the observation that the energy of MAV sound recordings is usually concentrated at isolated time-frequency bins, we propose a time-frequency processing framework to address this problem. We first estimate the direction of arrival of the sound at individual time-frequency bins. Then we formulate a set of spatially informed filters pointing at candidate directions in the search space. The output of the filtering tends to present high non-Gaussianity when the spatial filter is steered towards the target sound source. Finally, by measuring the non-Gaussianity of the spatial filtering outputs we build a spatial likelihood function from which we estimate the direction of the target sound. Experimental results with real-recorded MAV ego-noise show the superiority of the proposed method over the state of the art in performing source localization robustly.
Tracking video objects in cluttered background

We present an algorithm for tracking video objects which is based on a hybrid strategy. This strategy uses both object and region information to solve the correspondence problem. Low-level descriptors are exploited to track object's regions and to cope with track management issues. Appearance and disappearance of objects, splitting and partial occlusions are resolved through interactions between regions and objects. Experimental results demonstrate that this approach has the ability to deal with multiple deformable objects, whose shape varies over time. Furthermore, it is very simple, because the tracking is based on the descriptors, which represent a very compact piece of information about regions, and they are easy to define and track automatically. Finally, this procedure implicitly provides one with a description of the objects and their track, thus enabling indexing and manipulation of the video content.
Segmentation-driven perceptual quality metrics

We present a full-reference and a no-reference perceptual video quality metric that incorporate both low-level and high-level aspects of vision. Low-level aspects include color perception, contrast sensitivity, masking as well as artifact analysis. High-level aspects take into account the cognitive behavior of an observer when watching a video by means of semantic segmentation. Using the special case of semantic face segmentation, we evaluate the proposed segmentation-driven perceptual quality metrics using a range of test sequences and demonstrate an improvement of their prediction performance.
Tracking Multiple High-Density Homogeneous Targets

We present a framework for multitarget detection and tracking that infers candidate target locations in videos containing a high density of homogeneous targets. We propose a gradient-climbing technique and an isocontor slicing approach for intensity maps to localize targets. The former uses Markov chain Monte Carlo to iteratively fit a shape model onto the target locations, whereas the latter uses the intensity values at different levels to find consistent object shapes. We generate trajectories by recursively associating detections with a hierarchical graph-based tracker on temporal windows. The solution to the graph is obtained with a greedy algorithm that accounts for false-positive associations. The edges of the graph are weighted with a likelihood function based on location information. We evaluate the performance of the proposed framework on challenging datasets containing videos with high density of targets and compare it with six alternative trackers.
Cooperative Robots to Observe Moving Targets: Review

The deployment of multiple robots for achieving a common goal helps to improve the performance, efficiency, and/or robustness in a variety of tasks. In particular, the observation of moving targets is an important multirobot application that still exhibits numerous open challenges, including the effective coordination of the robots. This paper reviews control techniques for cooperative mobile robots monitoring multiple targets. The simultaneous movement of robots and targets makes this problem particularly interesting, and our review systematically addresses this cooperative multirobot problem for the first time. We classify and critically discuss the control techniques: cooperative multirobot observation of multiple moving targets, cooperative search, acquisition, and track, cooperative tracking, and multirobot pursuit evasion. We also identify the five major elements that characterize this problem, namely, the coordination method, the environment, the target, the robot and its sensor(s). These elements are used to systematically analyze the control techniques. The majority of the studied work is based on simulation and laboratory studies, which may not accurately reflect real-world operational conditions. Importantly, while our systematic analysis is focused on multitarget observation, our proposed classification is useful also for related multirobot applications.
Support Vector Motion Clustering

We present a closed-loop unsupervised clustering method for motion vectors extracted from highly dynamic video scenes. Motion vectors are assigned to nonconvex homogeneous clusters characterizing direction, size and shape of regions with multiple independent activities. The proposed method is based on support vector clustering. Cluster labels are propagated over time via incremental learning. The proposed method uses a kernel function that maps the input motion vectors into a high-dimensional space to produce nonconvex clusters. We improve the mapping effectiveness by quantifying feature similarities via a blend of position and orientation affinities. We use the Quasiconformal Kernel Transformation to boost the discrimination of outliers. The temporal propagation of the clusters’ identities is achieved via incremental learning based on the concept of feature obsolescence to deal with appearing and disappearing features. Moreover, we design an online clustering performance prediction algorithm used as a feedback that refines the cluster model at each frame in an unsupervised manner. We evaluate the proposed method on synthetic data sets and real-world crowded videos and show that our solution outperforms state-of-the-art approaches.
Macro-cuboïd based probabilistic matching for lip-reading digits

In this paper, we present a spatio-temporal feature representation and a probabilistic matching function to recognise lip movements from pronounced digits. Our model (1) automatically selects spatio-temporal features extracted from 10 digit model templates and (2) matches them with probe video sequences. Spatio-temporal features embed lip movements from pronouncing digits and contain more discriminative information than spatial features alone. A model template for each digit is represented by a set of spatio-temporal features at multiple scales. A probabilistic sequence matching function automatically segments a probe video sequence and matches the most likely sequence of digits recognised in the probe sequence. We demonstrate the proposed approach using the CUAVE database and compare our representational scheme with three alternative methods, based on optical flow, intensity gradient and block matching, respectively. The evaluation shows that the proposed approach outperforms the others in recognition accuracy and is robust in coping with variations in probe sequences.
Multi-camera track-before-detect

We present a novel multi-camera multi-target fusion and tracking algorithm for noisy data. Information fusion is an important step towards robust multi-camera tracking and allows us to reduce the effect of projection and parallax errors as well as of the sensor noise. Input data from each camera view are projected on a top-view through multi-level homographic transformations. These projected planes are then collapsed onto the top-view to generate a detection volume. To increase track consistency with the generated noisy data we propose to use a track-before-detect particle filter (TBD-PF) on a 5D state-space. TBD-PF is a Bayesian method which extends the target state with the signal intensity and evaluates each image segment against the motion model. This results in filtering components belonging to noise only and enables tracking without the need of hard thresholding the signal. We demonstrate and evaluate the proposed approach on real multi-camera data from a basketball match.
A Hierarchical Approach to Sensor Network Calibration by Joint Refinement of Sensor Pose and Object Trajectories

We present a Bayesian framework for absolute sensor calibration that naturally integrates scene maps as prior and observations of moving objects. In the proposed framework, we design a utility function that measures the statistical fitness of joint hypotheses for sensor positions and object trajectories. Because of the unknown trajectory, such utility function is defined over a space of varying dimensionality that has to be directly manipulated for accurate position estimation to exploit its discriminative properties. To this end, we design a hierarchical algorithm that, from an initial approximate sensor position, infers the most plausible paths followed by the observed moving objects through the scene. Then, a transdimensional Monte Carlo optimization created to iteratively improve the initially provided sensor estimate according to the MAP criterion. This paper instantiates the proposed general model for sensors providing relative 2-D object positions, such as video cameras, sonar, and radar (distance and bearing). The proposed model is applicable to different sensor types and can be used either as alternative to active positioning methods or as refinement for other calibration approaches.
Distributed and Decentralized Multicamera Tracking

We discussed emerging multicamera tracking algorithms that find their roots in signal processing, wireless sensor networks, and computer vision. Based on how cameras share estimates and fuse information, we classified these trackers as distributed, decentralized, and centralized algorithms. We also highlighted the challenges to be addressed in the design of decentralized and distributed tracking algorithms. In particular, we showed how the constraints derived from the topology of the networks and the nature of the task have favored so far decentralized architectures with multiple local fusion centers. Because of the availability of fewer fusion centers compared to distributed algorithms, decentralized algorithms can share larger amounts of data (e.g., occupancy maps) and can back-project estimates among views and fusion centers to validate results. Distributed tracking uses algorithms that can operate with smaller amounts of data at any particular node and obtain state estimates through iterative fusion. Despite recent advances, there are important issues to be addressed to achieve efficient multitarget multicamera tracking. Current algorithms either assume the track-to-measurement association information to be available for the tracker or operate on a small (known) number of targets. Algorithms performing track-to-measurement association for a time-varying number of targets with higher accuracy usually incur much higher costs, whose reduction is an important open problem to be addressed in multicamera networks.
Image Coding Using Depth Blurring for Aesthetically Acceptable Distortion

We introduce the concept of depth-based blurring to achieve an aesthetically acceptable distortion when reducing the bitrate in image coding. The proposed depth-based blurring is a prefiltering that reduces high-frequency components by mimicking the limited depth of field effect that occurs in cameras. To cope with the challenge of avoiding intensity leakage at the boundaries of objects when blurring at different depth levels, we introduce a selective blurring algorithm that simulates occlusion effects as occur in natural blurring. The proposed algorithm can handle any number of blurring and occlusion levels. Subjective experiments show that the proposed algorithm outperforms foveation filtering, which is the dominant approach for bitrate reduction by space-variant prefiltering.
Energy Consumption Models for Smart Camera Networks

Camera networks require heavy visual data processing and high-bandwidth communication. In this paper, we identify key factors underpinning the development of resource-aware algorithms and we propose a comprehensive energy consumption model for the resources employed by smart camera networks, which are composed of cameras that process data locally and collaborate with their neighbors. We account for the main parameters that influence consumption when sensing (frame size and frame rate), processing (dynamic frequency scaling and task load), and communication (output power and bandwidth) are considered. Next, we define an abstraction based on clock frequency and duty cycle that accounts for active, idle, and sleep operational states. We demonstrate the importance of the proposed model for a multicamera tracking task and show how one may significantly reduce consumption with only minor performance degradation when choosing to operate with an appropriately reduced hardware capacity. Moreover, we quantify the dependency on local computation resources and bandwidth availability. The proposed consumption model can be easily adjusted to account for new platforms, thus providing a valuable tool for the design of resource-aware algorithms and further research in resource-aware camera networks.
Grouping motion trajectories

We present a method to group trajectories of moving objects extracted from real-world surveillance videos. The trajectories are first mapped into a low dimensionality feature space generated through linear regression. Next the regression coefficients are clustered by a Gaussian Mixture Model initialized by K-means for improved efficiency. The model selection problem is solved with Bayesian Information Criterion that penalizes models with high complexity. We demonstrate the proposed approach on both synthetic and real-world scenes. Experimental results show that the proposed clustering method outperforms K-means and mixture of regression models, while also reducing the computational complexity compared to the latter.
Active visual tracking in multi-agent scenarios

We propose an active visual tracker with collision avoidance for camera-equipped robots in dense multi-agent scenarios. The objective of each tracking agent (robot) is to maintain visual fixation on its moving target while updating its velocity to avoid other agents. However, when multiple robots are present or targets intensively intersect each other, robots may have no accessible collision-avoiding paths. We address this problem with an adaptive mechanism that sets the pair-wise responsibilities to increase the total accessible collision-avoiding controls. The final collision-avoiding control accounts for motion smoothness and view performance, i.e. maintaining the target centered in the field of view and at a certain size. We validate the proposed approach under different target-intersecting scenarios and compare it with the Optimal Reciprocal Collision Avoidance and the Reciprocal Velocity Obstacle methods.