nils-herrmann/clean_scirepeval huggingface dataset:
Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study

Lately, multimodal approaches for automatic emotion recognition have gained significant scientific interest. In this paper, emotion recognition by combining physiological signals and facial expressions was studied. Heart rate variability parameters, respiration frequency, and facial expressions were used to classify person's emotions while watching pictures with emotional content. Three classes were used for both valence and arousal. The preliminary results show that, over the proposed channels, detecting arousal seem to be easier compared to valence. While the classification performance of 54.5% was attained with arousal, only 38.0% of the samples were classified correctly in terms of valence. In future, additional modalities as well as feature selection will be utilized to improve the results.
(Multiscale) Local Phase Quantisation histogram discriminant analysis with score normalisation for robust face recognition

In video based face recognition, faces typically experience challenging illumination conditions, blur, or localisation errors in several frames. To alleviate these challenges, quality measures can be used to remove the most severely degraded frames. Still, when the videos are taken in real life settings, degradations are likely to be present even in the highest quality frames, and robust recognition techniques are required. In this paper, a novel discriminative face representation derived by the Linear Discriminant Analysis of (Multiscale) Local Phase Quantisation (LPQ) histogram is proposed. First, a (multiscale) LPQ operator is applied to the face image. Next, histograms are extracted from local regions of resultant images, and projected into an LDA space to form a discriminative regional face descriptor. These methods are implemented and tested on the problem of video based face identification using the BANCA video database. Additionally, to verify their performance, experiments on standard still image FERET and BANCA face databases showing very promising results are reported.
Bi-Modal Person Recognition on a Mobile Phone: Using Mobile Phone Data

This paper presents a novel fully automatic bi-modal, face and speaker, recognition system which runs in real-time on a mobile phone. The implemented system runs in real-time on a Nokia N900 and demonstrates the feasibility of performing both automatic face and speaker recognition on a mobile phone. We evaluate this recognition system on a novel publicly-available mobile phone database and provide a well defined evaluation protocol. This database was captured almost exclusively using mobile phones and aims to improve research into deploying biometric techniques to mobile devices. We show, on this mobile phone database, that face and speaker recognition can be performed in a mobile environment and using score fusion can improve the performance by more than 25% in terms of error rates.
Video Texture Synthesis With Multi-Frame LBP-TOP and Diffeomorphic Growth Model

Video texture synthesis is the process of providing a continuous and infinitely varying stream of frames, which plays an important role in computer vision and graphics. However, it still remains a challenging problem to generate high-quality synthesis results. Considering the two key factors that affect the synthesis performance, frame representation and blending artifacts, we improve the synthesis performance from two aspects: 1) Effective frame representation is designed to capture both the image appearance information in spatial domain and the longitudinal information in temporal domain. 2) Artifacts that degrade the synthesis quality are significantly suppressed on the basis of a diffeomorphic growth model. The proposed video texture synthesis approach has two major stages: video stitching stage and transition smoothing stage. In the first stage, a video texture synthesis model is proposed to generate an infinite video flow. To find similar frames for stitching video clips, we present a new spatial-temporal descriptor to provide an effective representation for different types of dynamic textures. In the second stage, a smoothing method is proposed to improve synthesis quality, especially in the aspect of temporal continuity. It aims to establish a diffeomorphic growth model to emulate local dynamics around stitched frames. The proposed approach is thoroughly tested on public databases and videos from the Internet, and is evaluated in both qualitative and quantitative ways.
Spotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis

Spotting micro-expressions is a primary step for continuous emotion recognition from videos. Spotting in this context refers to automatically finding the temporal locations of the face-related events from a video sequence. Rapid facial movements mainly include micro-expressions and eye blinks. However, the role of eye blinks in expressing emotions is still controversial, and often they are considered as micro-expressions as well. In this paper a simple method for automatically spotting rapid facial movements from videos is proposed. The method relies on analyzing differences in appearance-based features of sequential frames. In addition to finding the temporal locations, the system is able to provide spatial information about the movements in the face. Micro-expression spotting experiments are carried out on three datasets consisting only of spontaneous micro-expressions. Baseline micro-expression spotting results are provided for these three datasets including the publicly available CASME database. Also an example of spatial localization of the spotted rapid movements is presented.
Multiscale Local Phase Quantization for Robust Component-Based Face Recognition Using Kernel Fusion of Multiple Descriptors

Face recognition subject to uncontrolled illumination and blur is challenging. Interestingly, image degradation caused by blurring, often present in real-world imagery, has mostly been overlooked by the face recognition community. Such degradation corrupts face information and affects image alignment, which together negatively impact recognition accuracy. We propose a number of countermeasures designed to achieve system robustness to blurring. First, we propose a novel blur-robust face image descriptor based on Local Phase Quantization (LPQ) and extend it to a multiscale framework (MLPQ) to increase its effectiveness. To maximize the insensitivity to misalignment, the MLPQ descriptor is computed regionally by adopting a component-based framework. Second, the regional features are combined using kernel fusion. Third, the proposed MLPQ representation is combined with the Multiscale Local Binary Pattern (MLBP) descriptor using kernel fusion to increase insensitivity to illumination. Kernel Discriminant Analysis (KDA) of the combined features extracts discriminative information for face recognition. Last, two geometric normalizations are used to generate and combine multiple scores from different face image scales to further enhance the accuracy. The proposed approach has been comprehensively evaluated using the combined Yale and Extended Yale database B (degraded by artificially induced linear motion blur) as well as the FERET, FRGC 2.0, and LFW databases. The combined system is comparable to state-of-the-art approaches using similar system configurations. The reported work provides a new insight into the merits of various face representation and fusion methods, as well as their role in dealing with variable lighting and blur degradation.
A discriminative feature space for detecting and recognizing faces

We introduce a novel discriminative feature space which is efficient not only for face detection but also for recognition. The face representation is based on local binary patterns (LBP) and consists of encoding both local and global facial characteristics into a compact feature histogram. The proposed representation is invariant with respect to monotonic gray scale transformations and can be derived in a single scan through the image. Considering the derived feature space, a second-degree polynomial kernel SVM classifier was trained to detect frontal faces in gray scale images. Experimental results using several complex images show that the proposed approach performs favorably compared to the state-of-the-art methods. Additionally, experiments with detecting and recognizing low-resolution faces from video sequences were carried out, demonstrating that the same facial representation can be efficiently used for both detection and recognition.
View-based recognition of 3D-textured surfaces

A new method for recognizing 3D-textured surfaces is proposed. Textures are modeled with multiple histograms of micro-textons, instead of the more macroscopic textons used in earlier studies. The micro-textons are extracted with a recently proposed multiresolution local binary pattern operator. Our approach has many advantages compared to the earlier approaches and provides the leading performance in the classification of Columbia-Utrecht database (CUReT) textures imaged under different viewpoints and illumination directions. An approach for learning appearance models for view-based texture recognition using self-organization of feature distributions is also proposed.. It can be used for quickly selecting model histograms and rejecting outliers, thus providing an efficient tool for vision system training, even when the feature data has a large variability.
An Image-Based Visual Speech Animation System

An image-based visual speech animation system is presented in this paper. A video model is proposed to preserve the video dynamics of a talking face. The model represents a video sequence by a low-dimensional continuous curve embedded in a path graph and establishes a map from the curve to the image domain. When selecting video segments for synthesis, we loosen the traditional requirement of using triphone as the unit to allow segments to contain longer natural talking motion. Dense videos are sampled from the segments, concatenated, and downsampled to train a video model that enables efficient time alignment and motion smoothing for the final video synthesis. Different viseme definitions are used to investigate the impact of visemes on the video realism of the animated talking face. The system is built on a public database and tested both objectively and subjectively.
Boosting Clusters of Samples for Sequence Matching in Camera Networks

This study introduces a novel classification algorithm for learning and matching sequences in view independent object tracking. The proposed learning method uses adaptive boosting and classification trees on a wide collection (shape, pose, color, texture, etc.) of image features that constitute a model for tracked objects. The temporal dimension is taken into account by using k-mean clusters of sequence samples. Most of the utilized object descriptors have a temporal quality also. We argue that with a proper boosting approach and decent number of reasonably descriptive image features it is feasible to do view-independent sequence matching in sparse camera networks. The experiments on real-life surveillance data support this statement.
From still image to video-based face recognition: an experimental analysis

In this work, we analyze the effects of face sequence length and image quality on the performance of video-based face recognition systems which use a spatio-temporal representation instead of a still image-based one. We experiment with two different databases and consider the temporal hidden Markov model as a baseline method for the spatio-temporal representation and PCA and LDA for the image-based one. We show that the face sequence length affects the joint spatio-temporal representation more than the static-image-based methods. On the other hand, the experiments indicate that static image-based systems are more sensitive to image quality than their spatio-temporal representation-based counterpart. The second major contribution in this work is the use of an efficient method for extracting the representative frames (exemplars) from raw video. We build an appearance-based face recognition system which uses the probabilistic voting strategy to assess the efficiency of our approach.
An Experimental Comparison of Autoregressive and Fourier-Based Descriptors in 2D Shape Classification

An experimental comparison of shape classification methods based on autoregressive modeling and Fourier descriptors of closed contours is carried out. The performance is evaluated using two independent sets of data: images of letters and airplanes. Silhouette contours are extracted from non-occluded 2D objects rotated, scaled, and translated in 3D space. Several versions of both types of methods are implemented and tested systematically. The comparison clearly shows better performance of Fourier-based methods, especially for images containing noise.
Adaptive document binarization

A new method is presented for adaptive document image binarization, where the page is considered as a collection of subcomponents such as text, background and picture. The problems caused by noise, illumination and many source type related degradations are addressed. The algorithm uses document characteristics to determine (surface) attributes, often used in document segmentation. Using characteristic analysis, two new algorithms are applied to determine a local threshold for each pixel. An algorithm based on soft decision control is used for thresholding the background and picture regions. An approach utilizing local mean and variance of gray values is applied to textual regions. Tests were performed with images including different types of document components and degradations. The results show that the method adapts and performs well in each case.
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions

Dynamic texture (DT) is an extension of texture to the temporal domain. Description and recognition of DTs have attracted growing attention. In this paper, a novel approach for recognizing DTs is proposed and its simplifications and extensions to facial image analysis are also considered. First, the textures are modeled with volume local binary patterns (VLBP), which are an extension of the LBP operator widely used in ordinary texture analysis, combining motion and appearance. To make the approach computationally simple and easy to extend, only the co-occurrences of the local binary patterns on three orthogonal planes (LBP-TOP) are then considered. A block-based method is also proposed to deal with specific dynamic events such as facial expressions in which local information and its spatial locations should also be taken into account. In experiments with two DT databases, DynTex and Massachusetts Institute of Technology (MIT), both the VLBP and LBP-TOP clearly outperformed the earlier approaches. The proposed block-based method was evaluated with the Cohn-Kanade facial expression database with excellent results. The advantages of our approach include local processing, robustness to monotonic gray-scale changes, and simple computation
Local Binary Pattern Descriptors for Dynamic Texture Recognition

Dynamic texture is an extension of texture to the temporal domain. In this paper, a new method for recognizing dynamic textures is proposed. The textures are modeled with concatenated local binary patterns in three orthonormal planes. The circular neighborhoods are generalized to elliptical sampling to fit to the space-time statistics. This is an extension of the LBP approach widely used in still texture analysis, combining the motion and appearance together. Our approach has many advantages compared with the earlier approaches providing a better performance for the DynTex and MIT databases
Matching Groups of People by Covariance Descriptor

In this paper, we present a new solution to the problem of matching groups of people across multiple non-overlapping cameras. Similar to the problem of matching individuals across cameras, matching groups of people also faces challenges such as variations of illumination conditions, poses and camera parameters. Moreover, people often swap their positions while walking in a group. In this paper, we propose to use covariance descriptor in appearance matching of group images. Covariance descriptor is shown to be a discriminative descriptor which captures both appearance and statistical properties of image regions. Furthermore, it presents a natural way of combining multiple heterogeneous features together with a relatively low dimensionality. Experimental results on two different datasets demonstrate the effectiveness of the proposed method.
A Bayesian Local Binary Pattern texture descriptor

In this paper, a Bayesian LBP operator is proposed. This operator is formulated in a novel filtering, labeling and statistic (FLS) framework for texture descriptors. In the framework, the local labeling procedure, which is a part of many popular descriptors such as LBP, SIFT and VZ, can be modeled as a probability and optimization process. This enables the use of more reliable prior and likelihood information and reduces the sensitivity to noise. The BLBP operator pursues a label image, when given the filtered vector image, by maximizing the joint probability of two images under the criterion of MAP. The proposed approach is evaluated on texture retrieval schemes using entire Brodatz database. The result reveals BLBP operator¿s efficient performance and FLS framework¿s capability to in-depth analysis of the texture descriptors on a common background.
Principal appearance and motion from boosted spatiotemporal descriptors

Feature definition and selection are two important aspects in visual analysis of motion. In this paper, spatiotemporal local binary patterns computed at multiple resolutions are proposed for describing dynamic events, combining static and dynamic information from different spatiotemporal resolutions. Appearance and motion are the key components for visual analysis related to movements. AdaBoost algorithm is utilized for learning the principal appearance and motion from spatiotemporal descriptors derived from three orthogonal planes, providing important information about the locations and types of features for further analysis. In addition, learners are designed for selecting the most important features for each specific pair of different classes. The experiments carried out on diverse visual analysis tasks: facial expression recognition and visual speech recognition, show the effectiveness of the approach.
Rotation-Invariant Image and Video Description With Local Binary Pattern Features

In this paper, we propose a novel approach to compute rotation-invariant features from histograms of local noninvariant patterns. We apply this approach to both static and dynamic local binary pattern (LBP) descriptors. For static-texture description, we present LBP histogram Fourier (LBP-HF) features, and for dynamic-texture recognition, we present two rotation-invariant descriptors computed from the LBPs from three orthogonal planes (LBP-TOP) features in the spatiotemporal domain. LBP-HF is a novel rotation-invariant image descriptor computed from discrete Fourier transforms of LBP histograms. The approach can be also generalized to embed any uniform features into this framework, and combining the supplementary information, e.g., sign and magnitude components of the LBP, together can improve the description ability. Moreover, two variants of rotation-invariant descriptors are proposed to the LBP-TOP, which is an effective descriptor for dynamic-texture recognition, as shown by its recent success in different application problems, but it is not rotation invariant. In the experiments, it is shown that the LBP-HF and its extensions outperform noninvariant and earlier versions of the rotation-invariant LBP in the rotation-invariant texture classification. In experiments on two dynamic-texture databases with rotations or view variations, the proposed video features can effectively deal with rotation variations of dynamic textures (DTs). They also are robust with respect to changes in viewpoint, outperforming recent methods proposed for view-invariant recognition of DTs.
Nonparametric multichannel texture description with simple spatial operators

A multichannel approach to texture description is proposed by approximating joint occurrences of multiple features with marginal distributions, as 1D histograms, and combining similarity scores for 1D histograms into an aggregate similarity score. A stepwise feature selection algorithm is used to choose the best feature combination in a particular dimension. In classification experiments with Brodatz textures and MeasTex test suites the proposed method performs favorably compared to GLCM, Gabor and GMRF features.
Separating color and pattern information for color texture discrimination

The analysis of colored surface textures is a challenging research problem in computer vision. Current approaches to this task can be roughly divided into two categories: methods that process color and texture information separately and those that utilize multispectral texture descriptions. Motivated by recent psychophysical findings, we find the former approach quite auspicious. We propose the use of complementary color and texture measures that are combined on a higher level, and empirically demonstrate the validity of our proposition using a large set of natural color textures.
Multimodal Framework for Analyzing the Affect of a Group of People

With the advances in multimedia and the world wide web, users upload millions of images and videos everyone on social networking platforms on the Internet. From the perspective of automatic human behavior understanding, it is of interest to analyze and model the affects that are exhibited by groups of people who are participating in social events in these images. However, the analysis of the affect that is expressed by multiple people is challenging due to the varied indoor and outdoor settings. Recently, a few interesting works have investigated face-based group-level emotion recognition (GER). In this paper, we propose a multimodal framework for enhancing the affective analysis ability of GER in challenging environments. Specifically, for encoding a person's information in a group-level image, we first propose an information aggregation method for generating feature descriptions of face, upper body, and scene. Later, we revisit localized multiple kernel learning for fusing face, upper body, and scene information for GER against challenging environments. Intensive experiments are performed on two challenging group-level emotion databases (HAPPEI and GAFF) to investigate the roles of the face, upper body, scene information, and the multimodal framework. Experimental results demonstrate that the multimodal framework achieves promising performance for GER.
Edge-based method for text detection from complex document images

Detection of text from documents in which text is embedded in complex colored and textured backgrounds is a very challenging problem. In this paper, we propose a simple texture-based approach based on edge information for this task. The performance of our method is compared to that obtained by a method based on the discrete cosine transform which was recently proposed by Y. Zhong et al. (2000) for text localization in compressed digital video. In our experiments, both methods performed about equally well for small-sized text, but our method was better in the case of large-sized text. The principal advantage of our approach is that in addition to the text detection problem, the same edge representation can also be used for other image interpretation tasks.