nils-herrmann/clean_scirepeval huggingface dataset:
Multimodal group action clustering in meetings

We address the problem of clustering multimodal group actions in meetings using a two-layer HMM framework. Meetings are structured as sequences of group actions. Our approach aims at creating one cluster for each group action, where the number of group actions and the action boundaries are unknown a priori. In our framework, the first layer models typical actions of individuals in meetings using supervised HMM learning and low-level audio-visual features. A number of options that explicitly model certain aspects of the data (e.g., asynchrony) were considered. The second layer models the group actions using unsupervised HMM learning. The two layers are linked by a set of probability-based features produced by the individual action layer as input to the group action layer. The methodology was assessed on a set of multimodal turn-taking group actions, using a public five-hour meeting corpus. The results show that the use of multiple modalities and the layered framework are advantageous, compared to various baseline methods.
Statistical transformations of frontal models for non-frontal face verification

In the framework of a face verification system using local features and a Gaussian mixture model based classifier, we address the problem of non-frontal face verification (when only a single (frontal) training image is available) by extending each client's frontal face model with artificially synthesized models for non-frontal views. Furthermore, we propose the maximum likelihood shift (MLS) synthesis technique and compare its performance against a maximum likelihood linear regression (MLLR) based technique (originally developed for adapting speech recognition systems) and the recently proposed "difference between two universal background models" (UBMdiff) technique. All techniques rely on prior information and learn how a generic face model for the frontal view is related to generic models at non-frontal views. Experiments on the FERET database suggest that that the proposed MLS technique is more suitable than MLLR (due to a lower number of free parameters) and UBMdiff (due to lack of heuristics). The results further suggest that extending frontal models considerably reduces errors.
Offline recognition of unconstrained handwritten texts using HMMs and statistical language models

This paper presents a system for the offline recognition of large vocabulary unconstrained handwritten texts. The only assumption made about the data is that it is written in English. This allows the application of statistical language models in order to improve the performance of our system. Several experiments have been performed using both single and multiple writer data. Lexica of variable size (from 10,000 to 50,000 words) have been used. The use of language models is shown to improve the accuracy of the system (when the lexicon contains 50,000 words, the error rate is reduced by /spl sim/50 percent for single writer data and by /spl sim/25 percent for multiple writer data). Our approach is described in detail and compared with other methods presented in the literature to deal with the same problem. An experimental setup to correctly deal with unconstrained text recognition is proposed.
Sequence Classification with Input-Output Hidden Markov Models

We present a training and testing method for Input-Output Hidden Markov Model that is particularly suited for classification of sequences in which class information accumulates over time. We discuss two such cases: the discrimination of mental tasks from sequences of EEG features, common in Brain Computer Interface research, and phoneme classification from sequences of acoustic features for speech recognition. The objective function is modified so that training focuses on the improvement of classification accuracy. For both tasks the algorithm performs significantly better than the alternative solution proposed in the literature, specifically designed for other types of sequences.
A Study of the Effects of Score Normalisation Prior to Fusion in Biometric Authentication Tasks

Although the subject of fusion is well studied, the effects of normalisation prior to fusion are somewhat less well investigated. In this study, four normalisation techniques and six commonly used fusion classifiers were examined. Based on 24 (fusion classifiers) as a result of pairing the normalisation techniques and classifiers applied on 32 fusion data sets, 4x6x32 means 768 fusion experiments were carried out on the XM2VTS score-level fusion benchmark database, it can be concluded that trainable fusion classifiers are potentially useful. It is found that some classifiers are very sensitive (in terms of Half Total Error Rate) to normalisation techniques such as Weighted sum with weights optimised using Fisher-ratio and Decision Template. The mean fusion operator and user-specific linear weight combination are relative less sensitive. It is also found that Support Vector Machines and Gaussian Mixture Model are the least sensitive to different normalisation techniques, while achieving the best generalisation performance. For these two techniques, score normalisation is unnecessary prior to fusion.
Evaluation of formant-like features on an automatic vowel classification task.

Numerous attempts have been made to find low-dimensional, formant-related representations of speech signals that are suitable for automatic speech recognition. However, it is often not known how these features behave in comparison with true formants. The purpose of this study was to compare two sets of automatically extracted formant-like features, i.e., robust formants and HMM2 features, to hand-labeled formants. The robust formant features were derived by means of the split Levinson algorithm while the HMM2 features correspond to the frequency segmentation of speech signals obtained by two-dimensional hidden Markov models. Mel-frequency cepstral coefficients (MFCCs) were also included in the investigation as an example of state-of-the-art automatic speech recognition features. The feature sets were compared in terms of their performance on a vowel classification task. The speech data and hand-labeled formants that were used in this study are a subset of the American English vowels database presented in Hillenbrand et al. . Classification performance was measured on the original, clean data and in noisy acoustic conditions. When using clean data, the classification performance of the formant-like features compared very well to the performance of the hand-labeled formants in a gender-dependent experiment, but was inferior to the hand-labeled formants in a gender-independent experiment. The results that were obtained in noisy acoustic conditions indicated that the formant-like features used in this study are not inherently noise robust. For clean and noisy data as well as for the gender-dependent and gender-independent experiments the MFCCs achieved the same or superior results as the formant features, but at the price of a much higher feature dimensionality.
USER AUTHENTICATION VIA ADAPTED GENERATIVE MODELS OF FACE IMAGES

It has been previously demonstrated that systems based on local features and relatively complex generative models, namely 1D Hidden Markov Models (HMMs) and pseudo-2D HMMs, are suitable for face recognition. Recently, a simpler generative model, namely the Gaussian Mixture Model (GMM), was also shown to perform well. In most of the previous literature related to these models, the experiments were performed with controlled images (perfect face localization, controlled lighting, background, pose, expression, etc.); however, for most secure authentication applications, the system has to be robust to more challenging conditions. In this article we evaluate the performance, robustness and complexity of GMM and HMM based approaches, using both perfect and automatic face localization, on the relatively difficult BANCA database. We also evaluate different training techniques for both GMM and HMM based systems; we show that the traditionally used Maximum Likelihood (ML) training approach has problems estimating robust model parameters when there is only a few training images available; we propose to tackle this problem through the use of Maximum a Posteriori (MAP) training, where the lack of data problem can be effectively circumvented. We show that models estimated with MAP are significantly more robust and are able to generalize to adverse conditions present in the BANCA database. A positive side-effect of MAP based training is that the number of client specific parameters is less than half of the number required for ML based training. We also propose to extend the GMM approach through the use of local features with embedded positional information (hence increasing performance without sacrificing the low complexity of the approach); we show that the proposed extended GMM approach obtains performance comparable to the 1D HMM approach, while being more robust and considerably less complex. We also show that while the pseudo-2D HMM approach has overall the best performance, it requires relatively long times for training and authentication.
Noise-robust multi-stream fusion for text-independent speaker authentication

Multi-stream approaches have proven to be very successful in speech recognition tasks and to a certain extent in speaker authentication tasks. In this study we propose a noise-robust multi-stream text-independent speaker authentication system. This system has two steps: first train the stream experts under clean conditions and then train the combination mechanism to merge the scores of the stream experts under both clean and noisy conditions. The idea here is to take advantage of the rather predictable reliability and diversity of streams under different conditions. Hence, noise-robustness is mainly due to the combination mechanism. This two-step approach offers several practical advantages: the stream experts can be trained in parallel (e.g., by using several machines); heterogeneous types of features can be used and the resultant system can be robust to different noise types (wide bands or narrow bands) as compared to sub-streams. An important finding is that a trade-off is often necessary between the overall good performance under all conditions (clean and noisy) and good performance under clean conditions. To reconcile this trade-off, we propose to give more emphasis or prior to clean conditions, thus, resulting in a combination mechanism that does not deteriorate under clean conditions (as compared to the best stream) yet is robust to noisy conditions.
STATISTICAL TRANSFORMATION TECHNIQUES FOR FACE VERIFICATION USING FACES ROTATED IN DEPTH

In the framework of a {B}ayesian classifier based on mixtures of gaussians, we address the problem of non-frontal face verification (when only a single (frontal) training image is available) by extending each frontal face model with artificially synthesized models for non-frontal views. {T}he synthesis methods are based on several implementations of {M}aximum {L}ikelihood {L}inear {R}egression ({MLLR}), as well as standard multi-variate linear regression ({LinReg}). All synthesis techniques rely on prior information and learn how face models for the frontal view are related to face models for non-frontal views. {T}he synthesis and extension approach is evaluated by applying it to two face verification systems: {PCA} based (holistic features) and {DCTmod2} based (local features). Experiments on the {FERET} database suggest that for the {PCA} based system, the {LinReg} based technique is more suited than the {MLLR} based techniques; for the {DCTmod2} based system, the results show that synthesis via a new {MLLR} implementation obtains better performance than synthesis based on traditional {MLLR}. {T}he results further suggest that extending frontal models considerably reduces errors. It is also shown that the {DCTmod2} based system is less affected by out-of-plane rotations than the {PCA} based system; this can be attributed to the local feature representation of the face, and, due to the classifier based on mixtures of gaussians, the lack of constraints on spatial relations between face parts, allowing for movement of facial areas.
An Investigation of F-ratio Client-Dependent Normalisation on Biometric Authentication Tasks

This study investigates a new client-dependent normalisation to improve biometric authentication systems. There exists many client-de-pendent score normalisation techniques applied to speaker authentication, such as Z-Norm, D-Norm and T-Norm. Such normalisation is intended to adjust the variation across different client models. We propose ``F-ratio'' normalisation, or F-Norm, applied to face and speaker authentication systems. This normalisation requires only that as few as two client-dependent accesses are available (the more the better). Different from previous normalisation techniques, F-Norm considers the client and impostor distributions simultaneously. We show that F-ratio is a natural choice because it is directly associated to Equal Error Rate. It has the effect of centering the client and impostor distributions such that a global threshold can be easily found. Another difference is that F-Norm actually ``interpolates'' between client-independent and client-dependent information by introducing a mixture parameter. This parameter can be optimised to maximise the class dispersion (the degree of separability between client and impostor distributions) while the aforementioned normalisation techniques cannot. unimodal experiments XM2VTS multimodal database show that such normalisation is advantageous over Z-Norm, client-dependent threshold normalisation or no normalisation.
Use of modular architectures for time series prediction

Recently, a lot of papers have been published in the field of time series prediction using connectionist models. Nevertheless we think that one of the major problem with is rarely treated in the literature is related to the choice of input parameters (embedding dimension and delay). In this paper, we propose two modular approaches to this problem and apply them to a sunspot-related time series. Experimental results are then compared to a single multi-layer perceptron in order to estimate performances of these models.
Towards Predicting Optimal Fusion Candidates: A Case Study on Biometric Authentication Tasks

Combining multiple information sources, typically from several data streams is a very promising approach, both in experiments and to some extend in various real-life applications. However, combining too many systems (base-experts) will also increase both hardware and computation costs. One way to selecting a subset of optimal base-experts out of N is to carry out the experiments explicitly. There are 2N–1 possible combinations. In this paper, we propose an analytical solution to this task when weighted sum fusion mechanism is used. The proposed approach is at least valid in the domain of person authentication. It has a complexity that is additive between the number of examples and the number of possible combinations while the conventional approach, using brute-force experimenting, is multiplicative between these two terms. Hence, our approach will scale better with large fusion problems. Experiments on the BANCA multi-modal database verified our approach. While we will consider here fusion in the context of identity verification via biometrics, or simply biometric authentication, it can also have an important impact in meetings because this a priori information can assist in retrieving highlights in meeting analysis as in “who said what”. Furthermore, automatic meeting analysis also requires many systems working together and involves possibly many audio-visual media streams. Development in fusion of identity verification will provide insights into how fusion in meetings can be done. The ability to predict fusion performance is another important step towards understanding the fusion problem.
Pump-priming PASCAL proposal : Large Margin Algorithms and Kernel Methods for Speech Applications

Research on large margin algorithms in conjunctions with kernels methods has been both exciting and successful. While there have been quite a few preliminary successes in applying kernel methods for speech applications, most the research efforts have focused on non-temporal problems such as text classification and optical character recognition (OCR). We propose to design, analyze, and implement learning algorithms and kernels for hierarchical-temporal speech utterances. Our first and primary end-goal is to build and test thoroughly a full-blown speech phoneme classifier that will be trained on millions of examples and will achieve the best results in this domain. We also plan to apply and test the resulting algorithms and kernels to other supervised problems in spoken language such as language identification, word spotting from phonemes, and speaker verification. 1  We propose an algorithmic research framework for supervised speech analysis problems that builds on recent advances in kernel methods and large margin classifiers. Specific applications include (but are not limited to) speech phoneme classification, spoken language identification, closed vocabulary word recognition/spotting, and speaker verification. Recent work on large margin methods such as support vector machines and boosting algorithms has shown to be effective in many decision tasks. While there has been some work on large margin methods for complex decision problems, most of the research focus still revolves around the less complex structures. Speech signals however exhibit a complex temporal structure in both the input space (acoustic signal) and the target space (e.g. phonetic transcription and word transcription). For instance, speech signals exhibit multi-scale temporal behavior and the set of speech phonemes is organized in an hierarchical structure. The current state-of-the-art models to handle such speech signals are based on generative models that capture some temporal dependencies such as Hidden Markov Models (HMMs).  and the many references therein.) Numerous reasons have underscored HMMs as the tool of choice in speech signal processing, such as their ability to constrain the space of possible solutions to legal and probable sequences of phonemes/words, through the design of speech-specific Markovian state topologies, as well as their insensitivity to large unbalanced datasets due to their generative nature.
Modeling Individual and Group Actions in Meetings: A Two-Layer HMM Framework

We address the problem of recognizing sequences of human interaction patterns in meetings, with the goal of structuring them in semantic terms. The investigated patterns are inherently group-based (defined by the individual activities of meeting participants, and their interplay), and multimodal (as captured by cameras and microphones). By defining a proper set of individual actions, group actions can be modeled as a two-layer process, one that models basic individual activities from low-level audio-visual features, and another one that models the interactions. We propose a two-layer Hidden Markov Model (HMM) framework that implements such concept in a principled manner, and that has advantages over previous works. First, by decomposing the problem hierarchically, learning is performed on low-dimensional observation spaces, which results in simpler models. Second, our framework is easier to interpret, as both individual and group actions have a clear meaning, and thus easier to improve. Third, different HMM models can be used in each layer, to better reflect the nature of each subproblem. Our framework is general and extensible, and we illustrate it with a set of eight group actions, using a public five-hour meeting corpus. Experiments and comparison with a single-layer HMM baseline system show its validity.
Links between perceptrons, MLPs and SVMs

We propose to study links between three important classification algorithms: Perceptrons, Multi-Layer Perceptrons (MLPs) and Support Vector Machines (SVMs). We first study ways to control the capacity of Perceptrons (mainly regularization parameters and early stopping), using the margin idea introduced with SVMs. After showing that under simple conditions a Perceptron is equivalent to an SVM, we show it can be computationally expensive in time to train an SVM (and thus a Perceptron) with stochastic gradient descent, mainly because of the margin maximization term in the cost function. We then show that if we remove this margin maximization term, the learning rate or the use of early stopping can still control the margin. These ideas are extended afterward to the case of MLPs. Moreover, under some assumptions it also appears that MLPs are a kind of mixture of SVMs, maximizing the margin in the hidden layer space. Finally, we present a very simple MLP based on the previous findings, which yields better performances in generalization and speed than the other models.
A Bayesian Framework for Score Normalization Techniques Applied to Text Independent Speaker Verification

The purpose of this paper is to unify several of the state-of-the-art score normalization techniques applied to text-independent speaker verification systems. We propose a new Bayesian framework for this purpose. The two well-known Zand T-normalization techniques can be easily interpreted in this framework as different ways to estimate score distributions. This is useful as it helps to understand the various assumptions behind these well-known score normalization techniques, and opens the door for yet more complex solutions. Finally, some experiments on the Switchboard database are performed in order to illustrate the validity of the new proposed framework.
Multi Channel Sequence Processing

This paper summarizes some of the current research challenges arising from multi-channel sequence processing. Indeed, multiple real life applications involve simultaneous recording and analysis of multiple information sources, which may be asynchronous, have different frame rates, exhibit different stationarity properties, and carry complementary (or correlated) information. Some of these problems can already be tackled by one of the many statistical approaches towards sequence modeling. However, several challenging research issues are still open, such as taking into account asynchrony and correlation between several feature streams, or handling the underlying growing complexity. In this framework, we discuss here two novel approaches, which recently started to be investigated with success in the context of large multimodal problems. These include the asynchronous HMM, providing a principled approach towards the processing of multiple feature streams, and the layered HMM approach, providing a good formalism for decomposing large and complex (multi-stream) problems into layered architectures. As briefly reported here, combination of these two approaches yielded successful results on several multi-channel tasks, ranging from audio-visual speech recognition to automatic meeting analysis.
Towards Computer Understanding of Human Interactions

People meet in order to interact – disseminating information, making decisions, and creating new ideas. Automatic analysis of meetings is therefore important from two points of view: extracting the information they contain, and understanding human interaction processes. Based on this view, this article presents an approach in which relevant information content of a meeting is identified from a variety of audio and visual sensor inputs and statistical models of interacting people. We present a framework for computer observation and understanding of interacting people, and discuss particular tasks within this framework, issues in the meeting context, and particular algorithms that we have adopted. We also comment on current developments and the future challenges in automatic meeting analysis.