nils-herrmann/clean_scirepeval huggingface dataset:
Variable-state latent conditional random fields for facial expression recognition and action unit detection

Automatic recognition of facial expressions of emotions, and detection of facial action units (AUs), from videos depends critically on modeling of their dynamics. These dynamics are characterized by changes in temporal phases (onset-apex-offset) and intensity of emotion/AUs, the appearance of which vary considerably among subjects, making the recognition/detection task very challenging. While state-of-the-art Latent Conditional Random Fields (LCRF) allow one to efficiently encode these dynamics via modeling of structural information (e.g., temporal consistency and ordinal constraints), their latent states are restricted to either unordered (nominal) or fully ordered (ordinal). However, such an approach is often too restrictive since, for instance, in the case of AU detection, the sequences of an active AU may better be described using ordinal latent states (corresponding to the AU intensity levels), while the sequences of this AU not being active may better be described using unordered (nominal) latent states. To this end, we propose the Variable-state LCRF model that automatically selects the optimal latent states (nominal or ordinal) for each sequence from each target class. This unsupervised adaptation of the model to individual sequence or subject contexts opens the possibility for improved model fitting and, subsequently, enhanced predictive performance. Our experiments on four public expression databases (CK+, AFEW, MMI and GEMEP-FERA) show that the proposed model consistently outperforms the state-of-the-art methods for both facial expression recognition and action unit detection from image sequences.
Transform image coding based on joint adaptation of filter banks and tree structures

Recent work on filter banks and related expansions has revealed an interesting insight: different filter bank trees can be regarded as different ways of constructing orthonormal bases for linear signal expansion. In particular, fast algorithms for finding best bases in an operational rate-distortion sense have been successfully used in image coding. Independently of this work, recent research has also explored the design of filter banks that optimize energy compaction for a single signal or a class of signals. In this paper we integrate these two different but complementary approaches to best-basis design and propose an image coder in which subband filter banks, tree structure and quantizers are chosen so as to optimize rate-distortion performance. These optimal filter banks, tree structure and quantizers represent side information. They are selected from a codebook designed from training data, using a rate-distortion criterion.
An efficient IP approach to constrained multiple face tracking and recognition

Tracking and recognition of objects, such as faces, in video is commonly accomplished in independent fashion. However, important information is contained in both problems that could be used to increase the overall recognition accuracy. We propose a unified integer program (IP) based framework for multi-object tracking and recognition in video, where the two tasks are conducted jointly, using a set of natural constraints. In the domain of multiple face recognition, pairing constraints limit the number of objects that can be labeled with the same identity while temporal constraints allow the important information about objects identities's to be used to improve tracking. Despite its appeal, the solving the IP objective can be inefficient in real-world scenarios. For this reason, we employ an approximate Generalized Assignment Problem (GAP) solution to the IP problem, which is both theoretically appealing and computationally highly efficient. We finally demonstrate that the IP and GAP methods of conducting multi-object tracking and recognition can be successfully applied to real world videos where the traditional methods of conducting tracking and recognition separately fail to produce satisfactory results.
Filling in the blanks: reconstructing microscopic crowd motion from multiple disparate noisy sensors

Tracking the movement of individuals in a crowd is an indispensable component to reconstructing crowd movement, with applications in crowd surveillance and data-driven animation. Typically, multiple sensors are distributed over wide area and often they have incomplete coverage of the area or the input introduces noise due to the tracking algorithm or hardware failure. In this paper, we propose a novel refinement method that complements existing crowd tracking solutions to reconstruct a holistic view of the microscopic movement of individuals in a crowd, from noisy tracked data with missing and even incomplete information. Central to our approach is a global optimization based trajectory estimation with modular objective functions. We empirically demonstrate the potential utility of our approach in various scenarios that are standard in crowd dynamic analysis and simulations.
A dynamic Bayesian network approach to figure tracking using learned dynamic models

The human figure exhibits complex and rich dynamic behavior that is both nonlinear and time-varying. However most work on tracking and synthesizing figure motion has employed either simple, generic dynamic models or highly specific hand-tailored ones. Recently, a broad class of learning and inference algorithms for time-series models have been successfully cast in the framework of dynamic Bayesian networks (DBNs). This paper describes a novel DBN-based switching linear dynamic system (SLDS) model and presents its application to figure motion analysis. A key feature of our approach is an approximate Viterbi inference technique for overcoming the intractability of exact inference in mixed-state DBNs. We present experimental results for learning figure dynamics from video data and show promising initial results for tracking, interpolation, synthesis, and classification using learned models.
Copula Ordinal Regression for Joint Estimation of Facial Action Unit Intensity

Joint modeling of the intensity of facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but also due to the lack of labelled target data. For this reason, majority of the methods proposed so far resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. Encoding this is expected to improve the estimation of target AU intensities, especially in the case of noisy image features, head-pose variations and imbalanced training data. To this end, we introduce a novel modeling framework, Copula Ordinal Regression (COR), that leverages the power of copula functions and CRFs, to detangle the probabilistic modeling of AU dependencies from the marginal modeling of the AU intensity. Consequently, the COR model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We show on two challenging datasets of naturalistic facial expressions that the proposed approach consistently outperforms (i) independent modeling of AU intensities, and (ii) the state-of the-art approach for the target task.
A visual computing environment for very large scale biomolecular modeling

Knowledge of the complex molecular structures of living cells is being accumulated at a tremendous rate. Key technologies enabling this success have been, high performance computing and powerful molecular graphics applications, but the technology is beginning to seriously lag behind challenges posed by the size and number of new structures and by the emerging opportunities in drug design and genetic engineering. A visual computing environment is being developed which permits interactive modeling of biopolymers by linking a 3D molecular graphics program with an efficient molecular dynamics simulation program executed on remote high-performance parallel computers. The system will be ideally suited for distributed computing environments, by utilizing both local 3D graphics facilities and the peak capacity of high-performance computers for the purpose of interactive biomolecular modeling. To create an interactive 3D environment three input methods will be explored: (1) a six degree of freedom "mouse" for controlling the space shared by the model and the user; (2) voice commands monitored through a microphone and recognized by a speech recognition interface; (3) hand gestures, detected through cameras and interpreted using computer vision techniques. Controlling 3D graphics connected to real time simulations and the use of voice with suitable language semantics, as well as hand gestures, promise great benefits for many types of problem solving environments. Our focus on structural biology takes advantage of existing sophisticated software, provides concrete objectives, defines a well-posed domain of tasks and offers a well-developed vocabulary for spoken communication.
Toward multimodal human-computer interface

Recent advances in various signal processing technologies, coupled with an explosion in the available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities: speech, vision-based gesture recognition, eye tracking, electroencephalograph, etc. Successful embodiment of these modalities into an interface has the potential of easing the HCI bottleneck that has become noticeable with the advances in computing and communication. It has also become increasingly evident that the difficulties encountered in the analysis and interpretation of individual sensing modalities may be overcome by integrating them into a multimodal human-computer interface. We examine several promising directions toward achieving multimodal HCI. We consider some of the emerging novel input modalities for HCI and the fundamental issues in integrating them at various levels, from early signal level to intermediate feature level to late decision level. We discuss the different computational approaches that may be applied at the different levels of modality integration. We also briefly review several demonstrated multimodal HCI systems and applications. Despite all the recent developments, it is clear that further research is needed for interpreting and fitting multiple sensing modalities in the context of HCI. This research can benefit from many disparate fields of study that increase our understanding of the different human communication modalities and their potential role in HCI.
Reinforcement Learning for Robust and Efficient Real-World Tracking

In this paper we present a new approach for combining several independent trackers into one robust real-time tracker. Unlike previous work that employ multiple tracking objectives used in unison, our tracker manages to determine an optimal sequence of individual trackers given the characteristics present in the video and the desire to achieve maximally efficient tracking. This allows for the selection of fast less-robust trackers when little movement is sensed, while using more robust but computationally intensive trackers in more dynamic scenes. We test this approach on the problem of real-world face tracking. Results show that this approach is a viable method for combining several independent trackers into one robust real-time tracker capable of tracking faces in varied lighting conditions, video resolutions, and with occlusions.
Pose Invariant Activity Classification for Multi-floor Indoor Localization

Smartphone based indoor localization caught massive interest of the localization community in recent years. Combining pedestrian dead reckoning obtained using the phone's inertial sensors with the Graph SLAM (Simultaneous Localization and Mapping) algorithm is one of the most effective approaches to reconstruct the entire pedestrian trajectory given a set of visited landmarks during movement. A key to Graph SLAM-based localization is the detection of reliable landmarks, which are typically identified using visual cues or via NFC tags or QR codes. Alternatively, human activity can be classified to detect organic landmarks such as visits to stairs and elevators while in movement. We provide a novel human activity classification framework that is invariant to the pose of the smartphone. Pose invariant features allow robust observation no matter how a user puts the phone in the pocket. In addition, activity classification obtained by an SVM (Support Vector Machine) is used in a Bayesian framework with an HMM (Hidden Markov Model) that improves the activity inference based on temporal smoothness. Furthermore, the HMM jointly infers activity and floor information, thus providing multi-floor indoor localization. Our experiments show that the proposed framework detects landmarks accurately and enables multi-floor indoor localization from the pocket using Graph SLAM.
Estimation of Human Figure Motion Using Robust Tracking of Articulated Layers

We propose a probabilistic method for tracking articulated objects, such as the human figure, across multiple layers in monocular image sequence. In this method, each link of a probabilistic articulated object is assigned to one individual image layer. The layered representation allows us to robustly model the pose and occlusion of object parts during its motion. Appearance of links is described in terms of learned statistics of basic image features, such as color, and geometric models of robust spatial kernels. This results in a highly efficient computational method for inference of the object’s pose. We apply this approach to tracking of the human figure in monocular video sequences. We show that the proposed method, coupled with a learned dynamic model, can lead to a robust articulated object tracker.
A Framework for Joint Estimation and Guided Annotation of Facial Action Unit Intensity

Manual annotation of facial action units (AUs) is highly tedious and time-consuming. Various methods for automatic coding of AUs have been proposed, however, their performance is still far below of that attained by expert human coders. Several attempts have been made to leverage these methods to reduce the burden of manual coding of AU activations (presence/absence). Nevertheless, this has not been exploited in the context of AU intensity coding, which is a far more difficult task. To this end, we propose an expertdriven probabilistic approach for joint modeling and estimation of AU intensities. Specifically, we introduce a Conditional Random Field model for joint estimation of the AU intensity that updates its predictions in an iterative fashion by relying on expert knowledge of human coders. We show in our experiments on two publicly available datasets of AU intensity (DISFA and FERA2015) that the AU coding process can significantly be facilitated by the proposed approach, allowing human coders to faster make decisions about target AU intensity.
Multi-output Laplacian dynamic ordinal regression for facial expression recognition and intensity estimation

Automated facial expression recognition has received increased attention over the past two decades. Existing works in the field usually do not encode either the temporal evolution or the intensity of the observed facial displays. They also fail to jointly model multidimensional (multi-class) continuous facial behaviour data; binary classifiers - one for each target basic-emotion class - are used instead. In this paper, intrinsic topology of multidimensional continuous facial affect data is first modeled by an ordinal manifold. This topology is then incorporated into the Hidden Conditional Ordinal Random Field (H-CORF) framework for dynamic ordinal regression by constraining H-CORF parameters to lie on the ordinal manifold. The resulting model attains simultaneous dynamic recognition and intensity estimation of facial expressions of multiple emotions. To the best of our knowledge, the proposed method is the first one to achieve this on both deliberate as well as spontaneous facial affect data.
Model-based motion clustering using boosted mixture modeling

Model-based clustering of motion trajectories can be posed as the problem of learning an underlying mixture density function whose components correspond to motion classes with different statistical properties. We propose a general framework for boosted modeling of mixtures of parametric densities. A density is represented as a parametric mixture of kernels where mixture components are now being added recursively, one at a time, until a best fit to data occurs and an optimal number of mixture components is selected. Optimal ML and MAP solutions to this problem are found using the functional gradient techniques. Unlike traditional mixture modeling techniques, the new method does not rely on random parameter initialization and exhaustive exploration of varying model orders (such as the number of mixture components.) The method justifies parameter estimation of new mixture components independently of that of the rest of the mixture, thus allowing tractable use of complex kernels such as HMMs or switching linear dynamic models. The relationship to traditional parametric EM-based mixture modeling algorithms is established We demonstrate the utility of the new algorithm on the problem of discovering motion sequence clusters. Our generative modeling framework has an important advantage over nonparametric approaches in that it can be used for classification as well as synthesis of the learned motion categories.
Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review

In a process for preparing phosphoric acid by contact of sulphuric acid and phosphate rock with filtration of the gypsum slurry and recycle of the rest for contact with fresh rock, a fraction of the recycle slurry is treated with sulphuric acid to convert at least some of the gypsum to calcium sulphate hemihydrate and the slurry comprising hemihydrate is returned to contact the mixture of phosphate rock, phosphoric acid and recycle gypsum slurry. The process gives an easily filtered gypsum slurry with low phosphate losses in the gypsum filter cake.
Context-Sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units

Modeling intensity of facial action units from spontaneously displayed facial expressions is challenging mainly because of high variability in subject-specific facial expressiveness, head-movements, illumination changes, etc. These factors make the target problem highly context-sensitive. However, existing methods usually ignore this context-sensitivity of the target problem. We propose a novel Conditional Ordinal Random Field (CORF) model for context-sensitive modeling of the facial action unit intensity, where the W5+ (who, when, what, where, why and how) definition of the context is used. While the proposed model is general enough to handle all six context questions, in this paper we focus on the context questions: who (the observed subject), how (the changes in facial expressions), and when (the timing of facial expressions and their intensity). The context questions who and howare modeled by means of the newly introduced context-dependent covariate effects, and the context question when is modeled in terms of temporal correlation between the ordinal outputs, i.e., intensity levels of action units. We also introduce a weighted softmax-margin learning of CRFs from data with skewed distribution of the intensity levels, which is commonly encountered in spontaneous facial data. The proposed model is evaluated on intensity estimation of pain and facial action units using two recently published datasets (UNBC Shoulder Pain and DISFA) of spontaneously displayed facial expressions. Our experiments show that the proposed model performs significantly better on the target tasks compared to the state-of-the-art approaches. Furthermore, compared to traditional learning of CRFs, we show that the proposed weighted learning results in more robust parameter estimation from the imbalanced intensity data.
Integrative Protein Function Transfer Using Factor Graphs and Heterogeneous Data Sources

We propose a novel approach for predicting protein functions of an organism by coupling sequence homology and PPI data between two (or more) species with multi-functional Gene Ontology information into a single computational model. Instead of using a network of one organism in isolation, we join networks of different species by inter-species sequence homology links of sufficient similarity. As a consequence, the knowledge of a protein's function is acquired not only from one species' network alone, but also through homologous links to the networks of different species. We apply our method to two largest protein networks, Yeast (Saccharomyces cerevisiae) and Fly (Drosophila melanogaster). Our joint Fly-Yeast network displays statistically significant improvements in precision, accuracy, and false positive rate over networks that consider either of the sources in isolation, while retaining the computational efficiency of the simpler models.
Discovering characteristic landmarks on ancient coins using convolutional networks

We propose a novel method to find characteristic landmarks and recognize ancient Roman imperial coins using deep convolutional neural networks (CNNs) combined with expert-designed domain hierarchies. We first propose a new framework to recognize the Roman coin which exploits the hierarchical knowledge structure embedded in the coin domain, which we combine with the CNN-based category classifiers. We next formulate an optimization problem to discover class-specific salient coin regions. Analysis of discovered salient regions confirms that they are largely consistent with human expert annotations. Experimental results show that the proposed framework is able to effectively recognize the ancient Roman coins as well as successfully identify landmarks in a general fine-grained classification problem. For this research, we have collected a new Roman coin dataset where all coins are annotated and consist of obverse (head) and reverse (tail) images.
Gestural interface to a visual computing environment for molecular biologists

In recent years there has been tremendous progress in 3-D, immersive display and virtual reality (VR) technologies. Scientific visualization af data is one of many applications that has benefited from this progress. To fully exploit the potential of these applications in the new environment there is a need for "natural" interfaces that allow the manipulation of such displays without burdensome attachments. This paper describes the use of visual hand gesture analysis enhanced with speech recognition for developing a bimodal gesture/speech interface for controlling a 3-D display. The interface augments an existing application, VMD, which is a VR visual computing environment for molecular biologists. The free hand gestures are used for manipulating the 3-D graphical display together with a set of speech commands. We concentrate on the visual gesture analysis techniques used in developing this interface. The dual modality of gesture/speech is found to greatly aid the interaction capability.