nils-herrmann/clean_scirepeval huggingface dataset:
Automatic Counting of Interacting People by using a Single Uncalibrated Camera

Automatic counting of people, entering or exiting a region of interest, is very important for both business and security applications. This paper introduces an automatic and robust people counting system which can count multiple people who interact in the region of interest, by using only one camera. Two-level hierarchical tracking is employed. For cases not involving merges or splits, a fast blob tracking method is used. In order to deal with interactions among people in a more thorough and reliable way, the system uses the mean shift tracking algorithm. Using the first-level blob tracker in general, and employing the mean shift tracking only in the case of merges and splits saves power and makes the system computationally efficient. The system setup parameter can be automatically learned in a new environment from a 3 to 5 minute-video with people going in or out of the target region one at a time. With a 2 GHz Pentium machine, the system runs at about 33 fps on 320times240 images without code optimization. Average accuracy rates of 98.5% and 95% are achieved on videos with normal traffic flow and videos with many cases of merges and splits, respectively
Comparative study of coarse head pose estimation

For many practical applications, it is sufficient to estimate a coarse head pose to infer gaze direction. Indeed, for any application in which the camera is situated unobtrusively in an overhead corner, the only possible inference is coarse pose because of the limitations of the quality and resolution of the incoming data. However, the vast majority of research in head pose estimation deals with tracking full rigid body motion (6 degrees of freedom) for a limited range of motion (typically /spl plusmn/45 degrees out-of-plane) and relatively high resolution data (usually 64/spl times/64 or more). We review the smaller body of research on coarse pose estimation. This work involves image-based learning, estimation of a wide range of pose, and is capable of real-time performance for low-resolution imagery. We evaluate two coarse pose estimation schemes, based on (1) a probabilistic model approach and (2) a neural network approach. We compare the results of the two techniques for varying resolution, head localization accuracy and required pose accuracy. We conclude with details for the implementation specifications for resolution and localization accuracy depending on system accuracy requirements.
Scene Text Recognition in Mobile Applications by Character Descriptor and Structure Configuration

Text characters and strings in natural scene can provide valuable information for many applications. Extracting text directly from natural scene images or videos is a challenging task because of diverse text patterns and variant background interferences. This paper proposes a method of scene text recognition from detected text regions. In text detection, our previously proposed algorithms are applied to obtain text regions from scene image. First, we design a discriminative character descriptor by combining several state-of-the-art feature detectors and descriptors. Second, we model character structure at each character class by designing stroke configuration maps. Our algorithm design is compatible with the application of scene text extraction in smart mobile devices. An Android-based demo system is developed to show the effectiveness of our proposed method on scene text information extraction from nearby objects. The demo system also provides us some insight into algorithm design and performance improvement of scene text extraction. The evaluation results on benchmark data sets demonstrate that our proposed scheme of text recognition is comparable with the best existing methods.
Recognizing expressions from face and body gesture by temporal normalized motion and appearance features

Recently, recognizing affects from both face and body gestures attracts more attentions. However, it still lacks of efficient and effective features to describe the dynamics of face and gestures for real-time automatic affect recognition. In this paper, we propose a novel approach, which combines both MHI-HOG and Image-HOG through temporal normalization method, to describe the dynamics of face and body gestures for affect recognition. The MHI-HOG stands for Histogram of Oriented Gradients (HOG) on the Motion History Image (MHI). It captures motion direction of an interest point as an expression evolves over the time. The Image-HOG captures the appearance information of the corresponding interesting point. Combination of MHI-HOG and Image-HOG can effectively represent both local motion and appearance information of face and body gesture for affect recognition. The temporal normalization method explicitly solves the time resolution issue in the video-based affect recognition. Experimental results demonstrate promising performance as compared with the state of the art. We also show that expression recognition with temporal dynamics outperforms frame-based recognition.
Pyramid of Spatial Relatons for Scene-Level Land Use Classification

Local feature with bag-of-words (BOW) representation has become one of the most popular approaches in object classification and image retrieval applications in the computer vision community. The recent efforts in the remote sensing community have demonstrated that the BOW approach can also effectively apply to geographic images for the applications of classification and retrieval. However, the BOW representation discards spatial information, which is critical for the remotely sensed land use classification. Several algorithms have incorporated spatial information into the BOW representation by hard encoding coordinates of local features. Such rigid spatial encoding is not robust to translation and rotation variations, which are common characteristics of geographic images. To effectively incorporate spatial information into the BOW model for the land use classification, we propose a pyramid-of-spatial-relatons (PSR) model to capture both absolute and relative spatial relationships of local features. Unlike the conventional cooccurrence approach to describe pairwise spatial relationships between local features, the PSR model employs a novel concept of spatial relation to describe relative spatial relationship of a group of local features. As the result, the storage cost of the PSR model only linearly increases with the visual word codebook size instead of the quadratic relationship as in the cooccurrence approach. The PSR model is robust to translation and rotation variations and demonstrates excellent performance for the application of remotely sensed land use classification. On the Land Use and Land Cover image database, the PSR achieves 8% higher in the classification accuracy than the state of the art. If using only gray images, it outperforms the state of the art by more than 11%.
Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity

Previous work suggests that Gabor-wavelet-based methods can achieve high sensitivity and specificity for emotion-specified expressions (e.g., happy, sad) and single action units (AUs) of the Facial Action Coding System (FACS). This paper evaluates a Gabor-wavelet-based method to recognize AUs in image sequences of increasing complexity. A recognition rate of 83% is obtained for three single AUs when image sequences contain homogeneous subjects and are without observable head motion. The accuracy of AU recognition decreases to 32% when the number of AUs increases to nine and the image sequences consist of AU combinations, head motion, and non-homogeneous subjects. For comparison, an average recognition rate of 87.6% is achieved for the geometry-feature-based method. The best recognition is a rate of 92.7% obtained by combining Gabor wavelets and geometry features.
An end-to-end eChronicling System for Mobile Human Surveillance

Rapid advances in mobile computing devices and sensor technologies are enabling the capture of unprecedented volumes of data by individuals involved in field operations in a variety of applications. As capture becomes ever more rich and pervasive the biggest challenge is in developing information processing and representation tools that maximize the utility of the captured multi-sensory data. The right tools hold the promise of converting captured data into actionable intelligence resulting in improved memory, enhanced situational understanding, and more efficient execution of operations. These tools need to be at least as rich and diverse as the sensors used for capture, and need to be unified within an effective system architecture. This paper presents our initial attempt at such a system and architecture that combines several emerging sensor technologies, state of the art analytic engines, and multi-dimensional navigation tools, into an end-to-end electronic chronicling solution for mobile surveillance by humans.
Computer vision-based mathematics learning enhancement for children with visual impairments

Children with visual impairments face disproportionate challenges in learning mathematics. In this paper, we explore how to apply computer vision techniques to enhance mathematics learning by fusion of multisensory information for children with visually impairments. The multisensory information includes visual and audio information. Our research focuses on the mathematics education for children with visual impairments in early ages (Kindergarten - Grade 3) by using arithmetic racks. A computer vision-based algorithm is developed to detect the numbers and positions of arithmetic rack beads and the detection results are then provided to students with visual impairments as speech guidance. Preliminary results demonstrate the effectiveness and efficiency of the proposed method for detecting the number, color, position, and moving directions of the arithmetic rack beads under different situations.
Visual speech learning from an e-tutor via dynamic lip movement-based video segmentation and comparison

This paper is motivated by the difficulties that deaf students encounter when learning speechreading and speaking; the skills that enable them to effectively communicate with hearing people. In this paper, we propose a speech learning prototype system based on the analysis and comparison of lip movements of an E-Tutor and those of a deaf student in a video. The main framework of our proposed system can be divided into two stages: lip movement segmentation and speech comparison. Lip movement segmentation fragments the frames of each word from a visual speech video sequence by analyzing the movement and shape of lips. Comparison determines whether a student is producing a correct word utterance or not, this is accomplished by comparing the lip shape and movements according to that of an e-tutor. To model lip movement, we compute two dynamic-based features by using a lip tracking method, which employs landmark points to define lip shapes. We utilize these dynamic features along with Space-Time Interest Points (STIP) to capture lip movements. In order to evaluate the effectiveness of our proposed methods, we collect a visual speech learning dataset consisting of 220 videos and 1100 word utterances. The proposed system achieves promising performances in both visual speech segmentation and visual speech comparison on this dataset.
Robust door detection in unfamiliar environments by combining edge and corner features

Camera-based indoor navigation and wayfinding can assist blind people to independently access unfamiliar buildings. In indoor environments, doors are significant landmarks and door detection plays important roles for navigation and wayfinding. Most existing algorithms of door detection are limited to work for familiar environments with restricted features without taking account of the diversity and variance of doors in different environments. In this paper, we present an image-based door detection algorithm that utilizes the general and stable features of doors - edges and corners. Furthermore, we develop a general geometric model to characterize the door shape by combining edge and corner features without a training process. To validate the robustness and generalizability of our method, we collected a large dataset of door images from a variety of environments. The proposed algorithm achieves 91.7% true positive rate with a low false positive rate of 2.9%. The results demonstrate that our door detection method is generic and robust to different environments with variations of color, texture, occlusions, illumination, scales, and viewpoints.
Subject Adaptive Affection Recognition via Sparse Reconstruction

Multimedia affection recognition from facial expressions and body gestures in RGB-D video sequences is a new research area. However, the large variance among different subjects, especially in facial expression, has made the problem more difficult. To address this issue, we propose a novel multimedia subject adaptive affection recognition framework via a 2-layer sparse representation. There are two main contributions in our framework. In the subjective adaption stage, an iterative subject selection algorithm is proposed to select most subject-related instances instead of using the whole training set. In the inference stage, a joint decision is made with confident reconstruction prior to composite information from facial expressions and body gestures. We also collect a new RGB-D dataset for affection recognition with large subjective variance. Experimental results demonstrate that the proposed affection recognition framework can increase the discriminative power especially for facial expressions. Joint recognition strategy is also demonstrated that it can utilize complementary information in both models so that to reach better recognition rate.
Automatic video description generation via LSTM with joint two-stream encoding

In this paper, we propose a novel two-stream framework based on combinational deep neural networks. The framework is mainly composed of two components: one is a parallel two-stream encoding component which learns video encoding from multiple sources using 3D convolutional neural networks and the other is a long-short-term-memory (LSTM)-based decoding language model which transfers the input encoded video representations to text descriptions. The merits of our proposed model are: 1) It extracts both temporal and spatial features by exploring the usage of 3D convolutional networks on both raw RGB frames and motion history images. 2) Our model can dynamically tune the weights of different feature channels since the network is trained end-to-end from learning combinational encoding of multiple features to LSTM-based language model. Our model is evaluated on three public video description datasets: one YouTube clips dataset (Microsoft Video Description Corpus) and two large movie description datasets (MPII Corpus and Montreal Video Annotation Dataset) and achieves comparable or better performance than the state-of-the-art approaches in video caption generation.
Evaluating effectiveness of Latent Dirichlet Allocation model for scene classification

Scene classification from images is a challenging problem in computer vision due to its significant variability of scale, illumination, and view. Recently, Latent Dirichlet Allocation (LDA) model has grown popular in computer vision field, especially in scene labeling and classification. However, the effectiveness of the LDA model for the scene classification has not yet been addressed thoroughly. Especially, there is little experimental evaluation on the model's performance for different types of features. Fusion of multiple types of features is usually necessary in the scene classification due to the complexity of scene images. In this paper, we investigate the effectiveness of the LDA model in scene classification by using 7 types of features (i.e. uniform grid based interest points, Harris corner based interest points, scale invariant feature transform (SIFT), texture, shape, color, and location) and their various combinations. Furthermore, we compare the performance of the LDA model with Support Vector Machine (SVM) classifier. All experiments are performed on the UIUC Sport Scene database. The experiments demonstrate that the performance of the LDA model 1) is significantly lower than the SVM classifier for the scene classification over different types of features; and 2) decreases by fusing multiple features while improvement shown in SVM classifier.
Robust and effective component-based banknote recognition by SURF features

Camera-based computer vision technology is able to assist visually impaired people to automatically recognize banknotes. A good banknote recognition algorithm for blind or visually impaired people should have the following features: 1) 100% accuracy, and 2) robustness to various conditions in different environments and occlusions. Most existing algorithms of banknote recognition are limited to work for restricted conditions. In this paper we propose a component-based framework for banknote recognition by using Speeded Up Robust Features (SURF). The component-based framework is effective in collecting more class-specific information and robust in dealing with partial occlusion and viewpoint changes. Furthermore, the evaluation of SURF demonstrates its effectiveness in handling background noise, image rotation, scale, and illumination changes. To authenticate the robustness and generalizability of the proposed approach, we have collected a large dataset of banknotes from a variety of conditions including occlusion, cluttered background, rotation, and changes of illumination, scaling, and viewpoints. The proposed algorithm achieves 100% recognition rate on our challenging dataset.
Recognizing upper face action units for facial expression analysis

We develop an automatic system to analyze subtle changes in upper face expressions based on both permanent facial features (brows, eyes, mouth) and transient facial features (deepening of facial furrows) in a nearly frontal image sequence. Our system recognizes fine-grained changes in facial expression based on Facial Action Coding System (FACS) action units (AUs). Multi-state facial component models are proposed for tracting and modeling different facial features, including eyes, brews, cheeks, and furrows. Then we convert the results of tracking to detailed parametric descriptions of the facial features. These feature parameters are fed to a neural network which recognizes 7 upper face action units. A recognition rate of 95% is obtained for the test data that include both single action units and AU combinations.
Facial image analysis using local feature adaptation prior to learning

Many facial image analysis methods rely on learning-based techniques such as Adaboost or SVMs to project classifiers based on the selection of local image filters (e.g., Haar and Gabor filters) from large sets of training data. In general, the learning process consists of selecting discriminative image filters from a large feature pool that contains filters uniformly sampled from the parameter space. In this paper, we argue that we are able to improve these methods by incorporating a local feature adaptation technique prior to learning, which generates a more compact and meaningful pool of image filters, consequently reducing both learning and detection/recognition computational costs, while at the same time improving accuracies. In the first stage of our approach, local feature adaptation is carried out by a nonlinear optimization method that determines image filter parameters (such as position, orientation and scale) in order to match the geometrical structure of each training sample. In the second stage, Adaboost feature selection technique is applied to the adapted feature pool to obtain the final set of discriminative local image filters. We demonstrate the effectiveness and efficiency of the proposed framework in the face detection domain. In the experiments, we have applied our method using a pool of wavelet features, including Haar and Gabor filters. The results showed that with local feature adaptation, significant improvements in terms of detection accuracy and computational cost reduction are achieved over learning based on the same features sampled uniformly from the parameter space.
Hidden Markov modeling of human normal gait using laser range finder for a mobility assistance robot

For an effective intelligent active mobility assistance robot, the walking pattern of a patient or an elderly person has to be analyzed precisely. A well-known fact is that the walking patterns are gaits, that is, cyclic patterns with several consecutive phases. These cyclic motions can be modeled using the consecutive gait phases. In this paper, we present a completely non-invasive framework for analyzing a normal human walking gait pattern. Our framework utilizes a laser range finder sensor to collect the data, a combination of filters to preprocess these data, and an appropriately synthesized Hidden Markov Model (HMM) for state estimation, and recognition of the gait data. We demonstrate the applicability of this setup using real data, collected from an ensemble of different persons. The results presented in this paper demonstrate that the proposed human data analysis scheme has the potential to provide the necessary methodological (modeling, inference, and learning) framework for a cognitive behavior-based robot control system. More specifically, the proposed framework has the potential to be used for the recognition of abnormal gait patterns and the subsequent classification of specific walking pathologies, which is needed for the development of a context-aware robot mobility assistant.
Measuring amplitude and frequency modulations in noise using multiband energy operators

The statistical properties of the nonlinear energy operator Psi (s)=(ds/dt)/sup 2/-sd/sup 2//dt/sup 2/ and a related energy separation algorithm (ESA) are developed. The ESA uses Psi to demodulate noisy AM-FM signals. The performance of Psi and the ESA when applied to bandpass noisy AM-FM signals is examined. The predicted performance is found to be greatly improved if the local signal frequencies occur within the filter passband. Using this observation, a multiband energy operator and ESA approach are devised. The results suggest that greatly improved practical strategies are feasible for tracking and identifying local pattern coherencies manifested as local concentrations of signal frequencies.<<ETX>>
Multiscale Fractal Analysis of Musical Instrument Signals With Application to Recognition

In this paper, we explore nonlinear methods, inspired by the fractal theory for the analysis of the structure of music signals at multiple time scales, which is of importance both for their modeling and for their automatic computer-based recognition. We propose the multiscale fractal dimension (MFD) profile as a short-time descriptor, useful to quantify the multiscale complexity and fragmentation of the different states of the music waveform. We have experimentally found that this descriptor can discriminate several aspects among different music instruments, which is verified by further analysis on synthesized sinusoidal signals. We compare the descriptiveness of our features against that of Mel frequency cepstral coefficients (MFCCs), using both static and dynamic classifiers such as Gaussian mixture models (GMMs) and hidden Markov models (HMMs). The method and features proposed in this paper appear to be promising for music signal analysis, due to their capability for multiscale analysis of the signals and their applicability in recognition, as they accomplish an error reduction of up to 32%. These results are quite interesting and render the descriptor of direct applicability in large-scale music classification tasks.