nils-herrmann/clean_scirepeval huggingface dataset:
A Benchmark and Comparative Study of Video-Based Face Recognition on COX Face Database

Face recognition with still face images has been widely studied, while the research on video-based face recognition is inadequate relatively, especially in terms of benchmark datasets and comparisons. Real-world video-based face recognition applications require techniques for three distinct scenarios: 1) Videoto-Still (V2S); 2) Still-to-Video (S2V); and 3) Video-to-Video (V2V), respectively, taking video or still image as query or target. To the best of our knowledge, few datasets and evaluation protocols have benchmarked for all the three scenarios. In order to facilitate the study of this specific topic, this paper contributes a benchmarking and comparative study based on a newly collected still/video face database, named COX1 Face DB. Specifically, we make three contributions. First, we collect and release a largescale still/video face database to simulate video surveillance with three different video-based face recognition scenarios (i.e., V2S, S2V, and V2V). Second, for benchmarking the three scenarios designed on our database, we review and experimentally compare a number of existing set-based methods. Third, we further propose a novel Point-to-Set Correlation Learning (PSCL) method, and experimentally show that it can be used as a promising baseline method for V2S/S2V face recognition on COX Face DB. Extensive experimental results clearly demonstrate that video-based face recognition needs more efforts, and our COX Face DB is a good benchmark database for evaluation.
Attention driven face recognition: A combination of spatial variant fixations and glance

This paper proposes a novel face recognition algorithm inspired by the selective attention of Human Visual System (HVS). We record four observers' eye movements when they are viewing 100 FRGC  frontal view face images and find that the observers are highly consistent in the regions fixated. Inspired by the fact that fovea of HVS has a much higher spatial acuity than the periphery, a face recognition algorithm based on spatial variant sampling is proposed to simulate such foveated imaging phenomenon, where more information is reserved for the fixated regions. Moreover, information extracted from glance which adopts the low spatial frequency components of the image is integrated into the face recognition system to elicit a percept that occurs before any fixations. The experimental results on FERET database  demonstrate that the proposed method not only reduces the computational cost, but also achieves comparable performance, which shows that the characteristics of the HVS provide valuable insights into face recognition.
HeadNet: Pedestrian Head Detection Utilizing Body in Context

Pedestrian head with arbitrary poses and size is prohibitively difficult to detect in many real world applications. An appealing alternative is to utilize object detection technologies, which tend to be more and more mature and faster. However, general object detection technologies can hardly work in complicated scenarios where many heads are often too small to detect. In this paper, we present a novel approach that learns a semantic connection between pedestrian head and other body parts for head detection. Specifically, the proposed model, named as HeadNet, is based on PVANet backbone and also introduces beneficial strategies including online hard example mining (OHEM), fine-grained feature maps, RoI Align and Body in Context (BiC). Experiments demonstrate that our approach is able to utilize spatial semantics of the entire body effectively, and gains inspiring performance for pedestrian head detection.
Report on the FG 2015 Video Person Recognition Evaluation

This report presents results from the Video Person Recognition Evaluation held in conjunction with the 11th IEEE International Conference on Automatic Face and Gesture Recognition. Two experiments required algorithms to recognize people in videos from the Point-and-Shoot Face Recognition Challenge Problem (PaSC). The first consisted of videos from a tripod mounted high quality video camera. The second contained videos acquired from 5 different handheld video cameras. There were 1401 videos in each experiment of 265 subjects. The subjects, the scenes, and the actions carried out by the people are the same in both experiments. Five groups from around the world participated in the evaluation. The video handheld experiment was included in the International Joint Conference on Biometrics (IJCB) 2014 Handheld Video Face and Person Recognition Competition. The top verification rate from this evaluation is double that of the top performer in the IJCB competition. Analysis shows that the factor most effecting algorithm performance is the combination of location and action: where the video was acquired and what the person was doing.
Cascaded Shape Space Pruning for Robust Facial Landmark Detection

In this paper, we propose a novel cascaded face shape space pruning algorithm for robust facial landmark detection. Through progressively excluding the incorrect candidate shapes, our algorithm can accurately and efficiently achieve the globally optimal shape configuration. Specifically, individual landmark detectors are firstly applied to eliminate wrong candidates for each landmark. Then, the candidate shape space is further pruned by jointly removing incorrect shape configurations. To achieve this purpose, a discriminative structure classifier is designed to assess the candidate shape configurations. Based on the learned discriminative structure classifier, an efficient shape space pruning strategy is proposed to quickly reject most incorrect candidate shapes while preserve the true shape. The proposed algorithm is carefully evaluated on a large set of real world face images. In addition, comparison results on the publicly available BioID and LFW face databases demonstrate that our algorithm outperforms some state-of-the-art algorithms.
Robust FEC-CNN: A High Accuracy Facial Landmark Detection System

Facial landmark detection, as a typical and crucial task in computer vision, is widely used in face recognition, face animation, facial expression analysis, etc. In the past decades, many efforts are devoted to designing robust facial landmark detection algorithms. However, it remains a challenging task due to extreme poses, exaggerated facial expression, unconstrained illumination, etc. In this work, we propose an effective facial landmark detection system, recorded as Robust FEC-CNN (RFC), which achieves impressive results on facial landmark detection in the wild. Considering the favorable ability of deep convolutional neural network, we resort to FEC-CNN as a basic method to characterize the complex nonlinearity from face appearance to shape. Moreover, face bounding box invariant technique is adopted to reduce the landmark localization sensitivity to the face detector while model ensemble strategy is adopted to further enhance the landmark localization performance. We participate the Menpo Facial Landmark Localisation in-the-Wild Challenge and our RFC significantly outperforms the baseline approach APS. Extensive experiments on Menpo Challenge dataset and IBUG dataset demonstrate the superior performance of the proposed RFC.
Viewpoint invariant sign language recognition

Sign language is the primary modality of communication among deaf and mute society all over the world. This paper proposes a viewpoint independent method for sign recognition. Considering that two sequences of the same sign can be roughly considered as the input of a stereo vision system after time-warping, and the fundamental matrix associated with two views should be unique, we can convert the temporal-spatial recognition task as a verification task within a stereo vision framework. After time-warping of the input sequences, the proposed framework can reach both temporal and viewpoint invariance. We demonstrate the efficiency of the proposed framework by recognizing a vocabulary of 100 words of Chinese sign language. The recognition rate is up to 97% at rank 3. Furthermore, the proposed framework can be easily extended to other recognition tasks, such as gait recognition and lip-reading recognition.
Towards monitoring human activities using an omnidirectional camera

We propose an approach for monitoring human activities in an indoor environment using an omnidirectional camera. Robustly tracking people is prerequisite for modeling and recognizing human activities. An omnidirectional camera mounted on the ceiling is less prone to problems of occlusion. We use the Markov Random Field (MRF) to present both background and foreground, and adapt models effectively against environment changes. We employ a deformable model to adapt the foreground models to optimally match objects in different position within a pattern of view of the omnidirectional camera. In order to monitor human activity, we represent positions of people as spatial points and analyze moving trajectories within a time-spatial window. The method provides an efficient way to monitoring high-level human activities without exploring identities.
Automatic detection and translation of text from natural scenes

Large amounts of information are embedded in natural scenes. Signs are good examples of natural objects with high information content. In this paper, we discuss problems in automatic detection and translation of text from natural scenes. We describe the chal1enges of automatic text detection and propose methods to address these chal1enges. We extend example based machine translation technology for sign translation and present a prototype system for Chinese sign translation. This system is capable of capturing images, automatically detecting and recognizing text, and translating the text into English. The translation can be displayed on a palm size PDA, or synthesized as a voice output message over the earphones.
Semantics constrained dictionary learning for signer-independent sign language recognition

In this paper, a sparse coding based framework is proposed for sign language recognition (SLR), especially for the signer-independent case. To deal with the inter-signer variation, a dictionary capturing the common features among different signers is learnt by considering the semantic constraint. Thus for a given sign from an unknown signer, the sparse representation, which maintains more information of this specific sign class while neglecting the identity information as much as possible, can be generated. In our implementation, each sign is partitioned into a fixed number of fragments and the features fusing hand shape and moving trajectory are extracted from the fragments. The dictionary learnt from the training fragments can be taken as the basic subunits of signs and each fragment of sign video can be coded by these basis vectors. Finally, the recognition result is achieved through SVM with the concatenated sparse coding features of the fragments. The experiments and comparisons show that our method is more effective for the signer-independent recognition problem than other baseline methods. At the same time, it also performs well for the signer-dependent case.
Recursive Spatial Transformer (ReST) for Alignment-Free Face Recognition

Convolutional Neural Network (CNN) has led to significant progress in face recognition. Currently most CNNbased face recognition methods follow a two-step pipeline, i.e. a detected face is first aligned to a canonical one predefined by a mean face shape, and then it is fed into a CNN to extract features for recognition. The alignment step transforms all faces to the same shape, which can cause loss of geometrical information which is helpful in distinguishing different subjects. Moreover, it is hard to define a single optimal shape for the following recognition, since faces have large diversity in facial features, e.g. poses, illumination, etc. To be free from the above problems with an independent alignment step, we introduce a Recursive Spatial Transformer (ReST) module into CNN, allowing face alignment to be jointly learned with face recognition in an end-to-end fashion. The designed ReST has an intrinsic recursive structure and is capable of progressively aligning faces to a canonical one, even those with large variations. To model non-rigid transformation, multiple ReST modules are organized in a hierarchical structure to account for different parts of faces. Overall, the proposed ReST can handle large face variations and non-rigid transformation, and is end-to-end learnable and adaptive to input, making it an effective alignment-free face recognition solution. Extensive experiments are performed on LFW and YTF datasets, and the proposed ReST outperforms those two-step methods, demonstrating its effectiveness.
Stacked Progressive Auto-Encoders (SPAE) for Face Recognition Across Poses

Identifying subjects with variations caused by poses is one of the most challenging tasks in face recognition, since the difference in appearances caused by poses may be even larger than the difference due to identity. Inspired by the observation that pose variations change non-linearly but smoothly, we propose to learn pose-robust features by modeling the complex non-linear transform from the non-frontal face images to frontal ones through a deep network in a progressive way, termed as stacked progressive auto-encoders (SPAE). Specifically, each shallow progressive auto-encoder of the stacked network is designed to map the face images at large poses to a virtual view at smaller ones, and meanwhile keep those images already at smaller poses unchanged. Then, stacking multiple these shallow auto-encoders can convert non-frontal face images to frontal ones progressively, which means the pose variations are narrowed down to zero step by step. As a result, the outputs of the topmost hidden layers of the stacked network contain very small pose variations, which can be used as the pose-robust features for face recognition. An additional attractiveness of the proposed method is that no pose estimation is needed for the test images. The proposed method is evaluated on two datasets with pose variations, i.e., MultiPIE and FERET datasets, and the experimental results demonstrate the superiority of our method to the existing works, especially to those 2D ones.
Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition

In this paper, we tackle the continuous gesture recognition problem with a two streams Recurrent Neural Networks (2S-RNN) for the RGB-D data input. In our framework, the spotting-recognition strategy is used, that means the continuous gestures are first segmented into separated gestures, and then each isolated gesture is recognized by using the 2S-RNN. Concretely, the gesture segmentation is based on the accurate hand positions provided by the hand detector trained from Faster R-CNN. While in the recognition module, 2S-RNN is designed to efficiently fuse multi-modal features, i.e. the RGB and depth channels. The experimental results on both the validation and test sets of the Continuous Gesture Dataset (ConGD) have shown promising performance of the proposed framework. We ranked 1st in the ChaLearn LAP Large-scale Continuous Gesture Recognition Challenge with the mean Jaccard Index of 0.286915.
Coupling Alignments with Recognition for Still-to-Video Face Recognition

The Still-to-Video (S2V) face recognition systems typically need to match faces in low-quality videos captured under unconstrained conditions against high quality still face images, which is very challenging because of noise, image blur, low face resolutions, varying head pose, complex lighting, and alignment difficulty. To address the problem, one solution is to select the frames of `best quality' from videos (hereinafter called quality alignment in this paper). Meanwhile, the faces in the selected frames should also be geometrically aligned to the still faces offline well-aligned in the gallery. In this paper, we discover that the interactions among the three tasks-quality alignment, geometric alignment and face recognition-can benefit from each other, thus should be performed jointly. With this in mind, we propose a Coupling Alignments with Recognition (CAR) method to tightly couple these tasks via low-rank regularized sparse representation in a unified framework. Our method makes the three tasks promote mutually by a joint optimization in an Augmented Lagrange Multiplier routine. Extensive experiments on two challenging S2V datasets demonstrate that our method outperforms the state-of-the-art methods impressively.
Visual Hull Embossment by Graph Cuts

In this paper, we present a visual hull embossment algorithm which works on the base of a visual hull to reconstruct a precise model from calibrated color images. The algorithm uses the geometry of the visual hull to compute voxel normals and visibilities. Then a photometric consistency field (PCF) is computed, which describes the photometric consistency score of every voxel. The visual hull is refined in a view dependent fashion. For each view, we compute a view dependent PCF (VDPCF) by resampling the PCF and then find an optimal depth image in VDPCF with graph cut based energy minimization, which takes into consideration data, smoothness and adjustment terms simultaneously. The depth images are used for carving voxels. Our algorithm can produce precise models with a moderate speed.
Calibration of a hybrid camera network

Visual surveillance using a camera network has imposed new challenges to camera calibration. An essential problem is that a large number of cameras may not have a common field of view or even be synchronized well. We propose to use a hybrid camera network that consists of catadioptric and perspective cameras for a visual surveillance task. The relations between multiple views of a scene captured from different cameras can be then calibrated under the catadioptric camera's coordinate system. We address the important issue of how to calibrate the hybrid camera network. We calibrate the hybrid camera network in three steps. First, we calibrate the catadioptric camera using only the vanishing points. In order to reduce computational complexity, we calibrate the camera without the mirror first and then calibrate the catadioptric camera system. Second, we determine 3D positions of some points using as few as two spatial parallel lines and some equidistance points. Finally, we calibrate other perspective cameras based on these known spatial points.
Learning Euclidean-to-Riemannian Metric for Point-to-Set Classification

In this paper, we focus on the problem of point-to-set classification, where single points are matched against sets of correlated points. Since the points commonly lie in Euclidean space while the sets are typically modeled as elements on Riemannian manifold, they can be treated as Euclidean points and Riemannian points respectively. To learn a metric between the heterogeneous points, we propose a novel Euclidean-to-Riemannian metric learning framework. Specifically, by exploiting typical Riemannian metrics, the Riemannian manifold is first embedded into a high dimensional Hilbert space to reduce the gaps between the heterogeneous spaces and meanwhile respect the Riemannian geometry of the manifold. The final distance metric is then learned by pursuing multiple transformations from the Hilbert space and the original Euclidean space (or its corresponding Hilbert space) to a common Euclidean subspace, where classical Euclidean distances of transformed heterogeneous points can be measured. Extensive experiments clearly demonstrate the superiority of our proposed approach over the state-of-the-art methods.
Multi-view Deep Network for Cross-View Classification

Cross-view recognition that intends to classify samples between different views is an important problem in computer vision. The large discrepancy between different even heterogenous views make this problem quite challenging. To eliminate the complex (maybe even highly nonlinear) view discrepancy for favorable cross-view recognition, we propose a multi-view deep network (MvDN), which seeks for a non-linear discriminant and view-invariant representation shared between multiple views. Specifically, our proposed MvDN network consists of two sub-networks, view-specific sub-network attempting to remove view-specific variations and the following common sub-network attempting to obtain common representation shared by all views. As the objective of MvDN network, the Fisher loss, i.e. the Rayleigh quotient objective, is calculated from the samples of all views so as to guide the learning of the whole network. As a result, the representation from the topmost layers of the MvDN network is robust to view discrepancy, and also discriminative. The experiments of face recognition across pose and face recognition across feature type on three datasets with 13 and 2 views respectively demonstrate the superiority of the proposed method, especially compared to the typical linear ones.
Hyperspectral Light Field Stereo Matching

In this paper, we describe how scene depth can be extracted using a hyperspectral light field capture (H-LF) system. Our H-LF system consists of a <inline-formula><tex-math notation="LaTeX">$5 \times 6$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>5</mml:mn><mml:mo>×</mml:mo><mml:mn>6</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="zhu-ieq1-2827049.gif"/></alternatives></inline-formula> array of cameras, with each camera sampling a different narrow band in the visible spectrum. There are two parts to extracting scene depth. The first part is our novel cross-spectral pairwise matching technique, which involves a new spectral-invariant feature descriptor and its companion matching metric we call bidirectional weighted normalized cross correlation (BWNCC). The second part, namely, H-LF stereo matching, uses a combination of spectral-dependent correspondence and defocus cues. These two new cost terms are integrated into a Markov Random Field (MRF) for disparity estimation. Experiments on synthetic and real H-LF data show that our approach can produce high-quality disparity maps. We also show that these results can be used to produce the complete plenoptic cube in addition to synthesizing all-focus and defocused color images under different sensor spectral responses.
Occlusion-Free Face Alignment: Deep Regression Networks Coupled with De-Corrupt AutoEncoders

Face alignment or facial landmark detection plays an important role in many computer vision applications, e.g., face recognition, facial expression recognition, face animation, etc. However, the performance of face alignment system degenerates severely when occlusions occur. In this work, we propose a novel face alignment method, which cascades several Deep Regression networks coupled with De-corrupt Autoencoders (denoted as DRDA) to explicitly handle partial occlusion problem. Different from the previous works that can only detect occlusions and discard the occluded parts, our proposed de-corrupt autoencoder network can automatically recover the genuine appearance for the occluded parts and the recovered parts can be leveraged together with those non-occluded parts for more accurate alignment. By coupling de-corrupt autoencoders with deep regression networks, a deep alignment model robust to partial occlusions is achieved. Besides, our method can localize occluded regions rather than merely predict whether the landmarks are occluded. Experiments on two challenging occluded face datasets demonstrate that our method significantly outperforms the state-of-the-art methods.